Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.orq.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
Overview

Visual Studio Code supports MCP servers through the GitHub Copilot extension. With the Orq MCP integration, access the Orq.ai workspace directly from Copilot Chat and the editor environment.

​
Prerequisites

​
Installation

​
Add MCP Server

  1. Open the Command Palette with ⌘⇧P (macOS) or Ctrl+Shift+P (Windows/Linux)
  2. Search for and select MCP: Add Server
  3. Select HTTP (HTTP or Server-Sent Events) as the server type
  4. Enter the server URL: https://my.orq.ai/v2/mcp
  5. Name the server orq when prompted
VS Code will create or update a .vscode/mcp.json file in the workspace root. Open it and replace the contents with: 
{
  "inputs": [
    {
      "type": "promptString",
      "id": "orq-api-key",
      "description": "Orq.ai API Key",
      "password": true
    }
  ],
  "servers": {
    "orq": {
      "url": "https://my.orq.ai/v2/mcp",
      "type": "http",
      "headers": {
        "Authorization": "Bearer ${input:orq-api-key}"
      }
    }
  }
}
When connecting to the server for the first time, VS Code will prompt for the Orq.ai API key and store it securely in the OS secret store. The key is never written to disk.

​
Verify Installation

  1. Open Copilot Chat with ⌃⌘I (macOS) or Ctrl+Alt+I (Windows/Linux)
  2. Ask:
Can you list the available models from Orq?
If the integration is working, a list of AI models from the Orq.ai workspace will appear. To check server status at any time, open the Command Palette and run MCP: List Servers.

​
Available Commands

Use natural language to ask Copilot to perform these operations:
  • create an agent with custom instructions and tools
  • get agent configuration for [agent-key]
  • update agent [agent-key] with new instructions or model
  • configure agent with evaluators and guardrails
  • get analytics overview for my workspace
  • show me workspace metrics for the last 7 days
  • query analytics filtered by deployment ID
  • create a dataset called "customer-queries"
  • list all datapoints in dataset [dataset-key]
  • add datapoints to dataset [dataset-key]
  • update datapoint [datapoint-id]
  • delete specific datapoints in dataset [dataset-key]
  • delete dataset [dataset-key]
  • create an experiment from dataset [dataset-key]
  • list all experiment runs
  • export experiment run [run-id] as CSV
  • run experiment and auto-evaluate results
  • get evaluator configuration for [evaluator-key]
  • create an LLM-as-a-Judge evaluator for tone
  • create a Python evaluator to check response length
  • add evaluator to experiment [experiment-key]
  • update evaluator [evaluator-key] with a new prompt
  • update Python evaluator [evaluator-key] with revised code
  • list traces from the last 24 hours
  • show me traces with errors
  • get span details for trace [trace-id]
  • find the slowest traces from today
  • show all traces for thread [thread-id]
  • list all available chat models
  • list all available embedding models
  • list registry keys for filtering traces
  • list top values for [attribute-key]
  • search the Orq.ai docs for [topic]
  • delete agent [agent-key]
  • delete experiment [experiment-key]
  • delete evaluator [evaluator-key]
  • delete prompt [prompt-key]
  • delete knowledge base [knowledge-base-key]
Use delete_dataset to delete a dataset along with all its datapoints.

​
Usage Examples

​
Chat Panel Commands

Use natural language in Copilot Chat: 
create a dataset called "API Tests" with 20 synthetic API request examples
The assistant will:
  1. Generate 20 synthetic API request examples
  2. Use create_dataset to create a new dataset named “API Tests”
  3. Use create_datapoints to add all examples to the dataset
  4. Confirm creation with the dataset ID
show me errors from the last 24 hours
The assistant will:
  1. Calculate the time range for the last 24 hours
  2. Use list_traces with error status filter
  3. Display trace IDs, error messages, and timestamps
  4. Provide a summary of error types and frequency
create an experiment comparing GPT-5.2 and Claude Sonnet 4.6 using the "user-queries" dataset
The assistant will:
  1. Search for the “user-queries” dataset using search_entities
  2. Use create_experiment with two configurations (one for GPT-5.2, one for Claude Sonnet 4.6)
  3. Run the experiment against all datapoints in the dataset
  4. Display the experiment ID and status

​
Inline Code Integration

Copilot can use Orq MCP context while coding:
  1. Select code in the editor
  2. Open Copilot Chat (⌃⌘I / Ctrl+Alt+I)
  3. Ask about traces or analytics related to the code
Example: 
# Select this function
def process_user_query(query):
    response = orq.deployments.invoke(...)
    return response
Then ask: 
how has this endpoint performed over the last week?
The assistant will:
  1. Extract the deployment key from the selected code
  2. Use query_analytics with the deployment filter
  3. Set time range to the last 7 days
  4. Analyze performance metrics (requests, errors, latency, tokens)
  5. Provide insights and recommendations based on the data

​
Dataset Creation from Code

[
  {"input": "What is AI?", "expected_output": "Artificial Intelligence..."},
  {"input": "Explain ML", "expected_output": "Machine Learning..."}
]

create a dataset from the JSON array above and add it to my workspace
The assistant will:
  1. Parse the JSON array from the editor
  2. Use create_dataset to create a new dataset with an auto-generated name
  3. Use create_datapoints to add each entry as a datapoint
  4. Confirm the dataset ID and number of datapoints added

​
Experiment Analysis

create an experiment using "customer-feedback" dataset, configure it with two prompts: one focused on empathy and one focused on brevity, then run it and summarize the results
The assistant will:
  1. Search for the “customer-feedback” dataset using search_entities
  2. Use create_experiment with two prompt variants (empathy-focused and brevity-focused) and auto-run enabled
  3. Execute both variants against all datapoints automatically via the auto-run option
  4. Use get_experiment_run to retrieve evaluation metrics
  5. Compare the two variants and provide a summary of which performed better

​
Performance Investigation

find the 5 slowest traces from today and show me their span details
The assistant will:
  1. Use list_traces with today’s date filter
  2. Sort traces by duration (descending)
  3. Retrieve the top 5 slowest traces
  4. Use list_spans to fetch span information for each trace
  5. Display latency breakdowns, bottlenecks, and performance insights

​
Synthetic Data Generation

generate 50 realistic customer support questions about a SaaS product and create a dataset called "Support Training Data"
The assistant will:
  1. Generate 50 synthetic customer support questions and expected responses
  2. Use create_dataset to create a dataset named “Support Training Data”
  3. Use create_datapoints to add all 50 examples to the dataset
  4. Confirm creation with the dataset ID and sample of generated questions

​
Troubleshooting

  1. Open the Command Palette and run MCP: List Servers
  2. Select the orq server and choose Show Output to view logs
  3. Restart VS Code and reconnect. VS Code will prompt for the API key again on first connection
  1. Confirm the API key is valid in Workspace Settings → API Keys
  2. Ensure the API key has the necessary permissions
  3. Try regenerating the API key, then restart VS Code so the input prompt appears again
  1. Run MCP: List Servers from the Command Palette and confirm the orq server status is active
  2. Ensure GitHub Copilot is signed in and active
  3. Check network connectivity
  4. Review the server output log for error details