Orq.ai Documentation - AI Gateway & LLM Collaboration Platform

MCP

Claude Desktop is Anthropic’s desktop application that supports Model Context Protocol (MCP) integrations. By configuring the Orq MCP server, you can access all Orq.ai features directly in your Claude Desktop conversations.

Prerequisites

Claude Desktop app installed
Active Orq.ai account
Orq.ai API key
Node.js installed (required for npx mcp-remote)

Installation

You can configure the Orq MCP server through Claude Desktop Settings or using the Terminal.

Option 1: Settings Path (Recommended)

Open Claude Desktop Settings by clicking Claude in the top-left menu, then select Settings
Click Developer in the sidebar
Click Edit Config to open the claude_desktop_config.json file
Paste the following configuration into the file:

{
  "preferences": {
    "sidebarMode": "chat",
    "coworkScheduledTasksEnabled": false
  },
  "mcpServers": {
    "orq": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://my.orq.ai/v2/mcp",
        "--header",
        "Authorization:${ORQ_AUTH_HEADER}"
      ],
      "env": {
        "ORQ_AUTH_HEADER": "Bearer <ORQ_API_TOKEN>"
      }
    }
  }
}

Replace <ORQ_API_TOKEN> with your actual API key from Workspace Settings → API Keys
Save the file and restart Claude Desktop

If the config file is empty, paste the entire JSON structure above. If it already has content, add only the orq entry to the existing mcpServers object. Do not overwrite the preferences block or any other existing keys.

Option 2: Terminal Path

If you prefer using the terminal, you can directly edit the config file:macOS:Run these commands in your terminal:

# Open the config file in your default text editor
open ~/Library/Application\ Support/Claude/claude_desktop_config.json

Windows:Run this command in your Command Prompt or PowerShell:

# Open the config file in Notepad
notepad %APPDATA%\Claude\claude_desktop_config.json

Linux:Run this command in your terminal:

# Open the config file in nano (or use your preferred editor)
nano ~/.config/Claude/claude_desktop_config.json

Then paste the following configuration:

{
  "preferences": {
    "sidebarMode": "chat",
    "coworkScheduledTasksEnabled": false
  },
  "mcpServers": {
    "orq": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://my.orq.ai/v2/mcp",
        "--header",
        "Authorization:${ORQ_AUTH_HEADER}"
      ],
      "env": {
        "ORQ_AUTH_HEADER": "Bearer <ORQ_API_TOKEN>"
      }
    }
  }
}

Replace <ORQ_API_TOKEN> with your actual API key, save the file, and restart Claude Desktop.

If the file doesn’t exist, the command will create it. Make sure to use valid JSON formatting.

Verify Installation

After restarting Claude Desktop, start a new conversation and ask:

Can you list the available models from Orq?

Claude will use the Orq MCP integration to fetch and display your available AI models.

What You Can Do

Once connected, you can use natural language in Claude Desktop to perform these operations:

Agents

Create an agent with custom instructions and tools
Get agent configuration for [agent-key]
Update agent [agent-key] with new instructions or model
Configure agent with evaluators and guardrails

Analytics

Get analytics overview for my workspace
Show me workspace metrics for the last 7 days
Query analytics filtered by deployment ID

Datasets

Create a dataset called "customer-queries"
List all datapoints in dataset [dataset-key]
Add datapoints to dataset [dataset-key]
Update datapoint [datapoint-id]
Delete specific datapoints in dataset [dataset-key]
Delete dataset [dataset-key]

Experiments

Create an experiment from dataset [dataset-key]
List all experiment runs
Export experiment run [run-id] as CSV
Run experiment and auto-evaluate results

Evaluators

Get evaluator configuration for [evaluator-key]
Create an LLM-as-a-Judge evaluator for tone
Create a Python evaluator to check response length
Add evaluator to experiment [experiment-key]
Update evaluator [evaluator-key] with a new prompt
Update Python evaluator [evaluator-key] with revised code

Traces

List traces from the last 24 hours
Show me traces with errors
Get span details for trace [trace-id]
Find the slowest traces from today
Show all traces for thread [thread-id]

Models

List all available chat models
List all available embedding models

Registry

List registry keys for filtering traces
List top values for [attribute-key]

Search for datasets named "customer"
Find experiments in project [project-id]
List directories in project [project-id]

Documentation

Search the Orq.ai docs for [topic]

Managing Entities

Delete agent [agent-key]
Delete experiment [experiment-key]
Delete evaluator [evaluator-key]
Delete prompt [prompt-key]
Delete knowledge base [knowledge-base-key]

Use delete_dataset to delete a dataset along with all its datapoints.

Usage Examples

Create Experiments

Create an experiment called "Model Comparison Test" using my "customer-queries" dataset.
Configure it with GPT-5.2 and Claude Sonnet 4.6, then run it.

Claude will:

Use search_entities to find the “customer-queries” dataset
Use create_experiment with the name “Model Comparison Test” and auto-run enabled
Configure two task columns (one for GPT-5.2, one for Claude Sonnet 4.6)
Execute both models against the dataset automatically via the auto-run option
Provide a summary of the results with evaluation metrics

Analyze Traces

Show me all errors from my Orq traces in the last 24 hours

Claude will:

Calculate the time range for the last 24 hours
Use list_traces with error status filter
Analyze the trace data
Provide error count and types, affected deployments, time distribution, and suggested fixes based on error patterns

Generate Synthetic Datasets

Generate 100 realistic customer support conversations about a SaaS product
and create a dataset called "Support Training" in Orq

Claude will:

Generate 100 realistic customer support conversation examples (questions and expected responses)
Use create_dataset to create a new dataset named “Support Training”
Use create_datapoints to add all 100 conversations to the dataset
Confirm creation with the dataset ID and sample of generated data

Performance Analysis

Has my system's performance improved over the last week?
Show latency trends and cost metrics from Orq analytics.

Claude will:

Use query_analytics with a 7-day time range
Analyze average latency changes over the week
Review token usage patterns and cost trends
Examine error rate fluctuations
Compare performance across different models
Provide a summary report with insights on whether performance has improved or decreased

Troubleshooting

MCP Server Not Connecting

Verify the config file path is correct for your OS
Check the JSON syntax is valid (no trailing commas, proper quotes)
Ensure your API key is valid and has the required permissions
Restart Claude Desktop after making config changes
Check the Claude Desktop logs for error messages

Authentication Errors

Confirm your API key is active in Orq.ai Settings
Make sure the API key has workspace access permissions
Verify the Authorization header format: Bearer YOUR_KEY
Try generating a new API key if the current one is expired

Slow Responses

MCP operations over HTTP can take a few seconds:

Be patient with large dataset operations
Break complex workflows into smaller steps
Check Orq.ai service status at status.orq.ai

Tool Not Found

Verify the Orq MCP server is properly configured in your config file
Restart Claude Desktop to reload the Orq MCP configuration
Try rephrasing your request
Check the MCP tools list

Additional Configuration

Multiple Workspaces

If you work with multiple Orq.ai workspaces, you can configure multiple MCP servers:

{
  "mcpServers": {
    "orq-production": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://my.orq.ai/v2/mcp",
        "--header",
        "Authorization:${ORQ_PROD_AUTH_HEADER}"
      ],
      "env": {
        "ORQ_PROD_AUTH_HEADER": "Bearer <PRODUCTION_API_TOKEN>"
      }
    },
    "orq-staging": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://my.orq.ai/v2/mcp",
        "--header",
        "Authorization:${ORQ_STAGING_AUTH_HEADER}"
      ],
      "env": {
        "ORQ_STAGING_AUTH_HEADER": "Bearer <STAGING_API_TOKEN>"
      }
    }
  }
}

Skills

Skills add pre-built agentic workflows to Claude for the full Build, Evaluate, Optimize lifecycle. See the Skills page for the full reference.

Installation

Skills for Claude Desktop are managed through the Claude.ai web interface and automatically apply across all your Claude clients, including the desktop app.

Custom Skills require a Pro, Max, Team, or Enterprise plan. To install, open Claude.ai in your browser, go to Settings → Features → Skills, and upload the orq-skills zip.

Available Skills

Once installed, Claude picks the right skill automatically based on what you describe.

Skill	Description
build-agent	Design, create, and configure an Orq.ai agent
build-evaluator	Create validated LLM-as-a-Judge evaluators
analyze-trace-failures	Read production traces and categorize failures
run-experiment	Create and run experiments with evaluation
generate-synthetic-dataset	Generate and curate evaluation datasets
optimize-prompt	Analyze and optimize system prompts
setup-observability	Instrument LLM applications with orq.ai tracing: AI Router for zero-code traces, or OpenTelemetry for framework-level spans
compare-agents	Run cross-framework agent comparisons using evaluatorq

Slash commands (/orq:quickstart, /orq:traces, etc.) are only available in Claude Code. See Skills for details.

Get Started

Assistants

Claude Desktop | MCP

MCP

Prerequisites

Installation

Verify Installation

What You Can Do

Usage Examples

Create Experiments

Analyze Traces

Generate Synthetic Datasets

Performance Analysis

Troubleshooting

Additional Configuration

Multiple Workspaces

Skills

Installation

Available Skills

Get Started

Assistants

​MCP

​Prerequisites

​Installation

​Verify Installation

​What You Can Do

​Usage Examples

​Create Experiments

​Analyze Traces

​Generate Synthetic Datasets

​Performance Analysis

​Troubleshooting

​Additional Configuration

​Multiple Workspaces

​Skills

​Installation

​Available Skills

MCP

Prerequisites

Installation

Verify Installation

What You Can Do

Usage Examples

Create Experiments

Analyze Traces

Generate Synthetic Datasets

Performance Analysis

Troubleshooting

Additional Configuration

Multiple Workspaces

Skills

Installation

Available Skills