Claude Code

MCP Integration

Access the Orq.ai workspace directly from Claude Code. Manage experiments, query traces, and configure agents using natural language.

AI Gateway Beta

Route Claude Code’s model calls through the AI Gateway.

MCP

Claude Code is Anthropic’s official CLI that brings Claude’s capabilities to the terminal and development workflow. With the Orq MCP integration, access all Orq.ai features directly through Claude Code’s conversational interface.

Prerequisites

Claude Code CLI installed
Active Orq.ai account
Orq.ai API key

Installation

Add the Orq MCP server to Claude Code with a single command:

claude mcp add --transport http orq https://my.orq.ai/v2/mcp --header "Authorization: Bearer ${ORQ_API_KEY}"

Set the ORQ_API_KEY environment variable before running the command:

export ORQ_API_KEY="your-api-key-here"

Verify Installation

Check that the Orq MCP is installed:

claude mcp list

You should see orq in the list of available MCP servers.

Available Commands

Once integrated, you can ask Claude Code to perform these operations:

Agents

Create an agent with custom instructions and tools
Get agent configuration for [agent-key]
Update agent [agent-key] with new instructions or model
Configure agent with evaluators and guardrails
Invoke agent [agent-key] with input [message]
Retrieve agent response [response-id]

Deployments

Create a deployment called [deployment-key]
Get deployment configuration for [deployment-key]

Skills

Create a skill called [skill-key]
List all skills in my workspace
Get skill [skill-key]
Update skill [skill-key]
Delete skill [skill-key]

Analytics

Get analytics overview for my workspace
Show me workspace metrics for the last 7 days
Query analytics filtered by deployment ID

Datasets

Create a dataset called "customer-queries"
List all datapoints in dataset [dataset-key]
Add datapoints to dataset [dataset-key]
Update datapoint [datapoint-id]
Delete specific datapoints in dataset [dataset-key]
Delete dataset [dataset-key]

Experiments

Create an experiment from dataset [dataset-key]
List all experiment runs
Export experiment run [run-id] as CSV
Run experiment and auto-evaluate results

Evaluators

Get evaluator configuration for [evaluator-key]
Create an LLM-as-a-Judge evaluator for tone
Create a Python evaluator to check response length
Add evaluator to experiment [experiment-key]
Update evaluator [evaluator-key] with a new prompt
Update Python evaluator [evaluator-key] with revised code

Traces

List traces from the last 24 hours
Show me traces with errors
Get span details for trace [trace-id]
Find the slowest traces from today
Show all traces for thread [thread-id]

Models

List all available chat models
List all available embedding models
Invoke model [model-id] with prompt [message]

Search for datasets named "customer"
Find experiments in project [project-id]
List directories in project [project-id]

Documentation

Search the Orq.ai docs for [topic]

Managing Entities

Delete agent [agent-key]
Delete experiment [experiment-key]
Delete evaluator [evaluator-key]
Delete prompt [prompt-key]
Delete knowledge base [knowledge-base-key]

Use delete_dataset to delete a dataset along with all its datapoints.

Usage Examples

Create an Experiment

Create an experiment called "GPT-5.6 Sol vs Claude Sonnet 5 Comparison" using the "customer-queries" dataset

Claude Code will:

Use search_entities to find the “customer-queries” dataset
Use create_experiment with the specified name and dataset ID
Configure task columns with GPT-5.6 Sol and Claude Sonnet 5 models
Return the experiment ID and configuration details

Query Trace Analytics

Has my system thrown any errors in the last 24 hours?

Claude Code will:

Calculate the time range for the last 24 hours
Use list_traces with error status filter
Analyze the error data
Provide a summary of total error count, error types and frequencies, affected traces, and time distribution

Create a Synthetic Dataset

Create a dataset called "Product Questions" with 50 synthetic customer questions about e-commerce products

Claude Code will:

Generate 50 synthetic customer questions about e-commerce products
Use create_dataset to create a new dataset named “Product Questions”
Use create_datapoints to add all 50 questions to the dataset
Confirm creation with the dataset ID and summary

Performance Analysis

Has my system's performance improved or decreased over the past week?

Claude Code will:

Use query_analytics with a 7-day time range
Analyze average latency trends over time
Review token usage patterns and cost variations
Compare error rate changes across the week
Provide insights on model performance comparisons and trends

Complete Experiment Creation

I have a CSV file with 100 customer queries. Create a dataset, add an LLM evaluator for tone and accuracy, then run an experiment comparing GPT-5.6 Sol and Claude Sonnet 5

Claude Code will:

Read and parse the CSV file
Use create_dataset to create a new dataset with an auto-generated name
Use create_datapoints to add all 100 customer queries from the CSV
Use create_llm_eval to create an LLM-as-a-Judge evaluator for tone
Use create_llm_eval again to create an LLM-as-a-Judge evaluator for accuracy
Use create_experiment with the dataset ID and auto-run enabled
Configure two task columns (one for GPT-5.6 Sol, one for Claude Sonnet 5)
Execute the experiment automatically via the auto-run option
Summarize the results with evaluation scores for both models

Trace Investigation

Show me the 10 slowest traces from yesterday and explain what might be causing the latency

Claude Code will:

Calculate yesterday’s date range
Use list_traces with latency sorting (descending) and limit of 10
Use list_spans to retrieve span information for each trace
Analyze the execution patterns and span durations
Provide performance insights identifying bottlenecks
Suggest optimization opportunities based on the data

Troubleshooting

Authentication Errors

Verify the API key is valid: echo $ORQ_API_KEY
Check the API key has the necessary permissions
Re-add the MCP with the correct API key

Connection Issues

Verify the endpoint URL is correct
Check internet connectivity
Try removing and re-adding the integration

Tool Not Found

Get MCP server details: claude mcp get orq
Verify the MCP is properly installed: claude mcp list

Plugins

The orq-ai/assistant-plugins marketplace exposes three plugins for Claude Code. Add the marketplace once, then install whichever plugins you need:

claude plugin marketplace add orq-ai/assistant-plugins

Plugin	Purpose
`orq-skills`	Pre-built agentic workflows and slash commands for the Build, Evaluate, Optimize lifecycle. Includes the Orq MCP server.
`orq-mcp`	Standalone Orq MCP server. Use this if you only want platform tool access without the skills bundle.
`orq-trace`	Automatically traces Claude Code sessions to Orq.ai: captures sessions, turns, tool calls, and LLM responses as hierarchical OTLP spans.

orq-skills already bundles the MCP server. Don’t install orq-mcp alongside it or the MCP will be registered twice.

Skills

Skills extend Claude Code with pre-built agentic workflows for the full Build, Evaluate, Optimize lifecycle. See the Skills page for the full reference.

Installation

# Requires marketplace to be added first (see Plugins section above)
# Installs skills, commands, agents, and the MCP server
claude plugin install orq-skills@orq-claude-plugin

# Clone and load locally as a fallback
git clone https://github.com/orq-ai/assistant-plugins.git ~/.orq-skills
claude --plugin-dir ~/.orq-skills

MCP server only

If you only want the Orq MCP server without the skills bundle, install orq-mcp instead:

claude plugin install orq-mcp@orq-claude-plugin

This gives Claude access to Orq.ai platform tools (agents, analytics, traces, experiments) without registering the agentic workflows.

Session tracing

The orq-trace plugin captures every Claude Code session as a hierarchical trace in Orq.ai: useful for reviewing past coding sessions, sharing context with teammates, or analyzing tool-call patterns across runs.

claude plugin install orq-trace@orq-claude-plugin

Set ORQ_API_KEY in the environment so the plugin can ship spans to the workspace. Sessions appear under Traces once the next session starts.

Commands

Quick slash-command actions available in Claude Code:

Command	Description	Usage
quickstart	Interactive onboarding: credentials, MCP setup, skills tour	`/orq:quickstart`
workspace	Workspace overview: Agents, Deployments, Prompts, Datasets, Experiments	`/orq:workspace [section]`
traces	Query and summarize Traces with filters	`/orq:traces [--deployment name] [--status error] [--last 24h]`
models	List available AI models by provider	`/orq:models [search-term]`
analytics	Usage Analytics: requests, cost, tokens, errors	`/orq:analytics [--last 24h] [--group-by model]`
orq-manage-skills	Manage Orq.ai Skills (platform entities): list, get, create, update, retire, delete	`/orq:orq-manage-skills [list\|get\|create\|update\|retire\|delete] [name-or-id]`

Available Skills

Triggered by describing what you need. Claude Code picks the right skill automatically.

Skill	Description	Source
build-agent	Design, create, and configure an Orq.ai Agent with tools, instructions, Knowledge Bases, and Memory	SKILL.md
build-evaluator	Create validated LLM-as-a-Judge Evaluators following evaluation best practices	SKILL.md
recommend-evaluators	Recommend the Evaluators an agent is missing, from its traces or, with no traffic, from its instructions and config; skips the ones already attached and creates each only after approval	SKILL.md
evaluator-alignment	Align an existing LLM judge (boolean, categorical, or numeric) to human judgment: measure how often it changes its mind, group the least reliable cases, rewrite the judge prompt, and recreate the Evaluator after approval	SKILL.md
analyze-traces	Read production Traces, identify what is failing, build failure taxonomies, and categorize issues	SKILL.md
improve-agent	Improve an underperforming Orq.ai Agent or Deployment: rewrite its instructions against a prompting framework, or move a configuration knob, grounded in the error-analysis file `analyze-traces` writes	SKILL.md
run-experiment	Create and run Orq.ai Experiments: compare configurations with specialized agent, conversation, and RAG evaluation	SKILL.md
generate-synthetic-dataset	Generate and curate evaluation Datasets: structured generation, quick from description, expansion, and dataset maintenance	SKILL.md
invoke-deployment	Invoke Orq.ai Deployments, Agents, and models via the Python SDK or HTTP API, with correct variable substitution, streaming, and identity tracking	SKILL.md
setup-observability	Instrument LLM applications with Orq.ai tracing. Covers AI Gateway (zero-code traces) and OpenTelemetry/OpenInference. Guides from framework detection through baseline verification to trace enrichment	SKILL.md
compare-agents	Run cross-framework agent comparisons: compare any combination of Orq.ai, LangGraph, CrewAI, OpenAI Agents SDK, or Vercel AI SDK agents head-to-head on the same dataset using `evaluatorq`	SKILL.md
red-team	Run adversarial attacks against deployed agents or static datasets with the `evaluatorq` red team CLI. Covers OWASP-ASI (agentic: goal hijacking, tool misuse) and OWASP-LLM (model-level: prompt injection, system prompt leakage)	SKILL.md
evaluatorq	Write and run `evaluatorq` evaluation scripts (Python or TypeScript) for a single agent or deployment. Supports custom scorers, dataset-driven runs, and LLM-as-a-Judge Evaluators	SKILL.md
simulate-agent	Run multi-turn simulations with `evaluatorq` primitives (`simulate()`, `generate_and_simulate()`, `wrap_simulation_agent()`): drive an agent under test with a simulated user and score each turn with a built-in judge	SKILL.md
manage-skills	List, inspect, create, update, retire, and delete Orq.ai Skills (platform entities). Handles naming rules, template integration (`{{skill.key}}`), reference scanning, and safe deletion	SKILL.md
create-skill	Build or update an agent skill from an API, CLI, or MCP surface: probe the surface, verify each claim, write a tested contract, and register it	SKILL.md
shared	Reference bundle the other skills read: the verified `orq` CLI trace query contract and the run-key preflight. Not invoked on its own	SKILL.md
orq-cli	Drive the `orq` command-line interface: install check, authentication, workspace selection, `orq doctor` troubleshooting, and read/write commands with JSON output	SKILL.md

AI Gateway

Beta Route Claude Code model requests through the Orq.ai AI Gateway. Set an Orq.ai API key as ORQ_API_KEY, then choose the credential to use for model requests.

Use the workspace’s provider configuration

Use the Orq.ai API key as the model client’s credential:

Shell

export ANTHROPIC_BASE_URL="https://my.orq.ai/v3/anthropic"
export ANTHROPIC_AUTH_TOKEN="$ORQ_API_KEY"
export ANTHROPIC_API_KEY=""
export ANTHROPIC_MODEL="anthropic/claude-sonnet-5"
export ENABLE_TOOL_SEARCH=true
unset CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS

To use an integration other than the workspace default, add x-orq-integration-anthropic with the integration id or its exact display name. See Provider integrations.

Shell

export ANTHROPIC_CUSTOM_HEADERS="x-orq-integration-anthropic: Production"

Use a Claude subscription

Sign in to Claude Code with a Claude subscription. Keep that login for Anthropic and send the Orq.ai key separately:

Shell

unset ANTHROPIC_AUTH_TOKEN ANTHROPIC_API_KEY
export ANTHROPIC_BASE_URL="https://my.orq.ai/v3/anthropic"
export ANTHROPIC_MODEL="anthropic/claude-sonnet-5"
export ANTHROPIC_CUSTOM_HEADERS="x-orq-api-key: $ORQ_API_KEY"
export ENABLE_TOOL_SEARCH=true
unset CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS
claude

Claude Code sends the subscription token and its OAuth beta header. The AI Gateway forwards the credential to Anthropic, so the subscription’s usage limits apply. Orq.ai still records the model request in Traces. For automated sessions, run claude setup-token and set the resulting token as CLAUDE_CODE_OAUTH_TOKEN before launching Claude Code. Keep ANTHROPIC_AUTH_TOKEN and ANTHROPIC_API_KEY unset so they do not replace the subscription credential.

Use an Anthropic API key

Set an Anthropic API key as ANTHROPIC_API_KEY and keep the Orq.ai key in the custom header:

Shell

unset ANTHROPIC_AUTH_TOKEN
export ANTHROPIC_BASE_URL="https://my.orq.ai/v3/anthropic"
export ANTHROPIC_API_KEY="<ANTHROPIC_API_KEY>"
export ANTHROPIC_MODEL="anthropic/claude-sonnet-5"
export ANTHROPIC_CUSTOM_HEADERS="x-orq-api-key: $ORQ_API_KEY"
export ENABLE_TOOL_SEARCH=true
unset CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS
claude

Anthropic bills model usage to the account that owns the API key. Both Anthropic credential options require a public model served directly by Anthropic. See authentication for the header reference.

Enable MCP tool search

Claude Code disables MCP tool search by default when ANTHROPIC_BASE_URL points to a custom host. After configuring the AI Gateway, enable tool search to load MCP tool definitions on demand:

Shell

export ENABLE_TOOL_SEARCH=true
unset CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS
claude

Keep experimental betas enabled; disabling them also disables tool search, even with ENABLE_TOOL_SEARCH=true. Use a compatible Claude model and check the Anthropic tool search configuration for model requirements. To persist the setting, merge this env setting into ~/.claude/settings.json, preserving existing entries, then start a new session:

JSON

{
  "env": {
    "ENABLE_TOOL_SEARCH": "true"
  }
}

Use true for on-demand loading. The auto setting loads tool definitions upfront until they reach 10% of the model’s context window, which is 100,000 tokens for a 1M-token model. Remove CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS from shell and settings configuration; contact the administrator if a managed policy sets it. In the new session, run /mcp to confirm the server is connected and /context to inspect MCP tool loading. Ask Claude Code to list the available Orq.ai models, then ask a follow-up that uses the result. Verify the tool search, actual MCP call, and follow-up all succeed; the context display alone does not verify tool execution. Deferred MCP tool loading and execution were validated with Claude Code 2.1.301 and Orq.ai responses-api:v4.16.4, using Claude Opus 5.5 through Anthropic and AWS Bedrock. Context savings depend on the configured tools. For Claude Desktop, use the separate Desktop tool-search setting.

Add request metadata

Tag requests with ANTHROPIC_CUSTOM_HEADERS. Separate headers with newlines and keep x-orq-api-key when using an Anthropic credential:

Shell

export ANTHROPIC_CUSTOM_HEADERS="x-orq-api-key: $ORQ_API_KEY"$'\nX-ORQ-METADATA-REPO: acme-api\nX-ORQ-METADATA-TICKET: PROJ-123'

See Request Metadata for the full header reference and precedence rules.

Claude Code requests routed through the AI Gateway appear in Traces.

See the Anthropic Messages API for endpoint details, authentication, and cache_control breakpoints.

Troubleshooting

High context usage before the first prompt

Follow Enable MCP tool search, then start a new session. Confirm ENABLE_TOOL_SEARCH=true, keep experimental betas enabled, and check that permissions do not deny ToolSearch. Inspect /context with the same model, MCP servers, Skills, and conversation when comparing usage. Tool search reduces MCP schema loading; system prompts, built-in tools, and conversation history also consume context.

400 invalid model format

The ANTHROPIC_MODEL value is missing the provider prefix. Use the provider/model_id format (e.g. anthropic/claude-sonnet-5, not claude-sonnet-5).

Claude Code calls Anthropic directly instead of Orq.ai

Set ANTHROPIC_BASE_URL to https://my.orq.ai/v3/anthropic in the terminal that launches Claude Code, then start a new session.

Authentication fails with an Anthropic credential

Keep the Orq.ai key in x-orq-api-key. Send the Anthropic credential through the selected setup above. When using a subscription, unset ANTHROPIC_AUTH_TOKEN and ANTHROPIC_API_KEY, then run /status to check the active Claude login.

Requests not appearing in Orq.ai Traces

Verify ANTHROPIC_BASE_URL is set to https://my.orq.ai/v3/anthropic. Requests sent to Anthropic’s own endpoint bypass the AI Gateway and produce no Traces.

Verification

Start a Claude Code session and send a prompt. The response appears in the terminal and the trace appears in Traces with the model identifier anthropic/claude-sonnet-5 (or whichever model ANTHROPIC_MODEL is set to).

Model Providers

Coding Agents

Chat Interfaces

Frameworks & SDKs

Automations

MCP Integration

AI Gateway Beta

MCP

Prerequisites

Installation

Verify Installation

Available Commands

Usage Examples

Create an Experiment

Query Trace Analytics

Create a Synthetic Dataset

Performance Analysis

Complete Experiment Creation

Trace Investigation

Troubleshooting

Plugins

Skills

Installation

MCP server only

Session tracing

Commands

Available Skills

AI Gateway

Use the workspace’s provider configuration

Use a Claude subscription

Use an Anthropic API key

Enable MCP tool search

Add request metadata

Troubleshooting

Verification

MCP Integration

AI Gateway Beta

​MCP

​Prerequisites

​Installation

​Verify Installation

​Available Commands

​Usage Examples

​Create an Experiment

​Query Trace Analytics

​Create a Synthetic Dataset

​Performance Analysis

​Complete Experiment Creation

​Trace Investigation

​Troubleshooting

​Plugins

​Skills

​Installation

​MCP server only

​Session tracing

​Commands

​Available Skills

​AI Gateway

​Use the workspace’s provider configuration

​Use a Claude subscription

​Use an Anthropic API key

​Enable MCP tool search

​Add request metadata

​Troubleshooting

​Verification

MCP

Prerequisites

Installation

Verify Installation

Available Commands

Usage Examples

Create an Experiment

Query Trace Analytics

Create a Synthetic Dataset

Performance Analysis

Complete Experiment Creation

Trace Investigation

Troubleshooting

Plugins

Skills

Installation

MCP server only

Session tracing

Commands

Available Skills

AI Gateway

Use the workspace’s provider configuration

Use a Claude subscription

Use an Anthropic API key

Enable MCP tool search

Add request metadata

Troubleshooting

Verification