Orq.ai Documentation - AI Gateway & LLM Collaboration Platform

What is the Orq MCP?

The Orq Model Context Protocol (MCP) server provides AI code assistants with direct access to your Orq.ai workspace. With 24 specialized tools, you can manage experiments, create datasets, configure evaluators, and analyze traces without leaving your IDE.

Quickstart

Point your assistant at the MCP server and authenticate with your API key:


Endpoint	`https://my.orq.ai/v2/mcp`
Auth Header	`Authorization: Bearer YOUR_ORQ_API_KEY`

For assistant-specific setup, select your assistant in the Code Assistants section below.

Key Capabilities

Agent Creation

Create and configure agents with custom instructions, tools, models, evaluators, and guardrails directly from conversation

Experiment Management

Create and run experiments, configure task columns with prompts or agents, and export results in multiple formats

Dataset Operations

Create synthetic datasets, reshape local data, manage datapoints, and map data to experiments

Analytics & Insights

Query workspace analytics, track performance metrics, and ask natural language questions about your traces

Evaluator & Guardrail Configuration

Create LLM-as-a-Judge evaluators, Python code evaluators, and attach guardrails for automated quality assessment and runtime safety

Available Tools

The Orq MCP provides 24 tools across 9 categories:

Category	Tool	Description
Agents	get_agent	Retrieve agent configuration and details
Agents	create_agent	Create a new agent with instructions, tools, models, evaluators, and guardrails
Agents	update_agent	Update an existing agent’s configuration (instructions, model, tools, evaluators, guardrails)
Analytics	get_analytics_overview	Get workspace snapshot (requests, cost, tokens, errors, latency, top models)
Analytics	query_analytics	Flexible drill-down with filtering and grouping
Dataset	create_dataset	Create a new dataset
Dataset	list_datapoints	List datapoints in a dataset
Dataset	create_datapoints	Create datapoints (max 100)
Dataset	update_datapoint	Update a datapoint
Dataset	delete_datapoints	Delete datapoints (max 100)
Dataset	delete_dataset	Delete a dataset and all datapoints
Evaluator	create_llm_eval	Create LLM-as-a-Judge evaluator
Evaluator	create_python_eval	Create Python code evaluator
Experiment	list_experiment_runs	List runs with cursor pagination
Experiment	get_experiment_run	Export run (JSON/JSONL/CSV)
Experiment	create_experiment	Create experiment from dataset with optional auto-run
Models	list_models	List all available AI models
Registry	list_registry_keys	List available attribute keys for filtering traces
Registry	list_registry_values	List top values for a specific attribute
Search	search_entities	Search projects, datasets, prompts, or experiments
Search	search_directories	List directories within a project
Traces	list_traces	List traces with filtering and sorting
Traces	get_span	Retrieve a single span (compact or full mode)
Traces	list_spans	List all spans in a trace

Examples

Investigating Traces

Find errors from the last 24 hours

Show me all traces with errors from the last 24 hours

The assistant will:

Calculate the unix timestamp for 24 hours ago
Use list_traces with filter status:=ERROR && timestamp:>TIMESTAMP and sort by timestamp:desc
Display trace IDs, names, durations, and timestamps
Summarize the most common error types and their frequency

Detect regressions after a model switch

After switching models yesterday, has latency increased or stabilized?

The assistant will:

Use query_analytics with metric: "latency" and group_by: ["model"] for the period before the switch
Repeat for the period after the switch
Compare average latency per model across both windows and surface any regressions

Find the slowest traces

Find the 5 slowest traces from today and show me their span details

The assistant will:

Use list_traces sorted by duration_ms:desc, filtered to today, limit 5
Use list_spans with each trace_id to retrieve the full span tree
Surface bottlenecks and latency outliers

Running Experiments

Compare two models on an existing dataset

Create an experiment comparing GPT-5.2 and Claude Sonnet 4.6 using the "user-queries" dataset

The assistant will:

Search for the “user-queries” dataset using search_entities
Use create_experiment with two model configurations and auto_run enabled
Return the experiment ID once both configurations have run

Compare two prompt strategies

Create an experiment using the "customer-feedback" dataset with two prompts: one focused on empathy and one on brevity. Run it and summarize the results.

The assistant will:

Search for the dataset using search_entities
Use create_experiment with two prompt variants and auto_run enabled
Use get_experiment_run to retrieve evaluation metrics
Compare the variants and summarize which performed better

Export experiment results

Export the latest experiment run as CSV

The assistant will:

Use list_experiment_runs to find the most recent run
Use get_experiment_run with CSV export format
Return a signed download URL for the CSV file

Managing Datasets

Create a synthetic dataset

Generate 50 realistic customer support questions about a SaaS product and create a dataset called "Support Training Data"

The assistant will:

Generate 50 synthetic question/answer pairs
Use create_dataset to create the dataset
Use create_datapoints to add all entries in bulk, each formatted as { inputs: { question: "..." }, expected_output: "..." }

Import data from code

Create a dataset from the JSON array above and add it to my workspace

The assistant will:

Parse the JSON from your selection or context
Use create_dataset with an appropriate name
Use create_datapoints to add each entry as a datapoint

Update or clean up a dataset

Delete all datapoints in the "staging-tests" dataset that have an empty expected_output field

The assistant will:

Use search_entities to find the “staging-tests” dataset and retrieve its ID
Use list_datapoints to retrieve all entries
Filter for datapoints with empty expected_output
Use delete_datapoints to remove them in batches

Creating Evaluators

Create an LLM-as-a-Judge evaluator

Create an LLM-as-a-Judge evaluator that scores responses on tone: professional, neutral, or aggressive

The assistant will:

Use create_llm_eval with a scoring rubric for tone classification
Confirm the evaluator ID and configuration

Create a Python evaluator

Create a Python evaluator that checks whether the response contains a valid JSON object

The assistant will:

Write a Python snippet that parses the response and validates JSON structure
Use create_python_eval to register it in your workspace

Create an experiment with evaluators

Create an experiment from the "qa-dataset" dataset with the "tone-scorer" evaluator attached

The assistant will:

Search for the dataset using search_entities
Use the evaluator ID — copy it from the Orq.ai UI, or use the ID returned by create_llm_eval / create_python_eval if created in the same session
Use create_experiment with both the dataset ID and evaluator ID, with auto_run enabled

Analytics

Get a workspace snapshot

Give me an overview of my workspace metrics for the last 7 days

The assistant will:

Use get_analytics_overview with a 7-day range
Return total requests, cost, tokens, error rate, latency, and top models

Drill into a specific model’s performance

How has gpt-5.2 performed this week? Focus on error rate and cost.

The assistant will:

Use query_analytics with metric: "errors", filtered by model and a 7-day range
Use query_analytics with metric: "cost", filtered by model and a 7-day range
Surface error rate trends and cost breakdown side by side

Identify your most expensive models

Which models are costing the most this month?

The assistant will:

Use query_analytics with metric: "cost", group_by: ["model"], and a 30-day range
Aggregate cost per model across all time buckets and rank them by total spend