> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. # Add an eval column > Links an existing evaluator. Each row gets one evaluator cell per task column. Config is updated at the evaluator source, not here. > [!NOTE] > > **Related guide**: Experiments guide. See the [Experiments guide](https://docs.orq.ai/ai-studio/optimize/experiments.md) for a walkthrough with examples. ## OpenAPI ````yaml post /v3/experiments/{experiment_id}/runs/{run_id}/evaluators openapi: 3.1.0 info: title: orq.ai API version: '2.0' description: orq.ai API documentation servers: - url: https://my.orq.ai security: - ApiKey: [] tags: - name: Chunking description: Split text into smaller chunks for retrieval and generation workflows. - name: File Systems description: >- Create and manage persistent file systems that agents and MCP clients read from and write to. - name: Knowledge Bases description: Create and manage knowledge bases used by agents and retrieval workflows. - name: Memory Stores description: Create and manage memory stores, memories, and memory documents. - name: Evals description: Run an evaluator against a conversation and its result - name: Logs description: >- OpenTelemetry log query API. Search, filter, aggregate, and facet log records ingested via OTLP. - name: Reporting description: >- GenAI reporting API over canonical analytics rollups. Accepts a metric name, time range, grain, group-by, and filters; returns a typed time series and optional totals. - name: Traces description: >- Query and inspect ingested trace data: search trace summaries, aggregate metrics, and read individual traces and their spans. - name: Experiments description: >- Create and run experiments: dataset rows against task columns, evaluators, and human review. - description: List models available through the AI Router. name: Models - name: Alerts description: >- Alerts evaluate a Reporting API metric on a fixed interval and fire notifications through notifiers when the value breaches a threshold. Each breach opens a trigger that tracks the incident until the value recovers. - name: Annotation Queues description: Annotation queues collect spans for human review. - name: API keys description: >- API keys authenticate programmatic access to the workspace. They expose opaque tokens, per-domain access grants, and budget and rate-limit constraints. - name: Audit Logs description: Audit logs record workspace entity changes and access-relevant events. - name: Budgets description: >- Budgets govern spend, token usage, and request rate across six scopes: workspace, project, identity, API key, provider, and model. Every applicable budget is enforced, and the most restrictive limit applies per dimension. - name: Files description: File upload and retrieval operations. - name: Guardrail Rules description: >- Guardrail Rules conditionally enforce evaluators and plugins for AI Gateway traffic. Rules may be scoped to a project or the whole workspace. - name: Hub description: Hub items are reusable templates available to a workspace. - name: Identities description: >- Identities represent end users from your system for usage and engagement tracking. - name: Management keys description: >- Management keys are workspace-scoped credentials that authenticate programmatic access to workspace administration surfaces (API keys, budgets). Unlike project-scoped API keys, a management key always operates at the workspace level. - name: MCP Gateway description: >- Register upstream MCP servers, discover and sync their tools, and assemble gateways that expose a curated tool surface to MCP clients. - name: Model Catalog description: >- Browse the orq.ai model catalog: every model orq offers, across every provider, with pricing, capabilities and benchmark data. List endpoints only return models that are not deprecated. This API is public, requires no authentication, and is rate limited to 120 requests per minute per IP. Responses carry a 5-minute cache-control max-age. - name: Model Fusions description: >- Model Fusions combine independent model answers through a panel, judge, and optional synthesizer. - name: Monitors description: >- Monitors are custom observability dashboards for traces, runtime metrics, and logs. Each monitor holds widgets that chart one telemetry query. List the metrics, operations, and dimensions available to your workspace with `GET /v3/telemetry/capabilities`, or start from a preset with `GET /v2/monitors/presets`. - name: Notifiers description: Notifier destinations used to send delivery and workflow notifications. - name: Projects description: Projects organize resources within a workspace - name: Routing Rules description: >- Routing Rules conditionally select models and enforce request plugins for AI Gateway traffic. Rules are evaluated by ascending priority and may be scoped to a project or the whole workspace. - name: Threads description: Threads group related trace invocations and their aggregate usage - name: Skills description: >- Skills are modular instructions you can use to codify processes and conventions - name: Smart Routers description: >- Create and manage workspace Smart Routers. A Smart Router selects a model from an eligible pool for each request according to a quality, balanced, or cost profile. - name: Webhooks description: >- Create and manage webhooks that deliver workspace events to external HTTPS endpoints. - name: Workspaces description: >- A workspace is the tenant. Create is called from a user session during onboarding; Get, List, and Update are the public management surface. - name: Workspace Security description: >- Workspace-level domain verification and IP allowlist controls. These operations are restricted to workspace administrators. - name: Workspace Settings description: >- Workspace-level settings managed with a workspace credential. A workspace is the tenant, so these settings are a singleton — there is nothing to create or delete, only read and update. - name: Responses - description: Run agents on a cron cadence. Minimum firing interval is 1 hour. name: Agent Schedules - name: Embeddings - name: Telemetry description: >- Unified query envelope for traces, metrics, and logs. One request shape, one filter dialect, and one response shape per source, validated by a per-source registry. - description: Beta. Run typed classification questions against a classify model. name: Classify - description: Search Gateway with managed credits or BYOK. name: Web Search - description: Beta. Run classification decisions with the classify contract. name: Decisions - name: Users description: >- Workspace user administration. Requires a Management Key with user permissions or an authenticated workspace admin session. Standard API keys are not supported. externalDocs: url: https://docs.orq.ai description: orq.ai Documentation paths: /v3/experiments/{experiment_id}/runs/{run_id}/evaluators: post: tags: - Experiments summary: Add an eval column description: >- Links an existing evaluator. Each row gets one evaluator cell per task column. Config is updated at the evaluator source, not here. operationId: ExperimentEvalCreate parameters: - name: experiment_id in: path description: ID of the experiment. required: true schema: type: string - name: run_id in: path description: ID of the run. required: true schema: type: string requestBody: content: application/json: schema: $ref: '#/components/schemas/CreateEvalColumnRequest' required: true responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/Column' '401': description: >- Authentication required. See https://docs.orq.ai/reference/errors for the error codes. content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' x-code-samples: - lang: curl label: Core - Add an evaluator column source: > curl 'https://my.orq.ai/v3/experiments//runs//evaluators' \ --header 'Authorization: Bearer $ORQ_API_KEY' \ --header 'Content-Type: application/json' \ --data '{ "evaluator_id": "" }' - lang: python label: Python - Add an evaluator column source: | import os from orq_ai_sdk import Orq client = Orq(api_key=os.environ["ORQ_API_KEY"]) column = client.experiments.create_eval( experiment_id="", run_id="", evaluator_id="", ) - lang: typescript label: Node.js - Add an evaluator column source: | import { Orq } from '@orq-ai/node'; const client = new Orq({ apiKey: process.env.ORQ_API_KEY }); await client.experiments.createEval({ experimentId: '', runId: '', evaluatorId: '', }); components: schemas: CreateEvalColumnRequest: required: - evaluator_id type: object properties: evaluator_id: type: string description: >- ID of the evaluator to add. Each row gets one evaluator cell per task column. position: type: integer description: 0-based position among the run's columns. Defaults to the end. format: int32 Column: type: object properties: column_id: type: string description: >- Column ID. Dataset, prompt, evaluator and review columns keep their ID in new runs; agent columns get a new one. key: type: string description: |- Column key, derived from the display name: lowercased, with spaces replaced by underscores. display_name: type: string description: Column name. position: type: integer description: >- 0-based position in the run, after the columns are grouped as dataset, task, job output, metric, evaluator and review columns. format: int32 active: type: boolean description: |- Whether the column is shown in the AI Studio grid. False for hidden columns, such as the token metric columns. prompt: allOf: - $ref: '#/components/schemas/PromptTaskConfig' description: Set on prompt task columns. agent: allOf: - $ref: '#/components/schemas/AgentTaskConfig' description: Set on agent task columns. openresponse: allOf: - $ref: '#/components/schemas/OpenResponseTaskConfig' description: Set on task columns whose cells are filled outside orq.ai. evaluator: allOf: - $ref: '#/components/schemas/EvalColumnConfig' description: Set on evaluator columns. review: allOf: - $ref: '#/components/schemas/ReviewColumnConfig' description: Set on human review columns. dataset: allOf: - $ref: '#/components/schemas/DatasetColumnConfig' description: 'Set on the dataset columns: inputs, messages and expected output.' metric: allOf: - $ref: '#/components/schemas/MetricColumnConfig' description: 'Set on metric columns: cost, latency and token counts.' job_output: allOf: - $ref: '#/components/schemas/JobOutputColumnConfig' description: Set on evaluatorq job output columns. width: type: integer description: |- Column width in the AI Studio grid. Task columns default to 280; evaluator, review and metric columns to 200. 0 when unset. format: int32 description: |- A column of a run. Exactly one of the config fields is set, matching the column type. ErrorResponse: type: object properties: message: description: Human-readable error message. type: string code: type: integer description: Numeric error code, when one is available. details: type: array items: {} description: Additional error details, when the service reports any. required: - message PromptTaskConfig: required: - prompt_config type: object properties: prompt_config: type: object additionalProperties: true description: |- Prompt to execute. Execution reads `model` (required), `provider`, `messages` and `model_parameters` such as `temperature`, `max_tokens` and `response_format`. messages_source: $ref: '#/components/schemas/MessagesSource' description: >- Messages sent to the model. `MESSAGES_SOURCE_PROMPT_CONFIG`: the prompt's messages only. `MESSAGES_SOURCE_MESSAGES_COLUMN`: the row's dataset messages only. `MESSAGES_SOURCE_COMBINED`: the prompt's messages followed by the row's messages. Defaults to `MESSAGES_SOURCE_PROMPT_CONFIG`. engine: type: string description: >- Template engine for variables in the messages: `text` (default), `jinja` or `mustache`. tools: type: array items: type: object additionalProperties: true description: Tools saved with the column. knowledge_bases: type: array items: type: object additionalProperties: true description: |- Knowledge bases searched on each execution, as `{knowledge_id, configuration}` with a configuration `type` of `last_user_message` or `query`. The retrieved context is available as `{{knowledge_retrieval}}`; when the prompt does not use it, it is appended to the system message. metadata: type: - object - 'null' additionalProperties: true description: >- Metadata saved with the column. Null when the column was saved without metadata. description: Configuration of a prompt task column. AgentTaskConfig: type: object properties: agent_key: type: string description: Key of the agent to run. Required. messages_source: $ref: '#/components/schemas/MessagesSource' description: >- Messages sent to the agent. `MESSAGES_SOURCE_MESSAGES_COLUMN`: the row's dataset messages. Any other value: `messages` from this config. System messages are dropped, since the agent's instructions provide them. messages: type: array items: type: object additionalProperties: true description: >- Conversation sent to the agent when `messages_source` is not the messages column. engine: type: string description: >- Template engine for variables in the messages: `text` (default), `jinja` or `mustache`. parent_id: type: string description: Saved with the column; not used during execution. agent_tag: type: string description: Agent version tag. When set, the column runs that tagged version. description: Configuration of an agent task column. OpenResponseTaskConfig: type: object properties: {} description: |- A task column whose cells are filled outside orq.ai, such as by an evaluatorq upload. It is never executed. EvalColumnConfig: type: object properties: evaluator_id: type: string description: ID of the evaluator. evaluator_name: type: string description: Evaluator display name. evaluator_type: type: string description: Evaluator type, such as `llm_eval` or `python_eval`. evaluator_output_type: type: string description: Type of value the evaluator returns, such as `number`. description: |- Configuration of an evaluator column. Name and types are read from the evaluator. ReviewColumnConfig: type: object properties: human_eval_id: type: string description: ID of the human review definition. human_eval_name: type: string description: Review name; the same as the column's display name. human_eval_type: type: string description: >- Value type the review accepts: `range`, `single_select`, `multi_select`, `text` or `correction`. Empty accepts any value. description: Configuration of a human review column. DatasetColumnConfig: type: object properties: column_type: type: string description: >- `inputs`, `messages` or `expected_output`. Dataset cells are read-only. description: Configuration of a dataset column. MetricColumnConfig: type: object properties: metric_key: type: string description: >- Metric shown: `cost`, `latency`, `ttft`, `input_tokens`, `output_tokens` or `reasoning_tokens`. description: Configuration of a metric column. JobOutputColumnConfig: type: object properties: job_name: type: string description: Name of the evaluatorq job. description: >- Output of one evaluatorq job; the column holds the text each job produced. MessagesSource: type: string enum: - MESSAGES_SOURCE_UNSPECIFIED - MESSAGES_SOURCE_PROMPT_CONFIG - MESSAGES_SOURCE_MESSAGES_COLUMN - MESSAGES_SOURCE_COMBINED securitySchemes: ApiKey: type: http scheme: bearer bearerFormat: JWT ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.