Annotations API

The Annotations API attaches structured human review values to a completed request’s Trace and span. Where the Annotations page covers the AI Studio review workflows (the Annotations panel in Traces and Logs, Annotation Queues, and experiments), this page is the API usage guide: how to capture the identifiers, shape the request body, and handle the response.

Use Cases

Programmatic ratings

Record thumbs up/down or categorical ratings programmatically as users interact with responses, without opening AI Studio.

Defect logging

Log structured defect tags such as hallucination, off-topic, or incomplete at scale for systematic quality analysis.

Response corrections

Store a corrected response alongside the original to build gold-standard pairs for evaluation datasets.

Evaluator overrides

Correct an LLM-as-a-judge result with a human review, attaching an explanation to the corrected output.

Attach Annotations to a Span

Each annotation must be defined before it can be applied. The definition sets the key, title, and value type; every annotation submitted through the API has to match one of these definitions, and a key that does not exist in the workspace is rejected with a 404. Create definitions under Optimization > Annotations in AI Studio, or under Settings > Annotations in AI Gateway, as described on the Annotations page.

Annotation definition form with Key set to star-rating, Title set to Star rating 1-5, Type set to Range, and Minimum 1 and Maximum 5. — Defining a range annotation with the key star-rating.

With the definition in place, two steps:

Capture the trace_id and span_id of the run to annotate. Both are required, and where they come from depends on the endpoint; see Where the identifiers come from.
Submit the annotations with the Annotate a Span API, using the key of an existing annotation definition.

curl -X POST "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "annotations": [
      {
        "key": "star-rating",
        "value": 4
      }
    ]
  }'

import { Orq } from "@orq-ai/node";

const orq = new Orq({
  apiKey: process.env.ORQ_API_KEY,
});

const result = await orq.annotations.create({
  traceId: "<trace_id>",
  spanId: "<span_id>",
  requestBody: {
    annotations: [
      {
        key: "star-rating",
        value: 4,
      },
    ],
  },
});

console.log(result);

from orq_ai_sdk import Orq
import os

orq = Orq(api_key=os.getenv("ORQ_API_KEY"))

result = orq.annotations.create(
    trace_id="<trace_id>",
    span_id="<span_id>",
    annotations=[
        {
            "key": "star-rating",
            "value": 4,
        }
    ],
)

print(result)

Some API error messages still refer to annotations as human reviews. The two names describe the same object: annotations were previously called human reviews, and a message such as The human review with key "star-rating" ... was not found. refers to the annotation definition with that key.

Annotations are available on chat completion and Responses API spans. When annotating a deployment span, the associated log is automatically annotated with the same values.

Where the identifiers come from

Every run returns both ids. Where the response body contains a telemetry object, take them from there; otherwise take them from the response headers:

Endpoint	`trace_id`	`span_id`
`POST /v3/router/responses`	`telemetry.trace_id` in the body, or the `x-orq-trace-id` header	`telemetry.span_id` in the body, or the `x-orq-trace-span-id` header
Other `/v3/router/*` endpoints (chat completions, completions, embeddings, images, moderations, rerank, speech, transcriptions, translations, ocr)	`x-orq-trace-id` header	`x-orq-trace-span-id` header
`POST /v2/deployments/invoke`	`telemetry.trace_id` in the body	`telemetry.span_id` in the body

The telemetry object sits at the top level of the response:

{
  "telemetry": {
    "trace_id": "01M1H2VST506ZX2WE8KDQTRBEQ",
    "span_id": "01M1H2VST543YJ5DN5RXW1B6EC"
  }
}

On POST /v2/deployments/invoke, the x-orq-trace-id header does not contain the trace id. It contains the workflow run id, which identifies the run’s root span; x-orq-workflow-run-id and the response id field carry the same value. Use telemetry.trace_id from the response body as trace_id — passing the header value returns a 404.

Both ids are matched as opaque strings: the Annotations API performs an exact string match against the stored ids and applies no format validation. Their format is an implementation detail that can differ between runs, so do not validate their shape or assume a fixed length.

Annotations Fields

Fields marked with * are always required. key and parent_annotation_id are conditionally required: each entry carries exactly one of them.

Field	Location	Type	Description
`trace_id*`	path	string	The id of the trace the span belongs to, taken from the run’s response; see Where the identifiers come from
`span_id*`	path	string	The id of the span being annotated, taken from the run’s response; see Where the identifiers come from
`annotations*`	body	array	The annotations to apply, between 1 and 10 per request
`annotations[].key`	body	string	The key of an existing annotation definition. Required for a standard annotation, and not accepted on a correction
`annotations[].value*`	body	string, number, boolean, or array of strings	The annotation value; the type depends on the annotation definition (see below)
`annotations[].parent_annotation_id`	body	string	The eval id of the evaluator annotation being corrected. Required for a correction, and not accepted on a standard annotation
`annotations[].explanation`	body	string	Optional explanation for a correction, up to 3000 characters
`metadata.identity_id`	body	string	Optional identity of the reviewer for attribution; use `identityId` in the Node SDK

An annotation entry is either a standard annotation with key and value, or a correction with parent_annotation_id, value, and optionally explanation. The two shapes are mutually exclusive.

Annotation Values

The value type must match the annotation’s definition:

Categorical, single-select: a string or a single-element array, for example "good" or ["good"]
Categorical, multi-select: an array of strings, for example ["grammatical", "hallucination"]
Range: a number within the annotation’s configured min/max, for example 4 on a 1-5 range
Boolean: true or false
Text or correction: a string, up to 3000 characters

String values outside the annotation’s configured options are rejected; see Error Handling.

Corrections

A correction replaces an existing evaluator output with a human-reviewed value:

curl -X POST "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "annotations": [
      {
        "parent_annotation_id": "<evaluator_annotation_id>",
        "value": false,
        "explanation": "The response omitted a required disclaimer."
      }
    ]
  }'

import { Orq } from "@orq-ai/node";

const orq = new Orq({
  apiKey: process.env.ORQ_API_KEY,
});

const result = await orq.annotations.create({
  traceId: "<trace_id>",
  spanId: "<span_id>",
  requestBody: {
    annotations: [
      {
        parentAnnotationId: "<evaluator_annotation_id>",
        value: false,
        explanation: "The response omitted a required disclaimer.",
      },
    ],
  },
});

console.log(result);

from orq_ai_sdk import Orq
import os

orq = Orq(api_key=os.getenv("ORQ_API_KEY"))

result = orq.annotations.create(
    trace_id="<trace_id>",
    span_id="<span_id>",
    annotations=[
        {
            "parent_annotation_id": "<evaluator_annotation_id>",
            "value": False,
            "explanation": "The response omitted a required disclaimer.",
        }
    ],
)

print(result)

The corrected value must match the evaluator annotation’s own output type. See Correct an Evaluator Result for the UI equivalent.

Batch Annotations

Send up to 10 annotation entries in a single request:

curl -X POST "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "annotations": [
      {
        "key": "rating",
        "value": "good"
      },
      {
        "key": "star-rating",
        "value": 4
      },
      {
        "key": "categories",
        "value": ["helpful", "accurate", "concise"]
      }
    ]
  }'

import { Orq } from "@orq-ai/node";

const orq = new Orq({
  apiKey: process.env.ORQ_API_KEY,
});

const result = await orq.annotations.create({
  traceId: "<trace_id>",
  spanId: "<span_id>",
  requestBody: {
    annotations: [
      { key: "rating", value: "good" },
      { key: "star-rating", value: 4 },
      { key: "categories", value: ["helpful", "accurate", "concise"] },
    ],
  },
});

console.log(result);

from orq_ai_sdk import Orq
import os

orq = Orq(api_key=os.getenv("ORQ_API_KEY"))

result = orq.annotations.create(
    trace_id="<trace_id>",
    span_id="<span_id>",
    annotations=[
        {"key": "rating", "value": "good"},
        {"key": "star-rating", "value": 4},
        {"key": "categories", "value": ["helpful", "accurate", "concise"]},
    ],
)

print(result)

Remove Annotations

Submit keys to the Remove an Annotation from a Span API to delete standard annotations, or parent_annotation_ids to delete corrections by the eval ids of their parent annotations. Up to 10 of each per request. The examples below show keys; to delete a correction, replace the body with {"parent_annotation_ids": ["<evaluator_annotation_id>"]}.

curl -X DELETE "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "keys": ["rating", "defects"]
  }'

import { Orq } from "@orq-ai/node";

const orq = new Orq({
  apiKey: process.env.ORQ_API_KEY,
});

const result = await orq.annotations.delete({
  traceId: "<trace_id>",
  spanId: "<span_id>",
  requestBody: {
    keys: ["rating", "defects"],
  },
});

console.log(result);

from orq_ai_sdk import Orq
import os

orq = Orq(api_key=os.getenv("ORQ_API_KEY"))

result = orq.annotations.delete(
    trace_id="<trace_id>",
    span_id="<span_id>",
    keys=["rating", "defects"],
)

print(result)

Where Annotations Surface

Annotations applied through the API appear in the Annotations panel of the span and log detail views in Traces and Logs.

Error Handling

Status Code	Error	Example Message	Solution
404	Annotation Not Found	`The human review with key "star-rating" for workspace abc123 was not found.`	Create an Annotation with the specified key before annotating.
404	Span Not Found	`Span with id xyz789 and trace id trace456 for workspace abc123 was not found.`	Verify the `trace_id` and `span_id` are correct and belong to the workspace.
400	Invalid Value	`Invalid value: poor. Valid options are: good, bad.`	Ensure the value matches the options defined in the Annotation.
400	Value Out of Range	`Value 15 is out of range [0, 10].`	Provide a number within the defined min/max range for the Annotation.
400	String Too Long	Request validation error on `annotations[].value`	Keep string values within 3000 characters.

See a complete feedback loop implemented from scratch. Read our cookbook Capturing User Feedback.

Constraints

Batch limits: up to 10 annotations per create request, up to 10 keys and 10 parent_annotation_ids per delete request
Value length: string values are limited to 3000 characters; annotation definitions do not impose a tighter limit
Metadata fields: the optional metadata object supports identity_id (identityId in the Node SDK) for reviewer attribution

Overview

Instrument your app

Explore

Monitor

Export

Review

Use Cases

Attach Annotations to a Span

Where the identifiers come from

Annotations Fields

Annotation Values

Corrections

Batch Annotations

Remove Annotations

Where Annotations Surface

Error Handling

Constraints

See Also

​Use Cases

​Attach Annotations to a Span

​Where the identifiers come from

​Annotations Fields

​Annotation Values

​Corrections

​Batch Annotations

​Remove Annotations

​Where Annotations Surface

​Error Handling

​Constraints

​See Also

Use Cases

Attach Annotations to a Span

Where the identifiers come from

Annotations Fields

Annotation Values

Corrections

Batch Annotations

Remove Annotations

Where Annotations Surface

Error Handling

Constraints

See Also