> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Annotations API
> Apply structured feedback to traces and spans via the Annotations API. Add ratings, defect tags, corrections, and evaluator overrides programmatically.
The **Annotations API** attaches structured human review values to a completed request's Trace and span. Where the [Annotations](/ai-studio/observability/annotations) page covers the **AI Studio** review workflows (the **Annotations** panel in [Traces](/ai-studio/observability/traces) and [Logs](/ai-studio/observability/logs), [Annotation Queues](/ai-studio/observability/annotation-queues), and experiments), this page is the API usage guide: how to capture the identifiers, shape the request body, and handle the response.
## Use Cases
Record thumbs up/down or categorical ratings programmatically as users interact with responses, without opening **AI Studio**.
Log structured defect tags such as hallucination, off-topic, or incomplete at scale for systematic quality analysis.
Store a corrected response alongside the original to build gold-standard pairs for evaluation datasets.
Correct an LLM-as-a-judge result with a human review, attaching an explanation to the corrected output.
## Attach Annotations to a Span
Each annotation must be defined before it can be applied. The definition sets the key, title, and value type; every annotation submitted through the API has to match one of these definitions, and a key that does not exist in the workspace is rejected with a **404**. Create definitions under **Optimization > Annotations** in **AI Studio**, or under **Settings > Annotations** in **AI Gateway**, as described on the [Annotations](/ai-studio/observability/annotations) page.
With the definition in place, two steps:
1. Capture the `trace_id` and `span_id` of the run to annotate. Both are required, and where they come from depends on the endpoint; see [Where the identifiers come from](#where-the-identifiers-come-from).
2. Submit the annotations with the [Annotate a Span API](/reference/annotations/annotate-a-span), using the `key` of an existing annotation definition.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"key": "star-rating",
"value": 4
}
]
}'
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
requestBody: {
annotations: [
{
key: "star-rating",
value: 4,
},
],
},
});
console.log(result);
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{
"key": "star-rating",
"value": 4,
}
],
)
print(result)
```
Some API error messages still refer to annotations as **human reviews**. The two names describe the same object: annotations were previously called human reviews, and a message such as `The human review with key "star-rating" ... was not found.` refers to the annotation definition with that key.
Annotations are available on chat completion and Responses API spans. When annotating a deployment span, the associated log is automatically annotated with the same values.
## Where the identifiers come from
Every run returns both ids. Where the response body contains a `telemetry` object, take them from there; otherwise take them from the response headers:
| Endpoint | `trace_id` | `span_id` |
| -------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------- |
| `POST /v3/router/responses` | `telemetry.trace_id` in the body, or the `x-orq-trace-id` header | `telemetry.span_id` in the body, or the `x-orq-trace-span-id` header |
| Other `/v3/router/*` endpoints (chat completions, completions, embeddings, images, moderations, rerank, speech, transcriptions, translations, ocr) | `x-orq-trace-id` header | `x-orq-trace-span-id` header |
| `POST /v2/deployments/invoke` | `telemetry.trace_id` in the body | `telemetry.span_id` in the body |
The `telemetry` object sits at the top level of the response:
```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
"telemetry": {
"trace_id": "01M1H2VST506ZX2WE8KDQTRBEQ",
"span_id": "01M1H2VST543YJ5DN5RXW1B6EC"
}
}
```
On `POST /v2/deployments/invoke`, the `x-orq-trace-id` header does **not** contain the trace id. It contains the workflow run id, which identifies the run's root span; `x-orq-workflow-run-id` and the response `id` field carry the same value. Use `telemetry.trace_id` from the response body as `trace_id` — passing the header value returns a 404.
Both ids are matched as opaque strings: the Annotations API performs an exact string match against the stored ids and applies no format validation. Their format is an implementation detail that can differ between runs, so do not validate their shape or assume a fixed length.
## Annotations Fields
Fields marked with `*` are always required. `key` and `parent_annotation_id` are conditionally required: each entry carries exactly one of them.
| Field | Location | Type | Description |
| ------------------------------------ | -------- | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `trace_id*` | path | string | The id of the trace the span belongs to, taken from the run's response; see [Where the identifiers come from](#where-the-identifiers-come-from) |
| `span_id*` | path | string | The id of the span being annotated, taken from the run's response; see [Where the identifiers come from](#where-the-identifiers-come-from) |
| `annotations*` | body | array | The annotations to apply, between 1 and 10 per request |
| `annotations[].key` | body | string | The key of an existing annotation definition. Required for a standard annotation, and not accepted on a correction |
| `annotations[].value*` | body | string, number, boolean, or array of strings | The annotation value; the type depends on the annotation definition (see below) |
| `annotations[].parent_annotation_id` | body | string | The eval id of the evaluator annotation being corrected. Required for a correction, and not accepted on a standard annotation |
| `annotations[].explanation` | body | string | Optional explanation for a correction, up to 3000 characters |
| `metadata.identity_id` | body | string | Optional identity of the reviewer for attribution; use `identityId` in the Node SDK |
An annotation entry is either a **standard annotation** with `key` and `value`, or a **correction** with `parent_annotation_id`, `value`, and optionally `explanation`. The two shapes are mutually exclusive.
## Annotation Values
The `value` type must match the annotation's definition:
* **Categorical, single-select**: a string or a single-element array, for example `"good"` or `["good"]`
* **Categorical, multi-select**: an array of strings, for example `["grammatical", "hallucination"]`
* **Range**: a number within the annotation's configured min/max, for example `4` on a 1-5 range
* **Boolean**: `true` or `false`
* **Text or correction**: a string, up to 3000 characters
String values outside the annotation's configured options are rejected; see [Error Handling](#error-handling).
## Corrections
A correction replaces an existing evaluator output with a human-reviewed value:
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"parent_annotation_id": "",
"value": false,
"explanation": "The response omitted a required disclaimer."
}
]
}'
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
requestBody: {
annotations: [
{
parentAnnotationId: "",
value: false,
explanation: "The response omitted a required disclaimer.",
},
],
},
});
console.log(result);
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{
"parent_annotation_id": "",
"value": False,
"explanation": "The response omitted a required disclaimer.",
}
],
)
print(result)
```
The corrected `value` must match the evaluator annotation's own output type. See [Correct an Evaluator Result](/ai-studio/observability/traces#correct-an-evaluator-result) for the UI equivalent.
## Batch Annotations
Send up to 10 annotation entries in a single request:
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"key": "rating",
"value": "good"
},
{
"key": "star-rating",
"value": 4
},
{
"key": "categories",
"value": ["helpful", "accurate", "concise"]
}
]
}'
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
requestBody: {
annotations: [
{ key: "rating", value: "good" },
{ key: "star-rating", value: 4 },
{ key: "categories", value: ["helpful", "accurate", "concise"] },
],
},
});
console.log(result);
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{"key": "rating", "value": "good"},
{"key": "star-rating", "value": 4},
{"key": "categories", "value": ["helpful", "accurate", "concise"]},
],
)
print(result)
```
## Remove Annotations
Submit `keys` to the [Remove an Annotation from a Span API](/reference/annotations/remove-an-annotation-from-a-span) to delete standard annotations, or `parent_annotation_ids` to delete corrections by the eval ids of their parent annotations. Up to 10 of each per request.
The examples below show `keys`; to delete a correction, replace the body with `{"parent_annotation_ids": [""]}`.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X DELETE "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"keys": ["rating", "defects"]
}'
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.delete({
traceId: "",
spanId: "",
requestBody: {
keys: ["rating", "defects"],
},
});
console.log(result);
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.delete(
trace_id="",
span_id="",
keys=["rating", "defects"],
)
print(result)
```
## Where Annotations Surface
Annotations applied through the API appear in the **Annotations** panel of the span and log detail views in [Traces](/ai-studio/observability/traces) and [Logs](/ai-studio/observability/logs).
## Error Handling
| Status Code | Error | Example Message | Solution |
| ----------- | -------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **404** | Annotation Not Found | `The human review with key "star-rating" for workspace abc123 was not found.` | Create an Annotation with the specified key before annotating. |
| **404** | Span Not Found | `Span with id xyz789 and trace id trace456 for workspace abc123 was not found.` | Verify the `trace_id` and `span_id` are correct and belong to the workspace. If the ids are correct and the request follows the generating call immediately, retry with a brief backoff: span ingestion is asynchronous and may not have completed yet. |
| **400** | Invalid Value | `Invalid value: poor. Valid options are: good, bad.` | Ensure the value matches the options defined in the Annotation. |
| **400** | Value Out of Range | `Value 15 is out of range [0, 10].` | Provide a number within the defined min/max range for the Annotation. |
| **400** | String Too Long | Request validation error on `annotations[].value` | Keep string values within 3000 characters. |
See a complete feedback loop implemented from scratch. Read our cookbook [Capturing User Feedback](/ai-studio/cookbooks/integrations-tooling/capturing-feedback-with-orq).
## Constraints
* **Batch limits**: up to 10 annotations per create request, up to 10 keys and 10 `parent_annotation_ids` per delete request
* **Value length**: string values are limited to 3000 characters; annotation definitions do not impose a tighter limit
* **Metadata fields**: the optional `metadata` object supports `identity_id` (`identityId` in the Node SDK) for reviewer attribution
## See Also
* [Annotations](/ai-studio/observability/annotations): the **AI Studio** review workflows
* [Annotation Queues](/ai-studio/observability/annotation-queues): bulk review workflows
* [Annotations SDK Reference](/reference/sdk/annotations): SDK method signatures
* [CLI Reference](/reference/cli): annotate with `orq traces create` and `orq traces delete`