> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Annotations API
> Apply structured human review values to Traces and spans programmatically via the Annotations API and SDK, from ratings and defect tags to corrections and evaluator overrides.
The **Annotations API** attaches structured human review values to a completed request's Trace and span. Where the [Annotations](/docs/ai-studio/observability/annotations) page covers the **AI Studio** review workflows (the **Annotations** panel in [Traces](/docs/ai-studio/observability/traces) and [Logs](/docs/ai-studio/observability/logs), [Annotation Queues](/docs/ai-studio/observability/annotation-queues), and experiments), this page is the API usage guide: how to capture the identifiers, shape the request body, and handle the response.
## Use Cases
Record thumbs up/down or categorical ratings programmatically as users interact with responses, without opening **AI Studio**.
Log structured defect tags such as hallucination, off-topic, or incomplete at scale for systematic quality analysis.
Store a corrected response alongside the original to build gold-standard pairs for evaluation datasets.
Correct an LLM-as-a-judge result with a human review, attaching an explanation to the corrected output.
## Attach Annotations to a Span
Each annotation must be defined before it can be applied. The definition sets the key, title, and value type; every annotation submitted through the API has to match one of these definitions, and a key that does not exist in the workspace is rejected with a **404**. Create definitions under **Optimization > Annotations** in **AI Studio**, or under **Settings > Annotations** in **AI Gateway**, as described on the [Annotations](/docs/ai-studio/observability/annotations) page.
With the definition in place, two steps:
1. Capture the `trace_id` and `span_id` from the completion response. The completion response returns telemetry that includes both identifiers for the request being annotated.
2. Submit the annotations with the [Annotate a Span API](/reference/annotations/annotate-a-span), using the `key` of an existing annotation definition.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"key": "star-rating",
"value": 4
}
]
}'
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
requestBody: {
annotations: [
{
key: "star-rating",
value: 4,
},
],
},
});
console.log(result);
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{
"key": "star-rating",
"value": 4,
}
],
)
print(result)
```
Some API error messages still refer to annotations as **human reviews**. The two names describe the same object: annotations were previously called human reviews, and a message such as `The human review with key "star-rating" ... was not found.` refers to the annotation definition with that key.
Annotations are available on chat completion and Responses API spans. When annotating a deployment span, the associated log is automatically annotated with the same values.
## Annotations Fields
Fields marked with `*` are always required. `key` and `parent_annotation_id` are conditionally required: each entry carries exactly one of them.
| Field | Location | Type | Description |
| ------------------------------------ | -------- | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `trace_id*` | path | string | The id of the trace the span belongs to, captured from the completion response |
| `span_id*` | path | string | The id of the span being annotated, captured from the completion response |
| `annotations*` | body | array | The annotations to apply, between 1 and 10 per request |
| `annotations[].key` | body | string | The key of an existing annotation definition. Required for a standard annotation, and not accepted on a correction |
| `annotations[].value*` | body | string, number, boolean, or array of strings | The annotation value; the type depends on the annotation definition (see below) |
| `annotations[].parent_annotation_id` | body | string | The eval id of the evaluator annotation being corrected. Required for a correction, and not accepted on a standard annotation |
| `annotations[].explanation` | body | string | Optional explanation for a correction, up to 3000 characters |
| `metadata.identity_id` | body | string | Optional identity of the reviewer for attribution; use `identityId` in the Node SDK |
An annotation entry is either a **standard annotation** with `key` and `value`, or a **correction** with `parent_annotation_id`, `value`, and optionally `explanation`. The two shapes are mutually exclusive.
## Annotation Values
The `value` type must match the annotation's definition:
* **Categorical, single-select**: a string or a single-element array, for example `"good"` or `["good"]`
* **Categorical, multi-select**: an array of strings, for example `["grammatical", "hallucination"]`
* **Range**: a number within the annotation's configured min/max, for example `4` on a 1-5 range
* **Boolean**: `true` or `false`
* **Text or correction**: a string, up to 3000 characters
String values outside the annotation's configured options are rejected; see [Error Handling](#error-handling).
## Corrections
A correction replaces an existing evaluator output with a human-reviewed value:
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"parent_annotation_id": "",
"value": false,
"explanation": "The response omitted a required disclaimer."
}
]
}'
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
requestBody: {
annotations: [
{
parentAnnotationId: "",
value: false,
explanation: "The response omitted a required disclaimer.",
},
],
},
});
console.log(result);
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{
"parent_annotation_id": "",
"value": False,
"explanation": "The response omitted a required disclaimer.",
}
],
)
print(result)
```
The corrected `value` must match the evaluator annotation's own output type. See [Correct an Evaluator Result](/docs/ai-studio/observability/traces#correct-an-evaluator-result) for the UI equivalent.
## Batch Annotations
Send up to 10 annotation entries in a single request:
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"key": "rating",
"value": "good"
},
{
"key": "star-rating",
"value": 4
},
{
"key": "categories",
"value": ["helpful", "accurate", "concise"]
}
]
}'
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
requestBody: {
annotations: [
{ key: "rating", value: "good" },
{ key: "star-rating", value: 4 },
{ key: "categories", value: ["helpful", "accurate", "concise"] },
],
},
});
console.log(result);
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{"key": "rating", "value": "good"},
{"key": "star-rating", "value": 4},
{"key": "categories", "value": ["helpful", "accurate", "concise"]},
],
)
print(result)
```
## Remove Annotations
Submit `keys` to the [Remove an Annotation from a Span API](/reference/annotations/remove-an-annotation-from-a-span) to delete standard annotations, or `parent_annotation_ids` to delete corrections by the eval ids of their parent annotations. Up to 10 of each per request.
The examples below show `keys`; to delete a correction, replace the body with `{"parent_annotation_ids": [""]}`.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X DELETE "https://my.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"keys": ["rating", "defects"]
}'
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.delete({
traceId: "",
spanId: "",
requestBody: {
keys: ["rating", "defects"],
},
});
console.log(result);
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.delete(
trace_id="",
span_id="",
keys=["rating", "defects"],
)
print(result)
```
## Where Annotations Surface
Annotations applied through the API appear in the **Annotations** panel of the span and log detail views in [Traces](/docs/ai-studio/observability/traces) and [Logs](/docs/ai-studio/observability/logs).
## Error Handling
| Status Code | Error | Example Message | Solution |
| ----------- | -------------------- | ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| **404** | Annotation Not Found | `The human review with key "star-rating" for workspace abc123 was not found.` | Create an Annotation with the specified key before annotating. |
| **404** | Span Not Found | `Span with id xyz789 and trace id trace456 for workspace abc123 was not found.` | Verify the `trace_id` and `span_id` are correct and belong to the workspace. |
| **400** | Invalid Value | `Invalid value: poor. Valid options are: good, bad.` | Ensure the value matches the options defined in the Annotation. |
| **400** | Value Out of Range | `Value 15 is out of range [0, 10].` | Provide a number within the defined min/max range for the Annotation. |
| **400** | String Too Long | Request validation error on `annotations[].value` | Keep string values within 3000 characters. |
See a complete feedback loop implemented from scratch. Read our cookbook [Capturing User Feedback](/docs/ai-studio/cookbooks/integrations-tooling/capturing-feedback-with-orq).
## Constraints
* **Batch limits**: up to 10 annotations per create request, up to 10 keys and 10 `parent_annotation_ids` per delete request
* **Value length**: string values are limited to 3000 characters; annotation definitions do not impose a tighter limit
* **Metadata fields**: the optional `metadata` object supports `identity_id` (`identityId` in the Node SDK) for reviewer attribution
## See Also
* [Annotations](/docs/ai-studio/observability/annotations): the **AI Studio** review workflows
* [Annotation Queues](/docs/ai-studio/observability/annotation-queues): bulk review workflows
* [Annotations SDK Reference](/reference/sdk/annotations): SDK method signatures
* [CLI Reference](/reference/cli): annotate with `orq traces create` and `orq traces delete`