> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Add Human Reviews | Add Human feedback to generations
> Add structured human feedback to LLM traces and spans. Capture quality assessments, corrections, and review AI responses via the AI Studio or API.
Annotations are structured key-value pairs that capture human feedback on traces and spans in your observability data. They enable quality assessment, human review workflows, and training dataset curation from human-reviewed traces.
**Use Cases**
Capture thumbs up/down ratings, custom scores, or categorical labels on AI responses. Build a feedback loop that surfaces low-quality generations for review.
Flag responses with specific defects (hallucination, off-topic, inappropriate content) using structured annotation keys shared across your team.
Annotate traces with corrections and quality labels, then export curated subsets as training datasets for future experiments.
Route traces to annotation queues for systematic expert review. Combine with Trace Automations to automatically surface traces that meet specific criteria.
**Concepts**
Three concepts work together to form the annotations system:
* **Human Review**: defines the schema (key, value type, options) that annotations must conform to
* **Annotations**: the actual feedback values applied to a trace or span
* **Annotation Queues**: organized workflows for reviewing traces in bulk via the AI Studio
Define annotation schemas: keys, value types, and validation rules. Available on all traces and spans in the project once created.
Apply structured human feedback to traces and spans via the AI Studio or programmatically via the API.
Organize human review workflows. Filter and present relevant traces for review in bulk.
## Create Human Review
Human Reviews define the structure and validation rules for annotations. Each annotation key must match an existing Human Review definition in the project.
To create a Human Review, head to **Project Settings > Human Review** and press the `+` button.
Three Human Review types are available:
* **Categorical**: button options with custom labels, such as good/bad or saved/deleted
* **Range**: a custom scoring slider, for example a scale from 0 to 100
* **Open field**: free-form text input for detailed comments
Once created, a Human Review is available on all traces and spans in the project. No additional configuration or filtering required.
### Common Annotation Types Legacy
Rate the overall quality of AI responses:
| Rating | Description |
| -------- | ----------------------------------------- |
| **good** | The response was helpful and accurate. |
| **bad** | The response was unhelpful or inaccurate. |
Identify specific issues with AI responses:
| Defect | Description |
| ------------------ | ------------------------------------------------------------- |
| **grammatical** | Responses that contain grammatical errors |
| **spelling** | Responses that contain spelling errors |
| **hallucination** | Responses that contain hallucinations or factual inaccuracies |
| **repetition** | Responses that contain unnecessary repetition |
| **inappropriate** | Responses that are deemed inappropriate or offensive |
| **off\_topic** | Responses that do not address the user's query |
| **incompleteness** | Responses that are incomplete or partially address the query |
| **ambiguity** | Responses that are vague or unclear |
You can select multiple defects for one response by using an array-type Human Review.
## Use Annotations
The annotation capabilities differ between [Logs](/docs/observability/logs) and [Traces](/docs/observability/traces). Logs support both human feedback and corrections, while Traces only support human feedback annotations.
Navigate to the [Traces](/docs/observability/traces) view and select a single trace. The **Annotations** panel will be displayed, allowing you to apply human feedback to the AI response.
Navigate to the [Logs](/docs/observability/logs) view and select a single log. The **Annotations** panel will be displayed, allowing you to apply human feedback and provide corrections to the AI response.
To make a correction, use the **Add correction** button below the AI-generated response:
Click to add a correction, which opens an editor where you can manually revise the model's response. Once you've finished editing, select **Save** to store the correction.
Corrections are valuable for building curated datasets. Learn more in [Creating a Curated Dataset](/docs/datasets/creating).
Here are examples on how to use the API to annotate LLM responses.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"key": "rating",
"value": "good"
}
]
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{
"key": "rating",
"value": "good"
}
]
)
print(result)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
annotations: [
{
key: "rating",
value: "good"
}
]
});
console.log(result);
```
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"key": "defects",
"value": ["grammatical", "spelling", "ambiguity"]
}
]
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{
"key": "defects",
"value": ["grammatical", "spelling", "ambiguity"]
}
]
)
print(result)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
annotations: [
{
key: "defects",
value: ["grammatical", "spelling", "ambiguity"]
}
]
});
console.log(result);
```
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"key": "confidence_score",
"value": 0.95
}
],
"metadata": {
"identityId": "user-123"
}
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{
"key": "confidence_score",
"value": 0.95
}
],
metadata={
"identityId": "user-123"
}
)
print(result)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
annotations: [
{
key: "confidence_score",
value: 0.95
}
],
metadata: {
identityId: "user-123"
}
});
console.log(result);
```
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"key": "correction",
"value": "The correct answer should emphasize scalability and fault tolerance."
}
]
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{
"key": "correction",
"value": "The correct answer should emphasize scalability and fault tolerance."
}
]
)
print(result)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
annotations: [
{
key: "correction",
value: "The correct answer should emphasize scalability and fault tolerance."
}
]
});
console.log(result);
```
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"annotations": [
{
"key": "rating",
"value": "good"
},
{
"key": "confidence_score",
"value": 0.92
},
{
"key": "categories",
"value": ["helpful", "accurate", "concise"]
}
]
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.create(
trace_id="",
span_id="",
annotations=[
{"key": "rating", "value": "good"},
{"key": "confidence_score", "value": 0.92},
{"key": "categories", "value": ["helpful", "accurate", "concise"]}
]
)
print(result)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.create({
traceId: "",
spanId: "",
annotations: [
{ key: "rating", value: "good" },
{ key: "confidence_score", value: 0.92 },
{ key: "categories", value: ["helpful", "accurate", "concise"] }
]
});
console.log(result);
```
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X DELETE "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"keys": ["rating", "defects"]
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
orq = Orq(api_key=os.getenv("ORQ_API_KEY"))
result = orq.annotations.delete(
trace_id="",
span_id="",
keys=["rating", "defects"]
)
print(result)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({
apiKey: process.env.ORQ_API_KEY,
});
const result = await orq.annotations.delete({
traceId: "",
spanId: "",
keys: ["rating", "defects"]
});
console.log(result);
```
**API Error Handling**
| Status Code | Error | Example Message | Solution |
| ----------- | ---------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------- |
| **404** | Human Review Not Found | `The human review with key "rating" for workspace abc123 was not found.` | Create a Human Review with the specified key before annotating. |
| **404** | Span Not Found | `Span with id xyz789 for workspace abc123 was not found.` | Verify the `trace_id` and `span_id` are correct and belong to your workspace. |
| **400** | Invalid Value | `Invalid value: poor. Valid options are: good, bad.` | Ensure the value matches the options defined in the Human Review. |
| **400** | Value Out of Range | `Value 15 is out of range [0, 10].` | Provide a number within the defined min/max range for the Human Review. |
| **400** | String Too Long | `String value exceeds maximum length of 200 characters.` | Shorten the string annotation to 200 characters or less. |
See a complete feedback loop implemented from scratch. Read our cookbook [Capturing User Feedback](/docs/tutorials/capturing-feedback-with-orq).
**Constraints**
* **Batch Limits**: up to 10 annotations per create request, up to 10 keys per delete request
* **String Length**: string values are limited to 200 characters maximum
* **Deployment Span Propagation**: when annotating a deployment span, the associated log is automatically annotated with the same values
* **Metadata Fields**: optional `metadata` object supports `identityId`, `source`, and `reviewerId` for tracking and attribution
## Create Annotation Queues
Annotation Queues help you organize and apply Human Reviews effectively to relevant incoming traces.
To create an Annotation Queue, head to **AI Studio > Annotation Queue**.
Choose **Create Annotation Queue**.
The following fields are configurable:
* The **Name** of the queue
* The **Description** of the Annotation Queue
* The [**Human Reviews**](/docs/projects/human-review) that traces will be reviewed by
Annotation Queues can be filled with traces using [Trace Automations](/docs/observability/trace-automation), which automatically route traces to the appropriate queue based on your configured rules.
## Use Annotation Queues
When opening an Annotation Queue, you have easy access to all traces that need feedback.
On the right side of the panel, review and apply feedback to the conversation. Use **Next** to access the next trace in the queue.
You can add any trace manually to a [Dataset](/docs/datasets/overview) to use in a future [Experiment](/docs/experiments/overview), to ensure further testing on model behavior.