Orq.ai Documentation - AI Gateway & LLM Collaboration Platform

Annotations are structured key-value pairs that capture human feedback on traces and spans in your observability data. They enable quality assessment, human review workflows, and training dataset curation from human-reviewed traces. Use Cases

Collecting quality feedback

Capture thumbs up/down ratings, custom scores, or categorical labels on AI responses. Build a feedback loop that surfaces low-quality generations for review.

Compliance and QA review

Flag responses with specific defects (hallucination, off-topic, inappropriate content) using structured annotation keys shared across your team.

Dataset curation

Annotate traces with corrections and quality labels, then export curated subsets as training datasets for future experiments.

Human-in-the-loop workflows

Route traces to annotation queues for systematic expert review. Combine with Trace Automations to automatically surface traces that meet specific criteria.

Concepts Three concepts work together to form the annotations system:

Human Review: defines the schema (key, value type, options) that annotations must conform to
Annotations: the actual feedback values applied to a trace or span
Annotation Queues: organized workflows for reviewing traces in bulk via the AI Studio

Human Review

Define annotation schemas: keys, value types, and validation rules. Available on all traces and spans in the project once created.

Annotations

Apply structured human feedback to traces and spans via the AI Studio or programmatically via the API.

Annotation Queues

Organize human review workflows. Filter and present relevant traces for review in bulk.

Create Human Review

Human Reviews define the structure and validation rules for annotations. Each annotation key must match an existing Human Review definition in the project.

AI Studio

To create a Human Review, head to Project Settings > Human Review and press the + button.

Three Human Review types are available:

Categorical: button options with custom labels, such as good/bad or saved/deleted
Range: a custom scoring slider, for example a scale from 0 to 100
Open field: free-form text input for detailed comments

Once created, a Human Review is available on all traces and spans in the project. No additional configuration or filtering required.

Common Annotation Types Legacy

Rating

Rate the overall quality of AI responses:

Rating	Description
good	The response was helpful and accurate.
bad	The response was unhelpful or inaccurate.

Defects

Identify specific issues with AI responses:

Defect	Description
grammatical	Responses that contain grammatical errors
spelling	Responses that contain spelling errors
hallucination	Responses that contain hallucinations or factual inaccuracies
repetition	Responses that contain unnecessary repetition
inappropriate	Responses that are deemed inappropriate or offensive
off_topic	Responses that do not address the user’s query
incompleteness	Responses that are incomplete or partially address the query
ambiguity	Responses that are vague or unclear

You can select multiple defects for one response by using an array-type Human Review.

Use Annotations

AI Studio
API & SDK

The annotation capabilities differ between Logs and Traces. Logs support both human feedback and corrections, while Traces only support human feedback annotations.

Traces
Logs

Navigate to the Traces view and select a single trace. The Annotations panel will be displayed, allowing you to apply human feedback to the AI response.

Here are examples on how to use the API to annotate LLM responses.

Add a Quality Rating

curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "annotations": [
      {
        "key": "rating",
        "value": "good"
      }
    ]
  }'

Add Multiple Defects

curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "annotations": [
      {
        "key": "defects",
        "value": ["grammatical", "spelling", "ambiguity"]
      }
    ]
  }'

Add a Numeric Score

curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "annotations": [
      {
        "key": "confidence_score",
        "value": 0.95
      }
    ],
    "metadata": {
      "identityId": "user-123"
    }
  }'

Add a Text Correction

curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "annotations": [
      {
        "key": "correction",
        "value": "The correct answer should emphasize scalability and fault tolerance."
      }
    ]
  }'

Batch Add Multiple Annotations

curl -X POST "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "annotations": [
      {
        "key": "rating",
        "value": "good"
      },
      {
        "key": "confidence_score",
        "value": 0.92
      },
      {
        "key": "categories",
        "value": ["helpful", "accurate", "concise"]
      }
    ]
  }'

Remove Annotations

curl -X DELETE "https://api.orq.ai/v2/traces/{trace_id}/spans/{span_id}/annotation" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "keys": ["rating", "defects"]
  }'

API Error Handling

Status Code	Error	Example Message	Solution
404	Human Review Not Found	`The human review with key "rating" for workspace abc123 was not found.`	Create a Human Review with the specified key before annotating.
404	Span Not Found	`Span with id xyz789 for workspace abc123 was not found.`	Verify the `trace_id` and `span_id` are correct and belong to your workspace.
400	Invalid Value	`Invalid value: poor. Valid options are: good, bad.`	Ensure the value matches the options defined in the Human Review.
400	Value Out of Range	`Value 15 is out of range [0, 10].`	Provide a number within the defined min/max range for the Human Review.
400	String Too Long	`String value exceeds maximum length of 200 characters.`	Shorten the string annotation to 200 characters or less.

See a complete feedback loop implemented from scratch. Read our cookbook Capturing User Feedback.

Constraints

Batch Limits: up to 10 annotations per create request, up to 10 keys per delete request
String Length: string values are limited to 200 characters maximum
Deployment Span Propagation: when annotating a deployment span, the associated log is automatically annotated with the same values
Metadata Fields: optional metadata object supports identityId, source, and reviewerId for tracking and attribution

Create Annotation Queues

Annotation Queues help you organize and apply Human Reviews effectively to relevant incoming traces.

AI Studio

To create an Annotation Queue, head to AI Studio > Annotation Queue.Choose Create Annotation Queue.The following fields are configurable:

The Name of the queue
The Description of the Annotation Queue
The Human Reviews that traces will be reviewed by

Annotation Queues can be filled with traces using Trace Automations, which automatically route traces to the appropriate queue based on your configured rules.

Use Annotation Queues

AI Studio

When opening an Annotation Queue, you have easy access to all traces that need feedback.

On the right side of the panel, review and apply feedback to the conversation. Use Next to access the next trace in the queue.

You can add any trace manually to a Dataset to use in a future Experiment, to ensure further testing on model behavior.

Get Started

AI Router

Observe

Build

Optimize

Manage Prompts

Add Human Reviews | Add Human feedback to generations

Human Review

Annotations

Annotation Queues

Create Human Review

Common Annotation Types Legacy

Use Annotations

Create Annotation Queues

Use Annotation Queues

Get Started

AI Router

Observe

Build

Optimize

Manage Prompts

Documentation Index

Human Review

Annotations

Annotation Queues

​Create Human Review

​Common Annotation Types Legacy

​Use Annotations

​Create Annotation Queues

​Use Annotation Queues

Create Human Review

Common Annotation Types Legacy

Use Annotations

Create Annotation Queues

Use Annotation Queues