Skip to main content
Annotation Queues organize Traces into review workflows so Annotations can be applied in bulk in AI Studio. To define annotations and apply them individually, through the API and SDK, or the CLI, see Annotations.

​
Create Annotation Queues

Annotation Queues help you organize and apply annotations effectively to relevant incoming Traces.
To create an Annotation Queue, head to AI Studio > Annotation Queue.Choose Create Annotation Queue.The following fields are configurable:
  • The Name of the queue
  • The Description of the Annotation Queue
  • The Annotations that Traces will be reviewed by
Create Annotation Queue panel with fields for name, description, and annotations, showing Defects, Interactions, and Rating tags selected.

Create Annotation Queue panel showing name, description, and Annotations fields.

​
Fill Annotation Queues

Once a queue exists, fill it with the Traces to review. Traces can be added automatically or manually.
Use Trace Automations to route Traces into a queue based on configured rules. Add an Add to Annotation Queue action to an automation and select the target queue. As matching Traces arrive, they are added to the queue without manual effort, which keeps a steady stream of relevant Traces ready for review.
Edit Automation panel with a metadata filter on request_id, an Add to Annotation Queue action selecting the fireflies_annotation queue, and an Apply Evaluator action marked Coming soon.

An automation with an Add to Annotation Queue action routing matching Traces into a queue.

​
Use Annotation Queues

Open an Annotation Queue to step through its Traces one at a time in the review screen.
Annotation Queue review screen showing Item 7 of 43 in the header, a left panel with Inputs, Metrics (Latency, Cost, tokens), and Task (Model claude-haiku-4-5, Provider anthropic), a center panel with the System instructions, User input, and Assistant output, and a right Annotations panel with a comment field and a rating with good and bad buttons. A dataset selector and Add to dataset button sit at the bottom.

Annotation Queue review screen. Left: Inputs, Metrics, and Task. Center: the full interaction. Right: the Annotations panel.

The screen is divided into three panels:
  • Left: details for the selected Trace.
    • Inputs: the variables mapped to inputs, when configured.
    • Metrics: latency, cost, and token usage.
    • Task: the model, provider, and other configuration parameters.
    The header shows the current position, the total number of items in the queue, and how many have already been reviewed.
  • Center: the full interaction for the selected Trace.
  • Right: the Annotations panel with the Annotations configured for the queue, such as a rating with categorical buttons or an open comment field. Selecting a value saves immediately and marks the Trace as reviewed.
Navigate between items with K (previous) and J (next), or use the up and down buttons at the top left. When a data point is worth reusing, select Add to dataset to send the Trace to a Dataset for use in a future Experiment.
Adding a Trace to a Dataset does not copy its annotations for now. The annotation values stay on the originating Trace, where they remain queryable via the Orq MCP.
Evaluator results shown in the review screen can also be corrected, not just annotated.
Evaluators panel listing agent_jailbreak_detection and agent_response_relevance, both marked No, showing the pencil icon used to open the correction popover.

Correcting an Evaluator result in the review screen.

See Correct an Evaluator Result on the Traces page for the full flow.