> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. # Annotation Queues > Organize production traces into annotation queues in AI Studio to review LLM outputs and apply human feedback annotations in bulk. Annotation Queues organize Traces into review workflows so [Annotations](/ai-studio/observability/annotations) can be applied in bulk in **AI Studio**. To define annotations and apply them individually, through the API and SDK, or the CLI, see [Annotations](/ai-studio/observability/annotations). ## Create Annotation Queues Annotation Queues help you organize and apply annotations effectively to relevant incoming Traces. To create an Annotation Queue, head to **AI Studio > Annotation Queue**. Choose **Create Annotation Queue**. The following fields are configurable: * The **Name** of the queue * The **Description** of the Annotation Queue * The [**Annotations**](/ai-studio/observability/annotations#create-annotations) that Traces will be reviewed by Create Annotation Queue panel with fields for name, description, and annotations, showing Defects, Interactions, and Rating tags selected. ## Fill Annotation Queues Once a queue exists, fill it with the Traces to review. Traces can be added automatically or manually. Use [Trace Automations](/ai-studio/observability/automations) to route Traces into a queue based on configured rules. Add an **Add to Annotation Queue** action to an automation and select the target queue. As matching Traces arrive, they are added to the queue without manual effort, which keeps a steady stream of relevant Traces ready for review. Edit Automation panel with a metadata filter on request_id, an Add to Annotation Queue action selecting the fireflies_annotation queue, and an Apply Evaluator action marked Coming soon. Open a Trace, select a span, and choose **Add to annotation queue** to send it to a queue one at a time. This is useful for ad hoc review of specific Traces that are worth a closer look. Trace detail view with a selected claude-haiku span showing Add to dataset, Add to annotation queue, and Try Prompt buttons, with Add to annotation queue highlighted. ## Use Annotation Queues Open an Annotation Queue to step through its Traces one at a time in the review screen. Annotation Queue review screen showing Item 7 of 43 in the header, a left panel with Inputs, Metrics (Latency, Cost, tokens), and Task (Model claude-haiku-4-5, Provider anthropic), a center panel with the System instructions, User input, and Assistant output, and a right Annotations panel with a comment field and a rating with good and bad buttons. A dataset selector and Add to dataset button sit at the bottom. The screen is divided into three panels: * **Left**: details for the selected Trace. * **Inputs**: the variables mapped to inputs, when configured. * **Metrics**: latency, cost, and token usage. * **Task**: the model, provider, and other configuration parameters. The header shows the current position, the total number of items in the queue, and how many have already been reviewed. * **Center**: the full interaction for the selected Trace. * **Right**: the **Annotations** panel with the [**Annotations**](/ai-studio/observability/annotations#create-annotations) configured for the queue, such as a rating with categorical buttons or an open comment field. Selecting a value saves immediately and marks the Trace as reviewed. Navigate between items with **`K`** (previous) and **`J`** (next), or use the up and down buttons at the top left. When a data point is worth reusing, select **Add to dataset** to send the Trace to a [Dataset](/ai-studio/optimize/datasets) for use in a future [Experiment](/ai-studio/optimize/experiments). Adding a Trace to a Dataset does not copy its annotations for now. The annotation values stay on the originating Trace, where they remain queryable via the **Orq MCP**. Evaluator results shown in the review screen can also be corrected, not just annotated. Evaluators panel listing agent_jailbreak_detection and agent_response_relevance, both marked No, showing the pencil icon used to open the correction popover. See [Correct an Evaluator Result](/ai-studio/observability/traces#correct-an-evaluator-result) on the Traces page for the full flow.