> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace evaluations

> Score production traces automatically by attaching evaluators to Deployments and Agents.

Trace evaluations run [Evaluators](/docs/ai-studio/optimize/evaluators) on production traffic passing through a [Deployment](/docs/ai-studio/ai-engineering/deployments) or [Agent](/docs/ai-studio/ai-engineering/build-agents). Results appear as spans in the [Traces](/docs/ai-studio/observability/traces) hierarchy, providing continuous quality monitoring without manual review.

## How trace evaluations work

An Evaluator attached to a Deployment or Agent runs on sampled requests. Input Evaluators score the prompt before the model generates; Output Evaluators score the response after generation. Both run **asynchronously** and never block the caller.

This differs from [Guardrails](/docs/ai-studio/ai-engineering/deployments#evaluators-and-guardrails), which run **synchronously** and can deny a generation if the evaluation fails.

<Frame caption="Guardrails execute synchronously and can block a generation, while Evaluators run asynchronously and never block the response.">
  <img src="https://mintcdn.com/orqai/E8L3R46ivX7g9-QI/images/docs/a6be2fe5b5e1290b3d2132212a9ec6e74287d7b6cb896d86f8f4838b5a9bcf73-Guardrails_and_Evaluators_-_Deployment.png?fit=max&auto=format&n=E8L3R46ivX7g9-QI&q=85&s=1c40c78aac37b6819e2ab2bfceaba12e" alt="Flow diagram showing a user query passing through Input Guardrails synchronously, then Deployment Model Generation, then Output Guardrails, with Input and Output Evaluators running asynchronously and fail paths returning an Error Response." width="3278" height="1779" data-path="images/docs/a6be2fe5b5e1290b3d2132212a9ec6e74287d7b6cb896d86f8f4838b5a9bcf73-Guardrails_and_Evaluators_-_Deployment.png" />
</Frame>

<Note>
  Evaluators do not run when using the Test panel in AI Studio. Invoke the Deployment via the [API or SDK](/docs/ai-studio/ai-engineering/deployments#invoke-a-deployment) to trigger evaluators.
</Note>

## Attach evaluators

<Tabs>
  <Tab title="Agent" icon="robot">
    In the Agent configuration, add Evaluators as input or output evaluators.

    * **Input evaluator**: evaluates the input before the model generates a response.
    * **Output evaluator**: evaluates the output after generation.

    Set a **Sample Rate** (0-100%) to control what percentage of matching traces trigger evaluation. A 100% sample rate evaluates every request; a 10% rate evaluates one in ten.

    See [Configure Evaluators and Guardrails](/docs/ai-studio/ai-engineering/build-agents#configure-evaluators-and-guardrails) for the full guide.
  </Tab>

  <Tab title="Deployment" icon="rocket">
    Evaluators attach to [Deployments](/docs/ai-studio/ai-engineering/deployments#evaluators-and-guardrails) with the same input/output stage and sample rate configuration as Agents.
  </Tab>

  <Tab title="Guardrail Rules" icon="shield-halved">
    For conditional evaluation, use [Guardrail Rules](/docs/ai-gateway/configuration/guardrail-rules) to define CEL-based conditions that determine when evaluators run. Evaluation can be triggered based on metadata, identity, model, or other request attributes.
  </Tab>
</Tabs>

## View evaluator results

Evaluator results attach to individual spans within a trace. Each result includes the Evaluator name, the evaluation score or verdict, and the evaluation stage (`input` or `output`).

In the [Traces](/docs/ai-studio/observability/traces) section, evaluator results appear in the trace hierarchy alongside LLM calls, tool invocations, and other spans. Use the **Evals** filter to narrow traces by evaluator name, score, or pass/fail status.

If an evaluator result needs correction, hover the result on a span and select the edit icon. Set the corrected value and optionally add an explanation. Corrections are written back to the span and are queryable via the [**Orq MCP**](/docs/ai-studio/integrations/code-assistants/orq-mcp).

## Create an Evaluator

Browse the [Hub](/docs/ai-studio/optimize/hub) for pre-built evaluators or create a custom one:

* **LLM Evaluator**: uses a model to judge outputs against criteria defined in a prompt.
* **Python Evaluator**: runs custom Python code for deterministic scoring.

See [Create Evaluators](/docs/ai-studio/optimize/evaluators) for the full guide.

<CardGroup cols={2}>
  <Card title="Create Evaluators" icon="flask" href="/docs/ai-studio/optimize/evaluators">
    Build LLM-as-a-Judge and Python evaluators for automatic scoring.
  </Card>

  <Card title="Trace Automations" icon="robot" href="/docs/ai-studio/observability/automations">
    Automatically route traces with low evaluator scores to Annotation Queues.
  </Card>
</CardGroup>