Skip to main content
Trace evaluations run Evaluators on production traffic passing through a Deployment or Agent. Results appear as spans in the Traces hierarchy, providing continuous quality monitoring without manual review.

​
How trace evaluations work

An Evaluator attached to a Deployment or Agent runs on sampled requests. Input Evaluators score the prompt before the model generates; Output Evaluators score the response after generation. Both run asynchronously and never block the caller. This differs from Guardrails, which run synchronously and can deny a generation if the evaluation fails.
Flow diagram showing a user query passing through Input Guardrails synchronously, then Deployment Model Generation, then Output Guardrails, with Input and Output Evaluators running asynchronously and fail paths returning an Error Response.

Guardrails execute synchronously and can block a generation, while Evaluators run asynchronously and never block the response.

Evaluators do not run when using the Test panel in AI Studio. Invoke the Deployment via the API or SDK to trigger evaluators.

​
Attach evaluators

In the Agent configuration, add Evaluators as input or output evaluators.
  • Input evaluator: evaluates the input before the model generates a response.
  • Output evaluator: evaluates the output after generation.
Set a Sample Rate (0-100%) to control what percentage of matching traces trigger evaluation. A 100% sample rate evaluates every request; a 10% rate evaluates one in ten.See Configure Evaluators and Guardrails for the full guide.

​
View evaluator results

Evaluator results attach to individual spans within a trace. Each result includes the Evaluator name, the evaluation score or verdict, and the evaluation stage (input or output). In the Traces section, evaluator results appear in the trace hierarchy alongside LLM calls, tool invocations, and other spans. Use the Evals filter to narrow traces by evaluator name, score, or pass/fail status. If an evaluator result needs correction, hover the result on a span and select the edit icon. Set the corrected value and optionally add an explanation. Corrections are written back to the span and are queryable via the Orq MCP.

​
Create an Evaluator

Browse the Hub for pre-built evaluators or create a custom one:
  • LLM Evaluator: uses a model to judge outputs against criteria defined in a prompt.
  • Python Evaluator: runs custom Python code for deterministic scoring.
See Create Evaluators for the full guide.

Create Evaluators

Build LLM-as-a-Judge and Python evaluators for automatic scoring.

Trace Automations

Automatically route traces with low evaluator scores to Annotation Queues.