Skip to main content

​
What Are Trace Automations

Trace Automations provide a way to build datasets, trigger actions, and streamline monitoring directly from trace activity.
Each automation consists of:
  • Filters — define which traces to target (for example, by status, tags, or feedback score)
  • Sampling rate — choose what percentage of matching traces should trigger actions
  • Actions — define what happens when a trace meets the conditions
This makes it easy to automatically collect, review, and evaluate traces as your application scales.
Trace automation interface

Configuring your automation

​
Why Use Automations

As your LLM applications grow, the number of traces generated can quickly become too large to review manually.
Automations ensure that your most important traces are captured and acted on without manual work — whether for quality assurance, retraining, or debugging. You can use automations to:
  • Automatically add traces to a dataset for future evaluation or fine-tuning
  • Send traces with low feedback scores to an annotation queue for human review
  • Trigger webhooks to launch remote evaluations or downstream processes
  • Extend data retention for traces with specific outcomes
  • Sample a percentage of traces for random quality checks

​
Setting Up Trace Automations

To create a new automation:
  1. Navigate to the Traces section in the Orq.ai Studio.
  2. Select the Automations tab at the top.
  3. Click Create Automation.
  4. Define your Filter, Sampling Rate, and Action.
  5. Save and enable your automation.
Once active, every new trace that matches your defined filters will trigger the configured action automatically.