What Are Trace Automations
Trace Automations provide a way to build datasets, trigger actions, and streamline monitoring directly from trace activity.Each automation consists of:
- Filters — define which traces to target (for example, by status, tags, or feedback score)
- Sampling rate — choose what percentage of matching traces should trigger actions
- Actions — define what happens when a trace meets the conditions
Configuring your automation
Why Use Automations
As your LLM applications grow, the number of traces generated can quickly become too large to review manually. Automations ensure that your most important traces are captured and acted on without manual work — whether for quality assurance, retraining, or debugging. You can use automations to:- Automatically add traces to a dataset for future evaluation or fine-tuning
- Send traces with low feedback scores to an annotation queue for human review
- Trigger webhooks to launch remote evaluations or downstream processes
- Extend data retention for traces with specific outcomes
- Sample a percentage of traces for random quality checks
Setting Up Trace Automations
To create a new automation:1
Navigate to Traces
Navigate to the Traces section in the Orq.ai Studio.
2
Create Automation
Select the Automations tab at the top.Click Create Automation.
3
Set Filters
Define the Filters for the Traces.Matching Traces will execute the desired Action.
4
Set Sampling Rate
Define the Sampling Rate: choose how many matching Traces will execute the Action
5
Define Action
Choose from the following action:
- Add to Dataset.
- Add to Annotation Queue.
6
Enable Automation
Once saved, your automation will only run on future Traces coming into the Platform.