LLM traces for debugging

What are Traces

With Traces, dive into the workflow of each model generation and understand the inner workings of an LLM call on orq.ai. Traces correspond to events within the generations. Events within a Deployment can be various:

Evaluators & Guardrails
Retrieval using Knowledge Base
- Embeddings
- Chunking
Caching
Retries and Fallbacks

Creating Traces

Traces are automatically generated when using Orq.ai Deployments. Create custom traces for application code using framework instrumentation or the Orq.ai SDK.

Framework Instrumentation

For popular AI frameworks and libraries, use automatic instrumentation with OpenTelemetry:

Framework Integrations

Explore automatic instrumentation for OpenAI, LangChain, LlamaIndex, CrewAI, Autogen, and 15+ other frameworks with OpenTelemetry integration.

Custom Tracing using the @traced decorator

For custom functions or application workflows, use the @traced decorator from the Python SDK. It works with both synchronous and async functions:

import os
from orq_ai_sdk import Orq
from orq_ai_sdk.traced import traced

@traced(name="process_document", type="agent")
def process_document(doc_id: str):
    result = {"status": "processed", "doc_id": doc_id}
    return result

@traced(name="fetch_context", type="retrieval")
def fetch_relevant_context(query: str):
    return {"context": "relevant information", "sources": ["doc1", "doc2"]}

@traced(name="format_response", type="function")
def format_response(raw_output: str, metadata: dict):
    return {"formatted": raw_output.strip(), "metadata": metadata}

@traced(name="document_pipeline", type="agent")
def document_pipeline(orq, doc_id: str):
    doc = process_document(doc_id)
    context = fetch_relevant_context("user query")

    response = orq.responses.create(
        model="agent/simple-agent",
        input=f"Summarize this document: {doc['status']}. Context: {context['context']}",
    )

    return format_response(response.output[0]["content"][0]["text"], {"doc_id": doc_id})

with Orq(api_key=os.getenv("ORQ_API_KEY")) as orq:
    result = document_pipeline(orq, "doc-123")
    print(result)

import os
import asyncio
from orq_ai_sdk import Orq
from orq_ai_sdk.traced import traced

@traced(name="process_document", type="agent")
async def process_document(doc_id: str):
    result = {"status": "processed", "doc_id": doc_id}
    return result

@traced(name="fetch_context", type="retrieval")
async def fetch_relevant_context(query: str):
    return {"context": "relevant information", "sources": ["doc1", "doc2"]}

@traced(name="format_response", type="function")
async def format_response(raw_output: str, metadata: dict):
    return {"formatted": raw_output.strip(), "metadata": metadata}

@traced(name="document_pipeline", type="agent")
async def document_pipeline(orq, doc_id: str):
    doc = await process_document(doc_id)
    context = await fetch_relevant_context("user query")

    response = await orq.responses.create_async(
        model="agent/simple-agent",
        input=f"Summarize this document: {doc['status']}. Context: {context['context']}",
    )

    return await format_response(response.output[0]["content"][0]["text"], {"doc_id": doc_id})

async def main():
    async with Orq(api_key=os.getenv("ORQ_API_KEY")) as orq:
        result = await document_pipeline(orq, "doc-123")
        print(result)

asyncio.run(main())

The result is a single trace with each decorated function appearing as a nested span:

Trace view showing a document_pipeline root span with process_document, fetch_context, and format_response nested as child spans, each with latency and token usage. — Custom trace with nested spans from @traced.

The @traced decorator supports the following span types:

Type	Description
`agent`	A high-level orchestration step or agent workflow
`embedding`	An embedding generation operation
`function`	A general-purpose function or processing step
`llm`	A direct LLM API call
`retrieval`	A retrieval or knowledge lookup operation
`tool`	An external tool call

Capture or suppress inputs/outputs with capture_input and capture_output, and attach custom metadata with the attributes parameter:

@traced(
    name="process_document",
    type="agent",
    capture_input=False,
    capture_output=False,
    attributes={"version": "1.0", "env": "production"}
)
def process_document(doc_id: str):
    ...

How to Lookup Traces

To find traces, head to the AI Studio and to the Traces section.

Explore traces for each action taken during a generation:

The left column shows the list of Traces available.
By selecting a trace, the middle column opens, showing the hierarchy of events shown in order of execution.
By selecting a single trace, the right column opens, showing details for the specific step.

Trace Views

Each trace can be inspected in three views to suit different debugging and analysis needs.

Trace
Thread
Timeline

The Trace view shows the full execution tree for a single run. Each step is displayed hierarchically, including LLM calls, tool invocations, knowledge retrievals, and memory interactions. Use this view to inspect inputs, outputs, token usage, and latency at every step.For multi-agent runs, the hierarchy renders as an agent graph: parent agents and their sub-agent calls are shown as a nested tree, making it easy to follow how work was delegated across agents and where time was spent.

Trace view in Orq.ai showing a multi-agent call graph: product-orchestrator delegates to agent-compositor, which calls agent-response, call_sub_agent, find_documents, and router-cache-agent. The right panel shows span properties, input, and output for the selected step.

Erroring Traces are shown with a icon.

Manually evaluate responses using Human Reviews directly on individual spans. Human Reviews defined in the project are available on all spans automatically. Learn more in Human Review.

Viewing Errors

Traces that encountered an error are marked with a red error status badge in the list. Use the Status filter to scope the list to errors only and identify failing traces quickly.

Traces list filtered to show only erroring traces, with red error badges on each row and a Status filter chip applied. — Traces list filtered to error status.

The Status filter combines with any other filter, for example:

Filter by Status: error and an Identity to see all errors for a specific user.
Combine with Project or Metadata to narrow down failures to a particular environment or deployment.

Filtering Traces

Click next to the search bar to open the filter menu. Select a category to expand a searchable list of values and check one or more to apply.

Trace filters — Trace filter categories.

When working with Agents, access traces directly from the Agent page with automatic filtering for that specific agent.

The following filter categories are available:

Filter	Description
Annotations	Filter by annotation labels applied to traces.
Evals	Filter by evaluator results attached to traces.
Identity	Filter by the identity that triggered the trace.
Metadata	Filter by metadata fields attached to traces.
Model	Filter by the model used in the trace.
Name	Filter by trace name.
Project	Filter by the project the trace belongs to.
Provider	Filter by the model provider.
Span ID	Filter by a specific span identifier.
Status	Filter by the status of the trace (e.g. error, success).
Thread ID	Filter by the thread the trace belongs to.
Total Cost	Filter by the total cost of the trace.
Total Tokens	Filter by the total token count of the trace.
Trace ID	Filter by a specific trace identifier.

Multiple filters can be combined. To remove a filter, click next to it in the filter bar.

Configuring Column

Columns can be enabled and toggled to display more available data in the traces list.

The above fields are displayable on the traces list.

Creating Custom Views

Save frequently used filter combinations as reusable views:

Set the desired filters
Click All Rows (top right)
Select Create New View
Enter a title for the view
Optionally check Set view private (default is shared with project members)
The filtered view is saved and accessible from the All Rows dropdown

A custom view filtering for all Errors is available by default.

Traces view dropdown showing a default Errors view alongside any custom saved views. — Available views

Review Traces

To generate a review queue on the fly, first apply one or more filters to scope the list. Once at least one filter is active, the Review button in the toolbar becomes available. Click Review to open the review screen and step through all matching traces one by one.

Traces toolbar with Reload and Review buttons. The Review button is highlighted. — The review screen for Traces. Apply filters first to enable the Review button.

The screen follows the same three-panel layout as Annotation Queues:

Left: Inputs and Metrics for the selected span.
Center: Full trace conversation.
Right: Human Review controls for manual annotation.

Annotations applied here are written back to the span and are queryable via the Orq MCP.

Threads

Visualize conversation history as Threads to follow the full sequence of messages across an agent session.

Reference

Span Attributes

Complete reference for all orq.* span attributes emitted on traces, webhook payloads, and trace exports.

Get Started

AI Gateway

AI Observability

AI Engineering

AI Governance

AI Chat

Organization

What are Traces

Creating Traces

Framework Instrumentation

Framework Integrations

Custom Tracing using the @traced decorator

How to Lookup Traces

Trace Views

Viewing Errors

Filtering Traces

Configuring Column

Creating Custom Views

Review Traces

Threads

Threads

Reference

Span Attributes

​What are Traces

​Creating Traces

​Framework Instrumentation

Framework Integrations

​Custom Tracing using the @traced decorator

​How to Lookup Traces

​Trace Views

​Viewing Errors

​Filtering Traces

​Configuring Column

​Creating Custom Views

​Review Traces

​Threads

Threads

​Reference

Span Attributes

What are Traces

Creating Traces

Framework Instrumentation

Custom Tracing using the @traced decorator

How to Lookup Traces

Trace Views

Viewing Errors

Filtering Traces

Configuring Column

Creating Custom Views

Review Traces

Threads

Reference