Agent Simulation

Agent Simulation drives the agent under test through realistic multi-turn conversations without writing test transcripts by hand. evaluatorq generates the personas, scenarios, and opening messages, then scores each conversation with an LLM judge. Three LLMs work together in every simulation:

Target agent: the system under test, either a hosted Orq.ai agent (target="agent:<key>") or any async function
User simulator: plays a persona pursuing a scenario goal, turn by turn
Judge: scores whether the goal was met and whether any rules were broken

Agent Simulation is available in the Python version of evaluatorq only. The TypeScript version does not support it.

This page covers running simulations and reading the results in Orq.ai. For the full library reference, provider resolution, and the complete example set, see the evaluatorq agent simulation guide.

Prerequisites

Python 3.10 or later
Install the package with the simulation extras:

uv add "evaluatorq[simulation]"

python -m pip install "evaluatorq[simulation]"

Set the API key. When targeting an Orq.ai agent, ORQ_API_KEY is the only key needed: the simulator, the judge, and the generators route through the Orq.ai router.

export ORQ_API_KEY=orq-...

First simulation

The fastest start is generate_and_simulate(): it synthesizes the personas, scenarios, and opening messages from a short description of the agent. num_personas × num_scenarios conversations run in parallel.

Python

import asyncio

from evaluatorq.simulation import generate_and_simulate

async def main():
    results = await generate_and_simulate(
        evaluation_name="support-agent-sim",
        target="agent:my-support-agent",       # hosted Orq agent, key from AI Studio
        agent_description=(
            "Customer support agent for an e-commerce store; "
            "handles refunds, orders, and product questions."
        ),
        num_personas=3,
        num_scenarios=4,                       # 12 conversations in total
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
    )

    passed = sum(r.goal_achieved for r in results)
    print(f"Pass rate: {passed}/{len(results)}")

if __name__ == "__main__":
    asyncio.run(main())

The result is a list of SimulationResult objects, one per conversation. evaluator_names selects the scorers applied to each conversation, see Simulation results. When ORQ_API_KEY is set, results are uploaded to Orq.ai as an Experiment by default. See Results in Orq.ai.

Agents with a memory store attached reject calls that carry no memory scope (a 400 with memory_entity_id_required). A fresh entity id is minted per conversation automatically, so parallel conversations never share memory. Pass memory_entity_id="..." to run every conversation against one specific, for example pre-seeded, entity instead.

Simulations run with parallelism=5 by default. Lower it if the target’s rate limits are strict, or raise it for faster runs.

Generate from a description

generate_and_simulate() creates personas, scenarios, and opening messages from a brief agent description. Use it for a quick first pass. Two target forms are supported:

A hosted Orq.ai agent: target="agent:<key>". The call is identical to First simulation
Any async function: pass it as target to test an agent that is not hosted in Orq.ai

The callback form, with upload_results=False for a local-only run:

Python

import asyncio

from openai import AsyncOpenAI

from evaluatorq.contracts import Message
from evaluatorq.simulation import generate_and_simulate

client = AsyncOpenAI()

SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful."

async def support_agent(messages: list[Message]) -> str:
    history = [{"role": "system", "content": SYSTEM}]
    history += [{"role": m.role, "content": m.content or ""} for m in messages]
    resp = await client.chat.completions.create(model="gpt-5.4-mini", messages=history)
    return resp.choices[0].message.content or ""

async def main():
    results = await generate_and_simulate(
        evaluation_name="support-agent-sim-openai",
        target=support_agent,
        agent_description=(
            "Customer support agent for an e-commerce store; "
            "handles refunds, orders, and product questions."
        ),
        num_personas=3,
        num_scenarios=4,
        sim_model="openai/gpt-5.4-mini",   # simulator and judge model
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
        upload_results=False,              # local-only run, no Orq experiment
    )

    passed = sum(r.goal_achieved for r in results)
    print(f"Pass rate: {passed}/{len(results)}")

if __name__ == "__main__":
    asyncio.run(main())

The only difference between the two forms is the target: a hosted Orq.ai agent key or the callback function. Personas, scenarios, criteria, and the returned results are identical. Two details matter for the callback form:

sim_model chooses the model for the simulator and judge. The provider resolves from ORQ_API_KEY (the Orq.ai router) or OPENAI_API_KEY (OpenAI-compatible)
The callback is plain Python: it calls AsyncOpenAI(), so it needs that provider’s key unless it routes through Orq.ai

Seed by archetype

Instead of specifying every trait by hand, name an archetype. generate_persona() fills in the rest of a Persona; generate_scenario() fills in the rest of a Scenario. The returned objects can be inspected, tweaked, and passed to simulate().

Python

import asyncio

from evaluatorq.simulation import generate_persona, generate_scenario, simulate

async def main():
    persona = await generate_persona(
        "angry customer",
        agent_description="e-commerce support agent",
    )
    scenario = await generate_scenario("disputes a refund denial")

    results = await simulate(
        evaluation_name="seeded-simulation",
        target="agent:my-support-agent",
        personas=[persona],
        scenarios=[scenario],
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
    )
    print(f"Goal achieved: {results[0].goal_achieved}")

if __name__ == "__main__":
    asyncio.run(main())

Batch forms generate_personas([...]) and generate_scenarios([...]) take a list of seed phrases and return one object per seed.

The seed guides generation; it is not a transcript to replay. Generation fills in the persona traits and scenario criteria and writes a natural opening message. Each run explores the space around the pattern rather than replaying one recorded conversation.

Full control

Define Persona, Scenario, and Criterion objects by hand to set exact traits, goals, and pass/fail rules. A persona is who is talking. name, patience, assertiveness, politeness, technical_level, communication_style, and background are required; only emotional_arc and cultural_context are optional. A scenario is what they want, plus the criteria the agent must, or must not, satisfy. Only name and goal are required.

Orq agent
Custom callback

Pass the agent key with the agent: prefix.

Python

import asyncio

from evaluatorq.simulation import (
    CommunicationStyle,
    Criterion,
    EmotionalArc,
    Persona,
    Scenario,
    StartingEmotion,
    simulate,
)

async def main():
    persona = Persona(
        name="Impatient Customer",
        patience=0.2,
        assertiveness=0.8,
        politeness=0.4,
        technical_level=0.3,
        communication_style=CommunicationStyle.terse,
        background="Received the wrong item and wants a refund urgently",
        emotional_arc=EmotionalArc.escalating,
    )
    scenario = Scenario(
        name="Wrong Item Refund",
        goal="Get a full refund for the wrong item received",
        context="Ordered headphones but received a phone case instead",
        starting_emotion=StartingEmotion.frustrated,
        criteria=[
            Criterion(description="Agent asks for order details", type="must_happen"),
            Criterion(description="Agent acknowledges the mistake", type="must_happen"),
            Criterion(description="Agent blames the customer", type="must_not_happen"),
        ],
    )

    results = await simulate(
        evaluation_name="basic-simulation-example",
        target="agent:my-support-agent",
        personas=[persona],
        scenarios=[scenario],
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
    )

    result = results[0]
    score = result.goal_completion_score or 0.0
    print(f"Goal achieved: {result.goal_achieved}  score={score:.2f}")
    for msg in result.messages:
        who = "User" if msg.role == "user" else "Agent"
        print(f"{who}: {msg.content}")

if __name__ == "__main__":
    asyncio.run(main())

Pass any async function as target. Set sim_model to choose the simulator and judge model; the provider resolves from ORQ_API_KEY or OPENAI_API_KEY. Set upload_results=False to keep the run local. A callback that calls a provider directly needs that provider’s own key.

Python

import asyncio

from openai import AsyncOpenAI

from evaluatorq.contracts import Message
from evaluatorq.simulation import (
    CommunicationStyle,
    Criterion,
    Persona,
    Scenario,
    simulate,
)

client = AsyncOpenAI()

SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful."

async def support_agent(messages: list[Message]) -> str:
    history = [{"role": "system", "content": SYSTEM}]
    history += [{"role": m.role, "content": m.content or ""} for m in messages]
    resp = await client.chat.completions.create(model="gpt-5.4-mini", messages=history)
    return resp.choices[0].message.content or ""

async def main():
    persona = Persona(
        name="Impatient Customer",
        patience=0.2,
        assertiveness=0.8,
        politeness=0.4,
        technical_level=0.3,
        communication_style=CommunicationStyle.terse,
        background="Received the wrong item and wants a refund urgently",
    )
    scenario = Scenario(
        name="Wrong Item Refund",
        goal="Get a full refund for the wrong item received",
        criteria=[
            Criterion(description="Agent asks for order details", type="must_happen"),
        ],
    )

    results = await simulate(
        evaluation_name="custom-agent-simulation",
        target=support_agent,
        personas=[persona],
        scenarios=[scenario],
        sim_model="openai/gpt-5.4-mini",
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
        upload_results=False,
    )

    result = results[0]
    score = result.goal_completion_score or 0.0
    print(f"Goal achieved: {result.goal_achieved}  score={score:.2f}")

if __name__ == "__main__":
    asyncio.run(main())

Replay saved cases

Every case a simulation runs is one SimulationDatapoint: a persona, a scenario, and the opening message. Save the generated cases to a file once, then re-run the same cases against any target. The file pins the personas, scenarios, and first messages, so a run is reproducible. Use it to compare two agent versions on an identical set of conversations.

# Generate cases once and keep them
eq sim generate --agent-description "e-commerce support agent" \
  --num-personas 3 --num-scenarios 4 \
  --datapoints cases.jsonl

# Re-run the exact same cases against any target
eq sim simulate --input cases.jsonl --target agent:my-support-agent

simulate() takes one input source per run. The five sources are mutually exclusive:

datapoints: cases loaded from a JSONL file with load_datapoints_from_jsonl()
dataset_id: an Orq.ai dataset
experiment_id: rows of an Orq.ai experiment
previous_run: a run stored under .evaluatorq/sim-runs/
personas and scenarios: passed inline

On the CLI, cases are written with --datapoints and replayed with --input. eq sim from-traces builds new cases from production traces instead. See the evaluatorq agent simulation guide for the full flag set.

Simulation results

Each persona/scenario pair produces one SimulationResult:

Field	Type	Description
`goal_achieved`	`bool`	Whether the agent satisfied the scenario goal
`goal_completion_score`	`float`	Numeric score (0-1) from the judge
`turn_count`	`int`	Number of conversation turns
`rules_broken`	`list[str]`	Criteria of type `must_not_happen` that were violated
`messages`	`list[Message]`	Full conversation transcript

The result also carries terminated_by, reason, token_usage, criteria_results, and last_trace_id. See the Python API reference for the full shape. evaluator_names accepts any of the built-in evaluatorq scorers:

Name	Score	Description
`goal_achieved`	1.0 / 0.0	1.0 if the agent satisfied the scenario goal, 0.0 otherwise
`criteria_met`	0.0-1.0	Ratio of criteria the agent satisfied across the simulation
`turn_efficiency`	0.0-1.0	Higher score for fewer turns; 0.0 if the goal was not achieved
`conversation_quality`	0.0-1.0	Composite: 40% goal, 30% criteria, 30% turn efficiency

"goal_achieved" and "criteria_met" are used by default when evaluator_names is omitted.

Results in Orq.ai

When ORQ_API_KEY is set, results are uploaded to the workspace as an Experiment run (upload_results defaults to True). A direct link is printed at the end of the run:

✅ Results sent to Orq: support-agent-sim (12 rows created)

📊 View your evaluation at: https://my.orq.ai/<workspace>/experiments/<id>?runId=<runId>

Each conversation is logged as a datapoint with its persona, scenario, transcript, and evaluator scores, so runs can be filtered, compared, and tracked over time. Pass upload_results=False for a local-only run.

Exploring runs locally

Every run is also saved to .evaluatorq/sim-runs/ in the working directory. The CLI saves runs by default (--no-save skips it). The SDK opts in per run with save=True on simulate() or generate_and_simulate(). The evaluatorq dashboard reads that store and renders the saved runs:

uv add "evaluatorq[dashboard]"

python -m pip install "evaluatorq[dashboard]"

eq dashboard .evaluatorq/sim-runs     # simulation runs only
eq dashboard                          # red team and simulation runs

The dashboard serves on http://127.0.0.1:8080 by default. Unlike a single-run view, it indexes every saved run, so simulations can be compared over time. List saved runs from the command line with eq sim runs.

The dashboard is a preview and still under active development. Its layout and options may change between releases.

Going further

The evaluatorq documentation covers the parts of Agent Simulation that sit outside Orq.ai: