> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. # Agent Simulation > Test agents through realistic multi-turn conversations with AI-generated personas, scenarios, and an LLM judge. Agent Simulation drives the agent under test through realistic multi-turn conversations without writing test transcripts by hand. **evaluatorq** generates the personas, scenarios, and opening messages, then scores each conversation with an LLM judge. Three LLMs work together in every simulation: * **Target agent**: the system under test, either a hosted **Orq.ai** agent (`target="agent:"`) or any async function * **User simulator**: plays a **persona** pursuing a **scenario** goal, turn by turn * **Judge**: scores whether the goal was met and whether any rules were broken Agent Simulation is available in the Python version of **evaluatorq** only. The TypeScript version does not support it. This page covers running simulations and reading the results in **Orq.ai**. For the full library reference, provider resolution, and the complete example set, see the [**evaluatorq** agent simulation guide](https://orq-ai.github.io/evaluatorq/guides/agent-simulation/). ## Prerequisites * Python 3.10 or later * Install the package with the `simulation` extras: ```bash uv theme={"theme":{"light":"github-light","dark":"github-dark"}} uv add "evaluatorq[simulation]" ``` ```bash pip theme={"theme":{"light":"github-light","dark":"github-dark"}} python -m pip install "evaluatorq[simulation]" ``` Set the API key. When targeting an **Orq.ai** agent, `ORQ_API_KEY` is the only key needed: the simulator, the judge, and the generators route through the **Orq.ai** router. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} export ORQ_API_KEY=orq-... ``` ## First simulation The fastest start is `generate_and_simulate()`: it synthesizes the personas, scenarios, and opening messages from a short description of the agent. `num_personas × num_scenarios` conversations run in parallel. ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} import asyncio from evaluatorq.simulation import generate_and_simulate async def main(): results = await generate_and_simulate( evaluation_name="support-agent-sim", target="agent:my-support-agent", # hosted Orq agent, key from AI Studio agent_description=( "Customer support agent for an e-commerce store; " "handles refunds, orders, and product questions." ), num_personas=3, num_scenarios=4, # 12 conversations in total max_turns=6, evaluator_names=["goal_achieved", "criteria_met"], ) passed = sum(r.goal_achieved for r in results) print(f"Pass rate: {passed}/{len(results)}") if __name__ == "__main__": asyncio.run(main()) ``` The result is a list of `SimulationResult` objects, one per conversation. `evaluator_names` selects the scorers applied to each conversation, see [Simulation results](#simulation-results). When `ORQ_API_KEY` is set, results are uploaded to **Orq.ai** as an **Experiment** by default. See [Results in Orq.ai](#results-in-orqai). Agents with a memory store attached reject calls that carry no memory scope (a 400 with `memory_entity_id_required`). A fresh entity id is minted per conversation automatically, so parallel conversations never share memory. Pass `memory_entity_id="..."` to run every conversation against one specific, for example pre-seeded, entity instead. Simulations run with `parallelism=5` by default. Lower it if the target's rate limits are strict, or raise it for faster runs. ## Generate from a description `generate_and_simulate()` creates personas, scenarios, and opening messages from a brief agent description. Use it for a quick first pass. Two target forms are supported: * A hosted **Orq.ai** agent: `target="agent:"`. The call is identical to [First simulation](#first-simulation) * Any async function: pass it as `target` to test an agent that is not hosted in **Orq.ai** The callback form, with `upload_results=False` for a local-only run: ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} import asyncio from openai import AsyncOpenAI from evaluatorq.contracts import Message from evaluatorq.simulation import generate_and_simulate client = AsyncOpenAI() SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful." async def support_agent(messages: list[Message]) -> str: history = [{"role": "system", "content": SYSTEM}] history += [{"role": m.role, "content": m.content or ""} for m in messages] resp = await client.chat.completions.create(model="gpt-5.4-mini", messages=history) return resp.choices[0].message.content or "" async def main(): results = await generate_and_simulate( evaluation_name="support-agent-sim-openai", target=support_agent, agent_description=( "Customer support agent for an e-commerce store; " "handles refunds, orders, and product questions." ), num_personas=3, num_scenarios=4, sim_model="openai/gpt-5.4-mini", # simulator and judge model max_turns=6, evaluator_names=["goal_achieved", "criteria_met"], upload_results=False, # local-only run, no Orq experiment ) passed = sum(r.goal_achieved for r in results) print(f"Pass rate: {passed}/{len(results)}") if __name__ == "__main__": asyncio.run(main()) ``` The only difference between the two forms is the target: a hosted **Orq.ai** agent key or the callback function. Personas, scenarios, criteria, and the returned results are identical. Two details matter for the callback form: * `sim_model` chooses the model for the simulator and judge. The provider resolves from `ORQ_API_KEY` (the **Orq.ai** router) or `OPENAI_API_KEY` (OpenAI-compatible) * The callback is plain Python: it calls `AsyncOpenAI()`, so it needs that provider's key unless it routes through **Orq.ai** ## Seed by archetype Instead of specifying every trait by hand, name an archetype. `generate_persona()` fills in the rest of a `Persona`; `generate_scenario()` fills in the rest of a `Scenario`. The returned objects can be inspected, tweaked, and passed to `simulate()`. ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} import asyncio from evaluatorq.simulation import generate_persona, generate_scenario, simulate async def main(): persona = await generate_persona( "angry customer", agent_description="e-commerce support agent", ) scenario = await generate_scenario("disputes a refund denial") results = await simulate( evaluation_name="seeded-simulation", target="agent:my-support-agent", personas=[persona], scenarios=[scenario], max_turns=6, evaluator_names=["goal_achieved", "criteria_met"], ) print(f"Goal achieved: {results[0].goal_achieved}") if __name__ == "__main__": asyncio.run(main()) ``` Batch forms `generate_personas([...])` and `generate_scenarios([...])` take a list of seed phrases and return one object per seed. The seed guides generation; it is not a transcript to replay. Generation fills in the persona traits and scenario criteria and writes a natural opening message. Each run explores the space around the pattern rather than replaying one recorded conversation. ## Full control Define `Persona`, `Scenario`, and `Criterion` objects by hand to set exact traits, goals, and pass/fail rules. A **persona** is who is talking. `name`, `patience`, `assertiveness`, `politeness`, `technical_level`, `communication_style`, and `background` are required; only `emotional_arc` and `cultural_context` are optional. A **scenario** is what they want, plus the `criteria` the agent must, or must not, satisfy. Only `name` and `goal` are required. **Pass the agent key** with the `agent:` prefix. ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} import asyncio from evaluatorq.simulation import ( CommunicationStyle, Criterion, EmotionalArc, Persona, Scenario, StartingEmotion, simulate, ) async def main(): persona = Persona( name="Impatient Customer", patience=0.2, assertiveness=0.8, politeness=0.4, technical_level=0.3, communication_style=CommunicationStyle.terse, background="Received the wrong item and wants a refund urgently", emotional_arc=EmotionalArc.escalating, ) scenario = Scenario( name="Wrong Item Refund", goal="Get a full refund for the wrong item received", context="Ordered headphones but received a phone case instead", starting_emotion=StartingEmotion.frustrated, criteria=[ Criterion(description="Agent asks for order details", type="must_happen"), Criterion(description="Agent acknowledges the mistake", type="must_happen"), Criterion(description="Agent blames the customer", type="must_not_happen"), ], ) results = await simulate( evaluation_name="basic-simulation-example", target="agent:my-support-agent", personas=[persona], scenarios=[scenario], max_turns=6, evaluator_names=["goal_achieved", "criteria_met"], ) result = results[0] score = result.goal_completion_score or 0.0 print(f"Goal achieved: {result.goal_achieved} score={score:.2f}") for msg in result.messages: who = "User" if msg.role == "user" else "Agent" print(f"{who}: {msg.content}") if __name__ == "__main__": asyncio.run(main()) ``` **Pass any async function** as `target`. Set `sim_model` to choose the simulator and judge model; the provider resolves from `ORQ_API_KEY` or `OPENAI_API_KEY`. Set `upload_results=False` to keep the run local. A callback that calls a provider directly needs that provider's own key. ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} import asyncio from openai import AsyncOpenAI from evaluatorq.contracts import Message from evaluatorq.simulation import ( CommunicationStyle, Criterion, Persona, Scenario, simulate, ) client = AsyncOpenAI() SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful." async def support_agent(messages: list[Message]) -> str: history = [{"role": "system", "content": SYSTEM}] history += [{"role": m.role, "content": m.content or ""} for m in messages] resp = await client.chat.completions.create(model="gpt-5.4-mini", messages=history) return resp.choices[0].message.content or "" async def main(): persona = Persona( name="Impatient Customer", patience=0.2, assertiveness=0.8, politeness=0.4, technical_level=0.3, communication_style=CommunicationStyle.terse, background="Received the wrong item and wants a refund urgently", ) scenario = Scenario( name="Wrong Item Refund", goal="Get a full refund for the wrong item received", criteria=[ Criterion(description="Agent asks for order details", type="must_happen"), ], ) results = await simulate( evaluation_name="custom-agent-simulation", target=support_agent, personas=[persona], scenarios=[scenario], sim_model="openai/gpt-5.4-mini", max_turns=6, evaluator_names=["goal_achieved", "criteria_met"], upload_results=False, ) result = results[0] score = result.goal_completion_score or 0.0 print(f"Goal achieved: {result.goal_achieved} score={score:.2f}") if __name__ == "__main__": asyncio.run(main()) ``` ## Replay saved cases Every case a simulation runs is one `SimulationDatapoint`: a persona, a scenario, and the opening message. Save the generated cases to a file once, then re-run the same cases against any target. The file pins the personas, scenarios, and first messages, so a run is reproducible. Use it to compare two agent versions on an identical set of conversations. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} # Generate cases once and keep them eq sim generate --agent-description "e-commerce support agent" \ --num-personas 3 --num-scenarios 4 \ --datapoints cases.jsonl # Re-run the exact same cases against any target eq sim simulate --input cases.jsonl --target agent:my-support-agent ``` `simulate()` takes one input source per run. The five sources are mutually exclusive: * `datapoints`: cases loaded from a JSONL file with `load_datapoints_from_jsonl()` * `dataset_id`: an **Orq.ai** dataset * `experiment_id`: rows of an **Orq.ai** experiment * `previous_run`: a run stored under `.evaluatorq/sim-runs/` * `personas` and `scenarios`: passed inline On the CLI, cases are written with `--datapoints` and replayed with `--input`. `eq sim from-traces` builds new cases from production traces instead. See the [**evaluatorq** agent simulation guide](https://orq-ai.github.io/evaluatorq/guides/agent-simulation/) for the full flag set. ## Simulation results Each persona/scenario pair produces one `SimulationResult`: | Field | Type | Description | | ----------------------- | --------------- | ----------------------------------------------------- | | `goal_achieved` | `bool` | Whether the agent satisfied the scenario goal | | `goal_completion_score` | `float` | Numeric score (0-1) from the judge | | `turn_count` | `int` | Number of conversation turns | | `rules_broken` | `list[str]` | Criteria of type `must_not_happen` that were violated | | `messages` | `list[Message]` | Full conversation transcript | The result also carries `terminated_by`, `reason`, `token_usage`, `criteria_results`, and `last_trace_id`. See the [Python API reference](https://orq-ai.github.io/evaluatorq/reference/evaluatorq/simulation/) for the full shape. `evaluator_names` accepts any of the built-in **evaluatorq** scorers: | Name | Score | Description | | ---------------------- | --------- | -------------------------------------------------------------- | | `goal_achieved` | 1.0 / 0.0 | 1.0 if the agent satisfied the scenario goal, 0.0 otherwise | | `criteria_met` | 0.0-1.0 | Ratio of criteria the agent satisfied across the simulation | | `turn_efficiency` | 0.0-1.0 | Higher score for fewer turns; 0.0 if the goal was not achieved | | `conversation_quality` | 0.0-1.0 | Composite: 40% goal, 30% criteria, 30% turn efficiency | `"goal_achieved"` and `"criteria_met"` are used by default when `evaluator_names` is omitted. ## Results in Orq.ai When `ORQ_API_KEY` is set, results are uploaded to the workspace as an **Experiment** run (`upload_results` defaults to `True`). A direct link is printed at the end of the run: ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} ✅ Results sent to Orq: support-agent-sim (12 rows created) 📊 View your evaluation at: https://my.orq.ai//experiments/?runId= ``` Each conversation is logged as a datapoint with its persona, scenario, transcript, and evaluator scores, so runs can be filtered, compared, and tracked over time. Pass `upload_results=False` for a local-only run. ## Exploring runs locally Every run is also saved to `.evaluatorq/sim-runs/` in the working directory. The CLI saves runs by default (`--no-save` skips it). The SDK opts in per run with `save=True` on `simulate()` or `generate_and_simulate()`. The **evaluatorq** dashboard reads that store and renders the saved runs: ```bash uv theme={"theme":{"light":"github-light","dark":"github-dark"}} uv add "evaluatorq[dashboard]" ``` ```bash pip theme={"theme":{"light":"github-light","dark":"github-dark"}} python -m pip install "evaluatorq[dashboard]" ``` ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} eq dashboard .evaluatorq/sim-runs # simulation runs only eq dashboard # red team and simulation runs ``` The dashboard serves on `http://127.0.0.1:8080` by default. Unlike a single-run view, it indexes every saved run, so simulations can be compared over time. List saved runs from the command line with `eq sim runs`. The dashboard is a preview and still under active development. Its layout and options may change between releases. ## Going further The **evaluatorq** documentation covers the parts of Agent Simulation that sit outside **Orq.ai**: Provider resolution, memory-backed agents, and trace-grounded case generation. Runnable examples covering tool simulation, hardening loops, and framework targets. Full signatures for `simulate()`, `generate_and_simulate()`, and the simulation contracts. Every `eq sim` command and flag. Step-by-step walkthrough of the same feature, including a plain-OpenAI setup.