> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Agent Simulation with Evaluatorq

> Test agents through realistic multi-turn conversations using AI-generated personas, scenarios, and a judge.

## What is Agent Simulation?

**Agent Simulation** is a feature of the [evaluatorq](https://github.com/orq-ai/orqkit/tree/main/packages/evaluatorq-py) library that drives agents through realistic multi-turn conversations. Three LLMs work together:

* **Target agent**: the system under test (an Orq Agent or any async callback)
* **User simulator**: plays a persona pursuing scenario goals turn-by-turn
* **Judge**: scores goal achievement and rule compliance after each simulation

Results are returned as `SimulationResult` objects and, for Orq deployments, are automatically uploaded to **AI Studio** as **Experiments**.

## Install

<CodeGroup>
  ```bash Orq deployment theme={"theme":{"light":"github-light","dark":"github-dark"}}
  pip install "evaluatorq[simulation]"
  export ORQ_API_KEY=...
  ```

  ```bash OpenAI theme={"theme":{"light":"github-light","dark":"github-dark"}}
  pip install "evaluatorq[simulation]" openai
  export OPENAI_API_KEY=sk-...
  ```
</CodeGroup>

## Approach 1: Generate from a description

`generate_and_simulate()` creates personas, scenarios, and opening messages from a brief agent description. Use this for a quick first pass.

<Tabs>
  <Tab title="Orq agent">
    `num_personas × num_scenarios` simulations run in parallel. Results are uploaded to **AI Studio** automatically.

    ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
    import asyncio
    from evaluatorq.simulation import generate_and_simulate

    async def main():
        results = await generate_and_simulate(
            evaluation_name="support-agent-sim",
            target="agent:my-support-agent",
            agent_description=(
                "Customer support agent for an e-commerce store; "
                "handles refunds, orders, and product questions."
            ),
            num_personas=3,
            num_scenarios=4,                     # → 12 simulations total
            max_turns=6,
            evaluator_names=["goal_achieved", "criteria_met"],
        )

        passed = sum(r.goal_achieved for r in results)
        print(f"Pass rate: {passed}/{len(results)}")

    if __name__ == "__main__":
        asyncio.run(main())
    ```
  </Tab>

  <Tab title="OpenAI">
    Pass any async function as `target_callback`. Set `sim_model` to route the simulator and judge through any provider. Set `upload_results=False` to keep results local.

    ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
    import asyncio
    from openai import AsyncOpenAI
    from evaluatorq.contracts import Message
    from evaluatorq.simulation import generate_and_simulate

    client = AsyncOpenAI()
    SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful."

    async def my_agent(messages: list[Message]) -> str:
        history = [{"role": "system", "content": SYSTEM}]
        history += [{"role": m.role, "content": m.content or ""} for m in messages]
        resp = await client.chat.completions.create(model="gpt-4o-mini", messages=history)
        return resp.choices[0].message.content or ""

    async def main():
        results = await generate_and_simulate(
            evaluation_name="support-agent-sim",
            target_callback=my_agent,
            agent_description=(
                "Customer support agent for an e-commerce store; "
                "handles refunds, orders, and product questions."
            ),
            num_personas=3,
            num_scenarios=4,
            sim_model="gpt-4o-mini",             # simulator + judge provider
            max_turns=6,
            evaluator_names=["goal_achieved", "criteria_met"],
            upload_results=False,
        )

        passed = sum(r.goal_achieved for r in results)
        print(f"Pass rate: {passed}/{len(results)}")

    if __name__ == "__main__":
        asyncio.run(main())
    ```
  </Tab>
</Tabs>

## Approach 2: Seed by archetype

Name an archetype to get back editable `Persona` and `Scenario` objects. Inspect or adjust them before running the simulation.

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import asyncio
from evaluatorq.simulation import generate_persona, generate_scenario, simulate

async def main():
    persona = await generate_persona(
        "angry customer",
        agent_description="e-commerce support agent",
    )
    scenario = await generate_scenario("disputes a refund denial")

    results = await simulate(
        evaluation_name="seeded-simulation",
        target="agent:my-support-agent",
        personas=[persona],
        scenarios=[scenario],
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
    )
    print(f"Goal achieved: {results[0].goal_achieved}")

if __name__ == "__main__":
    asyncio.run(main())
```

<Tip>
  Use `generate_personas([...])` and `generate_scenarios([...])` (also from `evaluatorq.simulation`) to generate multiple objects in one call.
</Tip>

<Note>
  Replace `target` with `target_callback=<my_async_fn>`, add `sim_model="gpt-4o-mini"` to set the simulator/judge provider, and set `upload_results=False` to keep results local.
</Note>

## Approach 3: Full control

Define `Persona`, `Scenario`, and `Criterion` objects by hand to set exact traits, goals, and pass/fail rules.

<Tabs>
  <Tab title="Orq agent">
    ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
    import asyncio
    from evaluatorq.simulation import simulate
    from evaluatorq.simulation.types import (
        CommunicationStyle,
        Criterion,
        EmotionalArc,
        Persona,
        Scenario,
        StartingEmotion,
    )

    async def main():
        persona = Persona(
            name="Impatient Customer",
            patience=0.2,
            assertiveness=0.8,
            politeness=0.4,
            technical_level=0.3,
            communication_style=CommunicationStyle.terse,
            background="Received the wrong item and wants a refund urgently",
            emotional_arc=EmotionalArc.escalating,
        )
        scenario = Scenario(
            name="Wrong Item Refund",
            goal="Get a full refund for the wrong item received",
            context="Ordered headphones but received a phone case instead",
            starting_emotion=StartingEmotion.frustrated,
            criteria=[
                Criterion(description="Agent asks for order details", type="must_happen"),
                Criterion(description="Agent acknowledges the mistake", type="must_happen"),
                Criterion(description="Agent blames the customer", type="must_not_happen"),
            ],
        )

        results = await simulate(
            evaluation_name="basic-simulation-example",
            target="agent:my-support-agent",
            personas=[persona],
            scenarios=[scenario],
            max_turns=6,
            evaluator_names=["goal_achieved", "criteria_met"],
        )

        result = results[0]
        score = result.goal_completion_score or 0.0
        print(f"Goal achieved: {result.goal_achieved}  score={score:.2f}")
        for msg in result.messages:
            who = "User" if msg.role == "user" else "Agent"
            print(f"{who}: {msg.content}")

    if __name__ == "__main__":
        asyncio.run(main())
    ```
  </Tab>

  <Tab title="OpenAI">
    Replace `target` with `target_callback` and set `sim_model` to run the simulator and judge on any provider.

    ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
    import asyncio
    from openai import AsyncOpenAI
    from evaluatorq.contracts import Message
    from evaluatorq.simulation import simulate
    from evaluatorq.simulation.types import CommunicationStyle, Criterion, Persona, Scenario

    client = AsyncOpenAI()
    SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful."

    async def my_agent(messages: list[Message]) -> str:
        history = [{"role": "system", "content": SYSTEM}]
        history += [{"role": m.role, "content": m.content or ""} for m in messages]
        resp = await client.chat.completions.create(model="gpt-4o-mini", messages=history)
        return resp.choices[0].message.content or ""

    async def main():
        persona = Persona(
            name="Impatient Customer",
            patience=0.2,
            assertiveness=0.8,
            politeness=0.4,
            technical_level=0.3,
            communication_style=CommunicationStyle.terse,
            background="Received the wrong item and wants a refund urgently",
        )
        scenario = Scenario(
            name="Wrong Item Refund",
            goal="Get a full refund for the wrong item received",
            criteria=[
                Criterion(description="Agent asks for order details", type="must_happen"),
            ],
        )

        results = await simulate(
            evaluation_name="custom-agent-simulation",
            target_callback=my_agent,
            personas=[persona],
            scenarios=[scenario],
            sim_model="gpt-4o-mini",
            max_turns=6,
            evaluator_names=["goal_achieved", "criteria_met"],
            upload_results=False,
        )

        result = results[0]
        score = result.goal_completion_score or 0.0
        print(f"Goal achieved: {result.goal_achieved}  score={score:.2f}")

    if __name__ == "__main__":
        asyncio.run(main())
    ```
  </Tab>
</Tabs>

## Simulation results

Each persona/scenario pair produces one `SimulationResult`:

| Field                   | Type            | Description                                           |
| ----------------------- | --------------- | ----------------------------------------------------- |
| `goal_achieved`         | `bool`          | Whether the agent satisfied the scenario goal         |
| `goal_completion_score` | `float`         | Numeric score (0–1) from the judge                    |
| `turn_count`            | `int`           | Number of conversation turns                          |
| `rules_broken`          | `list[str]`     | Criteria of type `must_not_happen` that were violated |
| `messages`              | `list[Message]` | Full conversation transcript                          |

`evaluator_names` accepts any of the built-in evaluatorq scorers:

| Name                   | Score     | Description                                                 |
| ---------------------- | --------- | ----------------------------------------------------------- |
| `goal_achieved`        | 1.0 / 0.0 | 1.0 if the agent satisfied the scenario goal, 0.0 otherwise |
| `criteria_met`         | 0.0–1.0   | Ratio of criteria the agent satisfied across the simulation |
| `turn_efficiency`      | 0.0–1.0   | Higher score for fewer turns; 0.0 if goal was not achieved  |
| `conversation_quality` | 0.0–1.0   | Composite: 40% goal, 30% criteria, 30% turn efficiency      |

`"goal_achieved"` and `"criteria_met"` are used by default when `evaluator_names` is omitted.

<Card title="Red Teaming LLMs with evaluatorq" icon="shield-halved" href="/docs/ai-studio/cookbooks/evaluation-safety/red-teaming" arrow="true">
  Probe deployments and agents for security vulnerabilities using the evaluatorq red teaming CLI.
</Card>