Run Agent Simulation with Evaluatorq

What is Agent Simulation?

Agent Simulation is a feature of the evaluatorq library that drives agents through realistic multi-turn conversations. Three LLMs work together:

Target agent: the system under test (an Orq Agent or any async callback)
User simulator: plays a persona pursuing scenario goals turn-by-turn
Judge: scores goal achievement and rule compliance after each simulation

Results are returned as SimulationResult objects and, for Orq deployments, are automatically uploaded to AI Studio as Experiments.

Install

pip install "evaluatorq[simulation]"
export ORQ_API_KEY=...

Approach 1: Generate from a description

generate_and_simulate() creates personas, scenarios, and opening messages from a brief agent description. Use this for a quick first pass.

Orq agent
OpenAI

num_personas × num_scenarios simulations run in parallel. Results are uploaded to AI Studio automatically.

Python

import asyncio
from evaluatorq.simulation import generate_and_simulate

async def main():
    results = await generate_and_simulate(
        evaluation_name="support-agent-sim",
        target="agent:my-support-agent",
        agent_description=(
            "Customer support agent for an e-commerce store; "
            "handles refunds, orders, and product questions."
        ),
        num_personas=3,
        num_scenarios=4,                     # → 12 simulations total
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
    )

    passed = sum(r.goal_achieved for r in results)
    print(f"Pass rate: {passed}/{len(results)}")

if __name__ == "__main__":
    asyncio.run(main())

Pass any async function as target_callback. Set sim_model to route the simulator and judge through any provider. Set upload_results=False to keep results local.

Python

import asyncio
from openai import AsyncOpenAI
from evaluatorq.contracts import Message
from evaluatorq.simulation import generate_and_simulate

client = AsyncOpenAI()
SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful."

async def my_agent(messages: list[Message]) -> str:
    history = [{"role": "system", "content": SYSTEM}]
    history += [{"role": m.role, "content": m.content or ""} for m in messages]
    resp = await client.chat.completions.create(model="gpt-4o-mini", messages=history)
    return resp.choices[0].message.content or ""

async def main():
    results = await generate_and_simulate(
        evaluation_name="support-agent-sim",
        target_callback=my_agent,
        agent_description=(
            "Customer support agent for an e-commerce store; "
            "handles refunds, orders, and product questions."
        ),
        num_personas=3,
        num_scenarios=4,
        sim_model="gpt-4o-mini",             # simulator + judge provider
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
        upload_results=False,
    )

    passed = sum(r.goal_achieved for r in results)
    print(f"Pass rate: {passed}/{len(results)}")

if __name__ == "__main__":
    asyncio.run(main())

Approach 2: Seed by archetype

Name an archetype to get back editable Persona and Scenario objects. Inspect or adjust them before running the simulation.

Python

import asyncio
from evaluatorq.simulation import generate_persona, generate_scenario, simulate

async def main():
    persona = await generate_persona(
        "angry customer",
        agent_description="e-commerce support agent",
    )
    scenario = await generate_scenario("disputes a refund denial")

    results = await simulate(
        evaluation_name="seeded-simulation",
        target="agent:my-support-agent",
        personas=[persona],
        scenarios=[scenario],
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
    )
    print(f"Goal achieved: {results[0].goal_achieved}")

if __name__ == "__main__":
    asyncio.run(main())

Use generate_personas([...]) and generate_scenarios([...]) (also from evaluatorq.simulation) to generate multiple objects in one call.

Replace target with target_callback=<my_async_fn>, add sim_model="gpt-4o-mini" to set the simulator/judge provider, and set upload_results=False to keep results local.

Approach 3: Full control

Define Persona, Scenario, and Criterion objects by hand to set exact traits, goals, and pass/fail rules.

Orq agent
OpenAI

Python

import asyncio
from evaluatorq.simulation import simulate
from evaluatorq.simulation.types import (
    CommunicationStyle,
    Criterion,
    EmotionalArc,
    Persona,
    Scenario,
    StartingEmotion,
)

async def main():
    persona = Persona(
        name="Impatient Customer",
        patience=0.2,
        assertiveness=0.8,
        politeness=0.4,
        technical_level=0.3,
        communication_style=CommunicationStyle.terse,
        background="Received the wrong item and wants a refund urgently",
        emotional_arc=EmotionalArc.escalating,
    )
    scenario = Scenario(
        name="Wrong Item Refund",
        goal="Get a full refund for the wrong item received",
        context="Ordered headphones but received a phone case instead",
        starting_emotion=StartingEmotion.frustrated,
        criteria=[
            Criterion(description="Agent asks for order details", type="must_happen"),
            Criterion(description="Agent acknowledges the mistake", type="must_happen"),
            Criterion(description="Agent blames the customer", type="must_not_happen"),
        ],
    )

    results = await simulate(
        evaluation_name="basic-simulation-example",
        target="agent:my-support-agent",
        personas=[persona],
        scenarios=[scenario],
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
    )

    result = results[0]
    score = result.goal_completion_score or 0.0
    print(f"Goal achieved: {result.goal_achieved}  score={score:.2f}")
    for msg in result.messages:
        who = "User" if msg.role == "user" else "Agent"
        print(f"{who}: {msg.content}")

if __name__ == "__main__":
    asyncio.run(main())

Replace target with target_callback and set sim_model to run the simulator and judge on any provider.

Python

import asyncio
from openai import AsyncOpenAI
from evaluatorq.contracts import Message
from evaluatorq.simulation import simulate
from evaluatorq.simulation.types import CommunicationStyle, Criterion, Persona, Scenario

client = AsyncOpenAI()
SYSTEM = "You are a customer support agent for Acme Corp. Be concise and helpful."

async def my_agent(messages: list[Message]) -> str:
    history = [{"role": "system", "content": SYSTEM}]
    history += [{"role": m.role, "content": m.content or ""} for m in messages]
    resp = await client.chat.completions.create(model="gpt-4o-mini", messages=history)
    return resp.choices[0].message.content or ""

async def main():
    persona = Persona(
        name="Impatient Customer",
        patience=0.2,
        assertiveness=0.8,
        politeness=0.4,
        technical_level=0.3,
        communication_style=CommunicationStyle.terse,
        background="Received the wrong item and wants a refund urgently",
    )
    scenario = Scenario(
        name="Wrong Item Refund",
        goal="Get a full refund for the wrong item received",
        criteria=[
            Criterion(description="Agent asks for order details", type="must_happen"),
        ],
    )

    results = await simulate(
        evaluation_name="custom-agent-simulation",
        target_callback=my_agent,
        personas=[persona],
        scenarios=[scenario],
        sim_model="gpt-4o-mini",
        max_turns=6,
        evaluator_names=["goal_achieved", "criteria_met"],
        upload_results=False,
    )

    result = results[0]
    score = result.goal_completion_score or 0.0
    print(f"Goal achieved: {result.goal_achieved}  score={score:.2f}")

if __name__ == "__main__":
    asyncio.run(main())

Simulation results

Each persona/scenario pair produces one SimulationResult:

Field	Type	Description
`goal_achieved`	`bool`	Whether the agent satisfied the scenario goal
`goal_completion_score`	`float`	Numeric score (0–1) from the judge
`turn_count`	`int`	Number of conversation turns
`rules_broken`	`list[str]`	Criteria of type `must_not_happen` that were violated
`messages`	`list[Message]`	Full conversation transcript

evaluator_names accepts any of the built-in evaluatorq scorers:

Name	Score	Description
`goal_achieved`	1.0 / 0.0	1.0 if the agent satisfied the scenario goal, 0.0 otherwise
`criteria_met`	0.0–1.0	Ratio of criteria the agent satisfied across the simulation
`turn_efficiency`	0.0–1.0	Higher score for fewer turns; 0.0 if goal was not achieved
`conversation_quality`	0.0–1.0	Composite: 40% goal, 30% criteria, 30% turn efficiency

"goal_achieved" and "criteria_met" are used by default when evaluator_names is omitted.

Red Teaming LLMs with evaluatorq

Probe deployments and agents for security vulnerabilities using the evaluatorq red teaming CLI.

​What is Agent Simulation?

​Install

​Approach 1: Generate from a description

​Approach 2: Seed by archetype

​Approach 3: Full control

​Simulation results

Red Teaming LLMs with evaluatorq

What is Agent Simulation?

Install

Approach 1: Generate from a description

Approach 2: Seed by archetype

Approach 3: Full control

Simulation results