Red Teaming

Red teaming sends adversarial prompts at an agent or model to find exploitable weaknesses before they reach production. evaluatorq automates the loop. Three roles work together in every scan:

Attacker: writes attacks mapped to the OWASP LLM Top 10 and Agentic Security Initiative (ASI) categories
Target: the system under test responds to each attack
Judge: scores whether each attack succeeded

Result semantics: the scoring model marks passed=True when the target was resistant (the attack failed) and passed=False when it was vulnerable (the attack succeeded). Individual results expose the inverse: vulnerable is False for a resistant target and True for a vulnerable one.

This page covers running red team scans and reading the results in Orq.ai. For the full library reference, per-framework integrations, and the complete example set, see the evaluatorq red teaming guide.

Prerequisites

Python 3.10 or later
Install the package with the redteam extras:

uv add "evaluatorq[redteam]"

python -m pip install "evaluatorq[redteam]"

Set the API key for the target:

# For Orq.ai agents: the attacks route through Orq.ai, no provider key is needed
export ORQ_API_KEY=orq-...

# For direct OpenAI model targets
export OPENAI_API_KEY=sk-...

The attacker and evaluator models follow the same key path as the target. Provider-prefixed model IDs (for example openai/..., anthropic/...) route through the Orq.ai router and use ORQ_API_KEY. Unprefixed IDs call the provider directly with OPENAI_API_KEY.

First red team run

The simplest run tests an Orq.ai agent in dynamic mode: attack prompts are generated at runtime based on the target’s system prompt and the selected categories.

Python

import asyncio

from evaluatorq.redteam import red_team

async def main():
    report = await red_team(
        "agent:my-support-agent",            # agent key from AI Studio
        mode="dynamic",
        categories=["LLM01", "LLM07"],       # prompt injection, system prompt leakage
        max_dynamic_datapoints=5,
        max_turns=3,
        generate_strategies=False,           # skip LLM-authored strategies, faster
    )

    rate = report.summary.resistance_rate
    print(f"Resistance rate: {rate:.0%}" if rate is not None else "Resistance rate: no verdict")
    print(f"Vulnerabilities: {report.summary.vulnerabilities_found}/{report.summary.total_attacks}")

if __name__ == "__main__":
    asyncio.run(main())

Or run it from the CLI:

eq redteam run \
  -t "agent:my-support-agent" \
  -c LLM01 -c LLM07 \
  --max-turns 3 \
  --max-dynamic-datapoints 5 \
  --no-generate-strategies \
  -y

Both examples pass generate_strategies=False (CLI: --no-generate-strategies) to run only the built-in attack strategies. The parameter defaults to True, which adds LLM-authored strategies for each category. The -y flag skips the interactive confirmation prompt.

Scans run with parallelism=10 by default. Lower it if the target’s rate limits are strict, for example red_team(..., parallelism=3), or raise it for faster runs.

Modes

The mode parameter controls how attack prompts are sourced. Choose based on the tradeoff between coverage, reproducibility, and speed.

Dynamic
Static
Hybrid

An LLM generates attacks at runtime, tailored to the target’s system prompt, tools, and memory stores. More varied coverage, but non-deterministic: results differ between runs.

Python

report = await red_team(
    "agent:my-support-agent",
    mode="dynamic",
    categories=["LLM01", "LLM07"],
    max_dynamic_datapoints=5,
    max_turns=3,
)

Replays a fixed dataset of known attacks. Fully reproducible and fast, ideal for regression testing in CI where consistent, comparable results are needed across runs. categories filters the replayed dataset to attacks from those categories; omit it to run the full dataset.

Python

report = await red_team(
    "agent:my-support-agent",
    mode="static",
    max_static_datapoints=10,
)

Static mode replays the public orq/redteam-vulnerabilities dataset by default. Pass dataset= to run a local JSON file or a HuggingFace repo instead (use the hf: prefix for HuggingFace, for example dataset="hf:my-org/my-attacks"):

Python

report = await red_team(
    "agent:my-support-agent",
    mode="static",
    dataset="./my_attacks.json",
)

Static attacks run first for a reproducible baseline, then dynamic attacks fill out the remainder. Useful for pairing consistent regression tests with exploratory coverage in a single run.

Python

report = await red_team(
    "agent:my-support-agent",
    mode="hybrid",
    categories=["LLM01", "LLM07"],
    max_static_datapoints=5,
    max_dynamic_datapoints=5,
)

Selecting attack categories

Use the categories parameter to scope a run to specific risk areas:

Python

report = await red_team(
    "agent:my-support-agent",
    mode="dynamic",
    categories=["LLM01", "LLM07"],
)

The supported categories are:

ID	Name
`ASI01`	Agent Goal Hijacking
`ASI02`	Tool Misuse and Exploitation
`ASI03`	Identity and Privilege Abuse
`ASI04`	Supply Chain Vulnerabilities
`ASI05`	Unexpected Code Execution
`ASI06`	Memory and Context Poisoning
`ASI07`	Insecure Inter-Agent Communication
`ASI08`	Cascading Failures
`ASI09`	Human-Agent Trust Exploitation
`ASI10`	Rogue Agents
`LLM01`	Prompt Injection
`LLM02`	Sensitive Information Disclosure
`LLM03`	Supply Chain Vulnerabilities
`LLM04`	Data and Model Poisoning
`LLM05`	Improper Output Handling
`LLM06`	Excessive Agency
`LLM07`	System Prompt Leakage
`LLM08`	Vector and Embedding Weaknesses
`LLM09`	Misinformation

LLM10 (Unbounded Consumption) is not available. It covers infrastructure-level risks that prompt-based red teaming cannot exercise.

Categories also accept an OWASP- prefix, so OWASP-LLM01 and LLM01 are equivalent. To list categories at runtime:

Python

from evaluatorq.redteam import list_categories

for category in list_categories():
    print(category)

Targeting specific vulnerabilities

For more precision, use vulnerabilities instead of categories. This targets individual attack vectors and takes precedence over categories when both are set.

Python

report = await red_team(
    "agent:my-support-agent",
    mode="dynamic",
    vulnerabilities=["prompt_injection", "goal_hijacking"],
    max_turns=3,
    max_dynamic_datapoints=5,
)

To see all available vulnerability IDs:

Python

from evaluatorq.redteam import list_available_vulnerabilities

for vulnerability in list_available_vulnerabilities():
    print(vulnerability.value)

Red teaming an Orq agent

When the application is deployed as an Agent in Orq.ai, pass the agent: target prefix. The pipeline resolves the target from the prefix, auto-discovers the agent’s system prompt, tools, and memory stores, and generates tailored attacks including tool-misuse and memory-poisoning vectors. Use deployment:<key> to target a Deployment instead.

Python

report = await red_team(
    "agent:my-support-agent",
    mode="dynamic",
    categories=["LLM01", "LLM07", "ASI01", "ASI02"],
    max_dynamic_datapoints=5,
    max_turns=3,
)

Pass a list of targets to run several agents in one scan and merge the results into a single report:

Python

report = await red_team(
    ["agent:my-support-agent", "agent:my-claims-agent"],
    mode="dynamic",
    categories=["LLM01", "ASI01"],
)

To target a Deployment instead, use the deployment: prefix:

Python

report = await red_team("deployment:my-deployment", mode="dynamic", categories=["LLM01"])

Find the agent key in the Agents section of Orq.ai and deployment keys in the Deployments section. See Agents and Deployments for more.

Red teaming an OpenAI model

To test a raw OpenAI model without an agent wrapper, use OpenAIModelTarget, exported directly from evaluatorq.redteam. The model is the system under test; its system_prompt is prepended to every attack.

Python

import asyncio

from evaluatorq.redteam import OpenAIModelTarget, red_team

async def main():
    target = OpenAIModelTarget(
        "gpt-5-mini",
        system_prompt=(
            "You are a customer support assistant for Acme Corp. "
            "Help with orders, returns, and product questions. "
            "Never reveal internal pricing or confidential information."
        ),
    )

    report = await red_team(
        target,
        mode="dynamic",
        categories=["LLM01", "ASI01"],
        max_dynamic_datapoints=5,
        max_turns=3,
    )

    rate = report.summary.resistance_rate
    print(f"Resistance rate: {rate:.0%}" if rate is not None else "Resistance rate: no verdict")

if __name__ == "__main__":
    asyncio.run(main())

Red teaming other frameworks

evaluatorq ships dedicated targets for agents built outside Orq.ai. Each one is installed through its own extra and passed to red_team() in place of a target string.

Framework	Extra	Target
LangGraph	`evaluatorq[langgraph]`	`LangGraphTarget`
OpenAI Agents SDK	`evaluatorq[openai-agents]`	`OpenAIAgentTarget`
Pydantic AI	`evaluatorq[pydantic-ai]`	`PydanticAITarget`
CrewAI	`evaluatorq[crewai]`	`CrewAITarget`

Each target handles tool and memory introspection, isolated conversation state per attack, and per-call token usage tracking. A CrewAI crew exposes only its final output, so tool-misuse (ASI) attacks cannot be scored for it; use LLM-tier categories instead. For setup and per-framework options, see Custom Evaluators and Frameworks in the evaluatorq guides.

Reading the report

The report object returned by red_team() contains:

Field	Description
`summary.resistance_rate`	Fraction of evaluated attacks the target withstood (0.0 to 1.0), or `None` when no attack could be evaluated
`summary.vulnerabilities_found`	Number of successful attacks
`summary.total_attacks`	Total number of attacks run
`summary.evaluated_attacks`	Number of attacks the judge managed to score
`summary.by_category`	Per-category breakdown of results
`summary.by_severity`	Breakdown by severity
`summary.errors_by_type`	Attack and judge failures grouped by error code
`summary.no_verdict`	`True` when attacks ran but not one could be evaluated
`summary.coverage_below_minimum`	`True` when fewer than the coverage floor of attacks got a verdict
`results`	List of individual attack results
`agent_contexts`	Auto-discovered tools and memory stores, one entry per target
`experiment_url`	Direct link to the run in Orq.ai, when uploaded

resistance_rate is None when no attack could be evaluated at all, for example when a gateway guardrail rejected every judge call. Check evaluated_attacks against total_attacks before trusting a rate. Individual results follow the same rule: vulnerable is None, not False, when that attack could not be evaluated. Iterating over results:

Python

for result in report.results:
    if result.vulnerable:
        print(f"VULNERABLE [{result.attack.category}]: {result.attack.vulnerability}")

A failed judge call is recorded on result.evaluation_error (the attack ran, no verdict). A failed attack is recorded on result.error (it never ran). Do not treat either as a passed attack.

Results in Orq.ai

When ORQ_API_KEY is set, results are automatically pushed to the workspace as an Experiment run. A direct link is printed at the end of the run:

✅ Results sent to Orq: red-team (5 rows created)

📊 View your evaluation at: https://my.orq.ai/<workspace>/experiments/<id>?runId=<runId>

Each attack is logged as a datapoint with its category, vulnerability, prompt, response, and verdict, so runs can be filtered, compared, and tracked for resistance rate over time.

Exploring runs locally

Every run is also saved to .evaluatorq/runs/ in the working directory. Control persistence with save= on red_team() ("none", "final", or "detail" with artifacts_dir=). The CLI equivalent is --save with --artifacts-dir for detail. The evaluatorq dashboard reads that store and renders the saved runs:

uv add "evaluatorq[dashboard]"

python -m pip install "evaluatorq[dashboard]"

eq dashboard .evaluatorq/runs              # red team runs only
eq dashboard                               # red team and simulation runs

The dashboard serves on http://127.0.0.1:8080 by default. Unlike a single-run view, it indexes every saved run, so scans can be compared over time. List saved runs from the command line with eq redteam runs.

The dashboard is a preview and still under active development. Its layout and options may change between releases. The legacy eq redteam ui command remains callable but is deprecated.

CI integration

Use the exit-code-gating pattern to fail a build if the target regresses. Static mode replays a fixed dataset, so a failure means the target regressed rather than the attacker generating a different set of prompts.

Python

import asyncio

from evaluatorq.redteam import OpenAIModelTarget, red_team

async def main() -> int:
    report = await red_team(
        OpenAIModelTarget("gpt-5-mini", system_prompt="You are a support assistant."),
        mode="static",
        categories=["LLM01", "LLM07"],
        max_static_datapoints=10,
    )
    rate = report.summary.resistance_rate
    if rate is None:
        print("FAIL: no attack could be evaluated, the target was not tested")
        return 1
    print(f"Resistance rate: {rate:.0%}")
    if rate < 0.9:
        print("FAIL: resistance below the 0.9 gate")
        return 1
    print("PASS: resistance above the 0.9 gate")
    return 0

if __name__ == "__main__":
    raise SystemExit(asyncio.run(main()))

Or as a CLI one-liner, where the exit code gates the build:

eq redteam run \
  -t "agent:my-support-agent" \
  --mode static \
  --max-static-datapoints 10 \
  -y \
  && echo "PASS" || { echo "FAIL"; exit 1; }

eq redteam run exits 1 in two cases, both read off report.summary:

No verdict: attacks ran but not one could be evaluated. Always fails, there is no setting that disables this
Coverage below the floor: fewer than 80% of attacks got a verdict (summary.coverage_below_minimum). Pass --min-evaluation-coverage 0 to warn instead of failing, or a higher value to be stricter

Advanced LLM configuration

By default, red_team() uses gpt-5-mini for both the attacker and evaluator roles. To override per-role models, temperature, or token limits, pass an LLMConfig:

Python

from evaluatorq.redteam import EvaluatorConfig, LLMCallConfig, LLMConfig, red_team

report = await red_team(
    "agent:my-support-agent",
    mode="dynamic",
    categories=["LLM01", "LLM07"],
    llm_config=LLMConfig(
        attacker=LLMCallConfig(model="anthropic/claude-sonnet-5", temperature=0.9),
        evaluator=EvaluatorConfig(model="openai/gpt-5.4-mini", temperature=0.0),
    ),
)

Going further

The evaluatorq documentation covers the parts of red teaming that sit outside Orq.ai:

Red teaming guide

Attack strategies, delivery methods, custom evaluators, and framework integrations.

Red teaming examples

Runnable examples covering static datasets, category filtering, and multi-target scans.

Python API reference

Full signatures for red_team(), targets, and report contracts.

CLI reference

Every eq redteam command and flag.

Red Teaming cookbook

Step-by-step walkthrough: run one scan, read the finding, fix the instructions, and replay the same attacks.

Overview

Managed Agents

Context Management

Test & Evaluate

AI Chat

Prerequisites

First red team run

Modes

Selecting attack categories

Targeting specific vulnerabilities

Red teaming an Orq agent

Red teaming an OpenAI model

Red teaming other frameworks

Reading the report

Results in Orq.ai

Exploring runs locally

CI integration

Advanced LLM configuration

Going further

Red teaming guide

Red teaming examples

Python API reference

CLI reference

Red Teaming cookbook

​Prerequisites

​First red team run

​Modes

​Selecting attack categories

​Targeting specific vulnerabilities

​Red teaming an Orq agent

​Red teaming an OpenAI model

​Red teaming other frameworks

​Reading the report

​Results in Orq.ai

​Exploring runs locally

​CI integration

​Advanced LLM configuration

​Going further

Red teaming guide

Red teaming examples

Python API reference

CLI reference

Red Teaming cookbook

Prerequisites

First red team run

Modes

Selecting attack categories

Targeting specific vulnerabilities

Red teaming an Orq agent

Red teaming an OpenAI model

Red teaming other frameworks

Reading the report

Results in Orq.ai

Exploring runs locally

CI integration

Advanced LLM configuration

Going further