> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. # Benchmark models head-to-head with Model Arena > Run a real head-to-head model benchmark on real prompts with orq-arena, and read a statistically defensible ranking instead of trusting a public leaderboard. TL;DR * A public leaderboard scores models on generic data. It does not show which model is cheap enough, and good enough, for **this product's** real prompts. * [orq-arena](https://github.com/orq-ai/orq-arena) is an open-source CLI. It runs models against each other on supplied prompts. An LLM jury judges the answers, through the **AI Gateway**. * The output is a report and a raw data file. Both are evidence a team can show stakeholders. Re-run the same benchmark later, when a new model ships. * This cookbook runs one real benchmark: a flagship model against two cheaper models. The question: is the extra cost worth it for this task? In this run, most of the flagship's lead came from writing longer answers, not better ones. ## What you'll build A benchmark of three models, two from **Anthropic** and one from **OpenAI**, on eight support and product prompts. A three-model jury judges the answers. The jury comes from providers outside the candidate pool. The result is a report that ranks all three models by quality, and shows cost against quality. ## What you'll learn Model selection is an ongoing decision, not a one-time pick. A leaderboard win does not prove a model is worth its price for a specific task. orq-arena turns "which model should we use" into a process anyone can repeat: run it on real prompts, get evidence, run it again later. Two things make the result trustworthy: * **Bias controls**: every pair is judged in both orders, so order alone cannot decide a result. A judge whose votes look unreliable is dropped from that round. * **Statistics**: rankings use a Bradley-Terry model, the same method behind chess ELO, with bootstrapped 95% confidence intervals. Two models with overlapping intervals are not automatically tied. The report checks this directly and states the answer. ## Prerequisites * Python 3.10 or later, and [uv](https://docs.astral.sh/uv/). * An **Orq.ai** API key with at least one chat model enabled. Create one under workspace settings: see the [API keys guide](/docs/ai-studio/organization/api-keys). * Install steps, the full config reference, and the method behind the ranking live at [orq-arena's own docs](https://orq-ai.github.io/orq-arena/) and on the [**Model Arena** page](/docs/ai-gateway/model-arena) of this site. This cookbook does not repeat them. `uv tool install` puts the `orq-arena` command on the global PATH. But the command reads `.env`, `--config`, and prompt files from the current directory, not the install location. Run every command below from inside the cloned repository. ## Step 1: Install and connect ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/orq-ai/orq-arena.git cd orq-arena uv tool install . cp .env.example .env ``` Open `.env`. Set `ORQ_API_KEY` to the key from Prerequisites. Never paste the key into a shared terminal, and never commit it. Confirm the install, and check which models the workspace has enabled, at zero cost: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} orq-arena --version orq-arena refresh-catalog --show ``` ## Step 2: Choose a model pool that tests a real decision The useful comparison is not "flagship against flagship." It is: **the team already pays for a flagship model on this task. Is that necessary?** This run tests two alternatives at once: a cheaper model from the same vendor, and a cheaper model from a different vendor. Save this as `cookbook.yaml`: ```yaml cookbook.yaml theme={"theme":{"light":"github-light","dark":"github-dark"}} match: max_rounds: 8 gateway: base_url: https://api.orq.ai/v3/router candidate_max_tokens: 2048 judge_max_tokens: 2048 stream_read_timeout_s: 1200 judge_timeout_ms: 90000 candidates: - model_id: anthropic/claude-sonnet-5 - model_id: anthropic/claude-haiku-4-5-20251001 - model_id: openai/gpt-5.4-mini reasoning: { reasoning_effort: none } judges: - google/gemini-2.5-flash-lite - mistral/mistral-medium-2508 - togetherai/meta-llama/Llama-3.3-70B-Instruct-Turbo criteria: >- Accuracy and correctness, helpfulness and completeness, clarity, and relevance to the prompt. min_successful_judges: 2 ``` Candidates: `claude-sonnet-5` is the incumbent flagship. `claude-haiku-4-5` is a cheaper model from the same vendor. `gpt-5.4-mini` is a cheaper model from a different vendor, with its reasoning turned off to keep the comparison fair. Judges: all three judges come from providers outside the candidate pool. A judge that shares a provider with a candidate can favor that candidate's writing style over its actual quality. This is a known risk in LLM-as-judge setups. Check the pool before spending anything: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} orq-arena pool --config cookbook.yaml ``` ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} Seed Name Model ID ---------------------------------------------------------------------- 1 claude-sonnet-5 anthropic/claude-sonnet-5 2 claude-haiku-4-5-20251001 anthropic/claude-haiku-4-5-20251001 3 gpt-5.4-mini openai/gpt-5.4-mini ``` A judge also needs to turn its own reasoning off. If it does not, a low `judge_max_tokens` cap can cut its answer short and turn its vote into a failure. This can push a round below quorum. Check the model's metadata for `supports_reasoning`, or watch for the CLI's own `pool is thinking-clean` message at the start of a run. ## Step 3: Write prompts that resemble production traffic orq-arena reads prompts from a JSONL file, one object per line. Trivia questions do not test a jury's judgment. Realistic task prompts do. Save this as `cookbook-prompts.jsonl`: ```json cookbook-prompts.jsonl theme={"theme":{"light":"github-light","dark":"github-dark"}} {"prompt": "A customer says their invoice shows a duplicate charge for the same subscription month. Draft a short, apologetic reply that explains support will investigate and refund within 3 business days if confirmed.", "category": "support-reply"} {"prompt": "Summarize this policy note in one sentence for a customer email: 'We are migrating to a new billing provider next month. Existing payment methods carry over automatically. No action is needed unless the card has expired, in which case customers must update it in Settings > Billing.'", "category": "summarize"} {"prompt": "A customer asks: 'Why was I charged twice this month?' List the three most likely causes an internal billing team should check before replying to the customer.", "category": "internal-triage"} {"prompt": "Explain, in plain language a non-technical customer can understand, why an API rate limit error (HTTP 429) happens and what they should do about it.", "category": "technical-explanation"} {"prompt": "A user reports: 'My export to CSV has been stuck at 40% for 20 minutes.' Write a troubleshooting reply that asks for the two pieces of information support needs to diagnose it, without sounding like a script.", "category": "support-reply"} {"prompt": "Classify this support ticket into exactly one category: Billing, Bug Report, Feature Request, or Account Access. Ticket: 'I can't log in anymore, it says my password is wrong even though I just reset it.' Answer with only the category name.", "category": "classification"} {"prompt": "A customer wants to downgrade from the annual Pro plan to the monthly Basic plan mid-cycle. Explain the proration policy in two sentences.", "category": "policy-explanation"} {"prompt": "Write a short changelog entry, 2 to 3 sentences, announcing that CSV exports now support custom date ranges.", "category": "product-writing"} ``` A local JSONL file works for a first run. An existing [Dataset](/docs/ai-studio/optimize/datasets) in the workspace also works, with `--prompts orq:`. No export step is needed. ## Step 4: Run the benchmark ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} orq-arena run \ --config cookbook.yaml \ --prompts cookbook-prompts.jsonl \ --output cookbook-battles.jsonl \ --rounds 8 ``` `--rounds 8` matches the prompt count, so the run uses every prompt. Without it, the run uses a smaller, seeded slice set by `match.max_rounds`. The command pauses before it spends anything: ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} preflight: 3 matches × 8 rounds → 48 model streams + 144 judge calls + 3 probe calls PROJECTED SPEND ≈ $1.33 worst case, retries + stand-ins ≈ $1.97 thinking probe… pool is thinking-clean ✓ Proceed (≈ $1.33, up to $1.97 with retries)? [y/N]: ``` Type `y` to run it. Add `-y` to skip this prompt in a script. `run` refuses to proceed on non-interactive input without `-y`, so this flag is also what a CI job or a scheduled re-run needs. This step spends real money: once per candidate call, once per judge call. Read the number the preflight prints before confirming. Agree a spending limit with whoever owns the workspace before running a pool for the first time. ## Step 5: Read the report The run writes three files. `cookbook-battles.jsonl` holds every judged round, with both full answers and every vote. `cookbook-battles.run.json` is a manifest recording the exact config, hashes, and candidate list the run used. `cookbook-battles.report.html` is one shareable file, with no external assets. This run produced: | # | Model | ELO | 95% CI | Win rate | Length-adj. | Avg length | | - | ------------------------- | ---- | --------- | -------- | ----------- | ---------- | | 1 | claude-sonnet-5 | 1133 | 1000-1471 | 77% | 895 | 259 chars | | 2 | claude-haiku-4-5-20251001 | 1015 | 855-1174 | 55% | 1074 | 133 chars | | 3 | gpt-5.4-mini | 851 | 610-966 | 21% | 1031 | 93 chars | Rendered orq-arena report showing the leaderboard table and cost-versus-quality value map for this cookbook's benchmark run, with claude-sonnet-5 leading on ELO but ranking last once response length is priced out The report's own headline: *"claude-sonnet-5 leads claude-haiku-4-5-20251001, but this run is too small to call it... That is not evidence they are equal, only that this run cannot separate them."* This is the first half of the answer. The flagship wins the raw ranking. But at this sample size, that win does not prove the cheaper, same-vendor model is worse. Overlapping confidence intervals do not by themselves mean two models are tied. The report checks this separately, in its own "Methodology in detail" section: > Two of them overlapping does not mean the two models are tied, because both are drawn from the same resamples and share the anchoring... Whether the top two separate is decided on the bootstrap of their difference... reported in Confidence stats. The second half of the answer is the **length-adjusted** column. `claude-sonnet-5` wrote the longest answers, 259 characters on average against 133 and 93. Once that length preference is priced out, its score drops from first place to last. Both cheaper models rank above it. This does not prove `claude-sonnet-5` writes worse answers. It shows that much of its lead in this run came from writing longer answers, which this jury rewarded, not necessarily from writing better ones. That is the real finding here: the number a team would use to justify the flagship's price was mostly a length effect, not a quality one. Two more numbers worth reading from the report: * **Jury agreement**: 92% in this run. Low agreement is a sign to fix the judging criteria before trusting a ranking from it. * **Inconclusive rounds**: 5 of 24 round-slots did not reach the `min_successful_judges` quorum, and were not rated. This is expected. A jury vote below quorum is reported honestly as "no result," not forced into one. ## Going further This cookbook covers the core loop: install, configure, run, read the report. orq-arena has three more commands, documented in the [CLI reference](https://orq-ai.github.io/orq-arena/cli/): * **`rejudge`** re-scores an existing run with a different jury. It costs judge tokens only, since the answers are already recorded. Use it to check whether a ranking holds under a different panel. * **`annotate`** and **`anchor`** build a blinded human-review page from a recorded run, then merge the resulting votes back in, to check how well the jury agrees with human raters. * **`report`** rebuilds the HTML page from a saved log, with no API calls. Use it to regenerate a report later, or after a manifest changes. Free, offline commands, for exploring the format without spending anything: `orq-arena report examples/quickstart/battles.jsonl`, `orq-arena pool`, and `orq-arena refresh-catalog --show`. Full prerequisites, first-run walkthrough, and how to read the report in depth.