> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. # Arena > Rank models head-to-head on real prompts with orq-arena, the Orq.ai benchmarking CLI. Pairwise LLM jury, Bradley-Terry ratings, confidence intervals. Public leaderboards rank models on someone else's data. Arena ranks them on the prompts that matter to the workload. **orq-arena** runs a round-robin tournament over a pool of models, has an LLM jury compare every pair of answers blind, and produces a chess-style Elo ladder with confidence intervals. Three parts make up every run: * **Candidates**: the model pool under test, any size from two upwards, called through the [**AI Gateway**](/docs/ai-gateway/using-the-router) so one API key covers every provider * **Jury**: a panel of judge models that compares the two answers, scoring each pair twice with the seats swapped * **Rating**: a Bradley-Terry fit over every judged round, reported with bootstrapped 95% confidence intervals Use it to pick a default model for a product, re-rank the pool when a new model ships, generate pairwise preference data, check whether reasoning helps on a given workload, or choose the strong and economical pair for the [**Smart Router**](/docs/ai-gateway/smart-router). A judge that votes differently when only the seat order changes abstains for that round, and the flip is recorded. A round needs at least two surviving votes, a configurable threshold, and those votes must reach a strict plurality. Rounds that fall short are recorded as **inconclusive** and never reach the rating, so a small run reports fewer rated rounds rather than a ranking built on coin flips. This page covers installing Arena, running a benchmark, and reading the results. For every command and flag, every `orq_arena.yaml` key, and the full scoring methodology, see the [**orq-arena** documentation](https://orq-ai.github.io/orq-arena/). ## Prerequisites Python 3.10 or later, [uv](https://docs.astral.sh/uv/getting-started/installation/), Git, and an **Orq.ai** workspace. Config validation requires at least two candidates and a non-empty judge panel, so enable the models for both before running; the shipped config expects its full eight-model pool and three judges. Arena is an open-source CLI installed from GitHub rather than a published package: ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} git clone https://github.com/orq-ai/orq-arena.git cd orq-arena uv tool install . cp .env.example .env ``` Add a workspace API key to `.env`, created per the [API keys guide](/docs/ai-studio/organization/api-keys): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} ORQ_API_KEY=your-orq-api-key ``` Every candidate call, judge call, and preflight probe routes through the **AI Gateway** with this one key. A variable already set in the shell wins over the file. Although `uv tool install` puts `orq-arena` on the PATH globally, it reads `.env`, the config, and the prompts file relative to the working directory. Run it from the cloned repository, or pass absolute paths to `--config` and `--prompts`. ## First benchmark run ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} orq-arena run --config orq_arena.yaml ``` The shipped config carries an eight-model pool and a three-judge panel. Every pair of candidates meets once in a full round-robin, both answers stream through the router, and the jury votes in both seat orders. Rounds are written to the battle log as they resolve, so an interrupted run keeps every round it already paid for, though re-running starts a fresh tournament rather than resuming: `run` refuses to overwrite a non-empty log unless `--overwrite` is passed. At the end, the standings print and the HTML report is written next to the log. Before any of that, the preflight prints the exact call counts and a RUN PLAN table, one row per candidate and judge with its call count, catalog price, and projected cost. It then stops on a confirmation carrying both figures: ``` Proceed (≈ $11.87, up to $36.70 with retries)? [y/N]: ``` Nothing has been spent at that point beyond a few tiny probe calls. The projection assumes every response hits its token cap, so a typical run costs less; the worst case adds a retry on every stream and a stand-in judge on every judge call. Neither is a hard cap, since prompt tokens are estimated from characters. Scripted runs need `-y`, since a pipe cannot answer the confirmation. Pool size drives cost quadratically. An N-model pool runs N×(N-1)/2 matches, so eight models is 28 matches and sixteen is 120. Trimming the pool saves far more than trimming rounds. No API key yet? A recorded run is committed in the repository. `orq-arena report examples/quickstart/battles.jsonl` rebuilds its report with no key and no network calls. ## Choosing the model pool The pool is the `candidates` list in the config passed to `--config`, using **AI Gateway** model IDs. `orq-arena refresh-catalog --show` lists the models enabled in the workspace, grouped by provider. ```yaml orq_arena.yaml theme={"theme":{"light":"github-light","dark":"github-dark"}} candidates: # the model pool: any size >= 2 - model_id: anthropic/claude-sonnet-4-6 - model_id: openai/gpt-5.4 - model_id: deepseek/deepseek-chat - model_id: google/gemini-3.5-flash reasoning: { thinking: { type: disabled } } # per-model overrides inline judges: # the jury; every pair judged in both seat orders - anthropic/claude-haiku-4-5-20251001 - google/gemini-2.5-flash-lite - openai/gpt-5.4-nano match: max_rounds: 5 # prompts judged per match ``` `reasoning` fields are forwarded to the router verbatim, so each provider takes its own shape; the [configuration reference](https://orq-ai.github.io/orq-arena/configuration/) lists the per-provider recipes. The repository also ships preset pools under `configs/` for frontier, budget, and sixteen-model fields, plus `configs/reasoning_arena.yaml`, the thinking-on counterpart to the thinking-off default, for measuring whether reasoning earns its cost. Judges from the same provider family as a candidate favor their own family's prose, and seat swapping does not correct for it. The preflight warns when it detects an overlap. A jury drawn entirely from families outside the pool is the clean setup, worth arranging before publishing a ranking. ## Supplying prompts The shipped `prompts/starter.jsonl` is a 30-prompt smoke test sized to exercise every mechanism, not to defend a ranking. Swap it with `--prompts`. One JSON object per line. `prompt` is the only required field; `category` is optional and feeds per-category ratings. Any other keys ride along into the battle log, so results can be joined back to the source data. ```json prompts.jsonl theme={"theme":{"light":"github-light","dark":"github-dark"}} {"prompt": "Write a Python function that finds the longest palindromic substring.", "category": "code"} {"prompt": "Summarize the key trade-offs between SQL and NoSQL for a startup.", "category": "reasoning"} ``` ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} orq-arena run --config orq_arena.yaml --prompts prompts.jsonl ``` Pass `orq:` to run over a [Dataset](/docs/ai-studio/optimize/datasets) from the workspace with the same API key, nothing to export. ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} orq-arena run --config orq_arena.yaml --prompts orq:my_dataset_id ``` Each datapoint's last `user` message becomes a prompt, with `{{var}}` placeholders filled from its `inputs`. The manifest records the dataset's ID, name, and studio URL, and the report links it by name, so a shared report traces back to the data it ran on. `match.max_rounds` caps how many prompts each match judges. With the default of `5` against the 30-prompt starter bank, every match sees a seeded random slice of five. Pass `--rounds 30` to use every prompt in every match. ## Where results land Every run writes three files, named after the battle log and written beside it (the working directory by default, or wherever `--output` points): `battles.jsonl` (one line per judged round, with both responses, per-judge votes, and token accounting), `battles.run.json` (the manifest: config, content hashes, panel, seed, agreement stats, no credentials), and `battles.report.html` (a single-file report, no server or external assets, safe to share). Everything downstream works from the battle log alone. `orq-arena report` rebuilds the page, `annotate` renders a blind version for human raters and `anchor` scores their votes against the panel, and `rejudge` re-scores the recorded responses with a different jury for judge tokens only, reporting the rank correlation against the original. That last one is the direct test of whether a ranking depends on who judged it. ## Reading the report The report opens on a verdict, then the Elo ladder with error bars, a quality-versus-cost value map, speed, and dollar spend. orq-arena HTML report headed "top spot not resolved at this size", reporting that gemini-3.5-flash leads claude-sonnet-4-6 but 140 rounds cannot separate them, above a leaderboard of eight models with ELO, 95% confidence interval, a shared-scale interval bar, length-adjusted rating, and average answer length. Two columns carry most of the meaning: * **95% CI** comes from a seeded 1000-resample bootstrap, plotted on one shared scale. The intervals are marginal, so read them one at a time: two of them overlapping does not mean the two models are tied, or even that they are indistinguishable. * **Length-adj.** prices out the jury's preference for longer answers. A large gap between a model's rating and its length-adjusted rating means verbosity, not quality, was doing the separating. The column appears only when that preference is identified: the fit converged, and a bootstrap interval over the same rounds excludes zero. When the top two cannot be separated, the report says so and reports how often the leader came out ahead across resamples, instead of declaring a winner the data does not support. ## How far to trust a run Every run publishes the numbers needed to challenge its own ranking: mean judge agreement, chance-corrected kappa, and each judge's flip rate between seat orders. Three things are worth knowing before quoting a result: * **A failed stream never becomes a verdict.** One retry per side, then the round is voided: logged, but never judged or rated. Truncated answers are judged as-is and flagged, so the jury sees what a reader would see. * **Small runs produce wide intervals**, and that is the honest output. The top spot counts as separated only when the bootstrapped interval on the *difference* between the top two excludes zero. * **The defaults are a smoke test.** A ranking worth defending takes a real prompt set, hundreds of rounds, and judges from families outside the pool. The [methodology reference](https://orq-ai.github.io/orq-arena/methodology/) covers the Bradley-Terry fit, the bootstrap, length control, and the human-anchor workflow in full. ## After the run Arena keeps its results in the three local files rather than writing them back to **Orq.ai**. Candidates and judges share one router client, so every call a tournament makes is recorded in [Traces](/docs/ai-studio/observability/traces) alongside the workspace's other **AI Gateway** traffic, with its own latency, token usage, and cost. Share the HTML report, then act on the ranking: pin the winner in a [Deployment](/docs/ai-studio/ai-engineering/deployments), or put the strongest and most economical models into a [**Smart Router**](/docs/ai-gateway/smart-router) pool and let it choose per request. ## Going further Install, credentials, first tournament, and troubleshooting. Every command and flag with its expected output. Every `orq_arena.yaml` key, its type and default, plus the prompts file format. Example runs, preset model pools, and the implementation. MIT licensed.