Skip to main content
Public leaderboards rank models on someone else’s data. Arena ranks them on the prompts that matter to the workload. orq-arena runs a round-robin tournament over a pool of models, has an LLM jury compare every pair of answers blind, and produces a chess-style Elo ladder with confidence intervals. Three parts make up every run:
  • Candidates: the model pool under test, any size from two upwards, called through the AI Gateway so one API key covers every provider
  • Jury: a panel of judge models that compares the two answers, scoring each pair twice with the seats swapped
  • Rating: a Bradley-Terry fit over every judged round, reported with bootstrapped 95% confidence intervals
Use it to pick a default model for a product, re-rank the pool when a new model ships, generate pairwise preference data, check whether reasoning helps on a given workload, or choose the strong and economical pair for the Smart Router.
A judge that votes differently when only the seat order changes abstains for that round, and the flip is recorded. A round needs at least two surviving votes, a configurable threshold, and those votes must reach a strict plurality. Rounds that fall short are recorded as inconclusive and never reach the rating, so a small run reports fewer rated rounds rather than a ranking built on coin flips.
This page covers installing Arena, running a benchmark, and reading the results. For every command and flag, every orq_arena.yaml key, and the full scoring methodology, see the orq-arena documentation.

​
Prerequisites

Python 3.10 or later, uv, Git, and an Orq.ai workspace. Config validation requires at least two candidates and a non-empty judge panel, so enable the models for both before running; the shipped config expects its full eight-model pool and three judges. Arena is an open-source CLI installed from GitHub rather than a published package:
Add a workspace API key to .env, created per the API keys guide:
Every candidate call, judge call, and preflight probe routes through the AI Gateway with this one key. A variable already set in the shell wins over the file.
Although uv tool install puts orq-arena on the PATH globally, it reads .env, the config, and the prompts file relative to the working directory. Run it from the cloned repository, or pass absolute paths to --config and --prompts.

​
First benchmark run

The shipped config carries an eight-model pool and a three-judge panel. Every pair of candidates meets once in a full round-robin, both answers stream through the router, and the jury votes in both seat orders. Rounds are written to the battle log as they resolve, so an interrupted run keeps every round it already paid for, though re-running starts a fresh tournament rather than resuming: run refuses to overwrite a non-empty log unless --overwrite is passed. At the end, the standings print and the HTML report is written next to the log. Before any of that, the preflight prints the exact call counts and a RUN PLAN table, one row per candidate and judge with its call count, catalog price, and projected cost. It then stops on a confirmation carrying both figures:
Nothing has been spent at that point beyond a few tiny probe calls. The projection assumes every response hits its token cap, so a typical run costs less; the worst case adds a retry on every stream and a stand-in judge on every judge call. Neither is a hard cap, since prompt tokens are estimated from characters. Scripted runs need -y, since a pipe cannot answer the confirmation.
Pool size drives cost quadratically. An N-model pool runs N×(N-1)/2 matches, so eight models is 28 matches and sixteen is 120. Trimming the pool saves far more than trimming rounds.
No API key yet? A recorded run is committed in the repository. orq-arena report examples/quickstart/battles.jsonl rebuilds its report with no key and no network calls.

​
Choosing the model pool

The pool is the candidates list in the config passed to --config, using AI Gateway model IDs. orq-arena refresh-catalog --show lists the models enabled in the workspace, grouped by provider.
orq_arena.yaml
reasoning fields are forwarded to the router verbatim, so each provider takes its own shape; the configuration reference lists the per-provider recipes. The repository also ships preset pools under configs/ for frontier, budget, and sixteen-model fields, plus configs/reasoning_arena.yaml, the thinking-on counterpart to the thinking-off default, for measuring whether reasoning earns its cost.
Judges from the same provider family as a candidate favor their own family’s prose, and seat swapping does not correct for it. The preflight warns when it detects an overlap. A jury drawn entirely from families outside the pool is the clean setup, worth arranging before publishing a ranking.

​
Supplying prompts

The shipped prompts/starter.jsonl is a 30-prompt smoke test sized to exercise every mechanism, not to defend a ranking. Swap it with --prompts.
One JSON object per line. prompt is the only required field; category is optional and feeds per-category ratings. Any other keys ride along into the battle log, so results can be joined back to the source data.
prompts.jsonl
match.max_rounds caps how many prompts each match judges. With the default of 5 against the 30-prompt starter bank, every match sees a seeded random slice of five. Pass --rounds 30 to use every prompt in every match.

​
Where results land

Every run writes three files, named after the battle log and written beside it (the working directory by default, or wherever --output points): battles.jsonl (one line per judged round, with both responses, per-judge votes, and token accounting), battles.run.json (the manifest: config, content hashes, panel, seed, agreement stats, no credentials), and battles.report.html (a single-file report, no server or external assets, safe to share). Everything downstream works from the battle log alone. orq-arena report rebuilds the page, annotate renders a blind version for human raters and anchor scores their votes against the panel, and rejudge re-scores the recorded responses with a different jury for judge tokens only, reporting the rank correlation against the original. That last one is the direct test of whether a ranking depends on who judged it.

​
Reading the report

The report opens on a verdict, then the Elo ladder with error bars, a quality-versus-cost value map, speed, and dollar spend. orq-arena HTML report headed "top spot not resolved at this size", reporting that gemini-3.5-flash leads claude-sonnet-4-6 but 140 rounds cannot separate them, above a leaderboard of eight models with ELO, 95% confidence interval, a shared-scale interval bar, length-adjusted rating, and average answer length. Two columns carry most of the meaning:
  • 95% CI comes from a seeded 1000-resample bootstrap, plotted on one shared scale. The intervals are marginal, so read them one at a time: two of them overlapping does not mean the two models are tied, or even that they are indistinguishable.
  • Length-adj. prices out the jury’s preference for longer answers. A large gap between a model’s rating and its length-adjusted rating means verbosity, not quality, was doing the separating. The column appears only when that preference is identified: the fit converged, and a bootstrap interval over the same rounds excludes zero.
When the top two cannot be separated, the report says so and reports how often the leader came out ahead across resamples, instead of declaring a winner the data does not support.

​
How far to trust a run

Every run publishes the numbers needed to challenge its own ranking: mean judge agreement, chance-corrected kappa, and each judge’s flip rate between seat orders. Three things are worth knowing before quoting a result:
  • A failed stream never becomes a verdict. One retry per side, then the round is voided: logged, but never judged or rated. Truncated answers are judged as-is and flagged, so the jury sees what a reader would see.
  • Small runs produce wide intervals, and that is the honest output. The top spot counts as separated only when the bootstrapped interval on the difference between the top two excludes zero.
  • The defaults are a smoke test. A ranking worth defending takes a real prompt set, hundreds of rounds, and judges from families outside the pool.
The methodology reference covers the Bradley-Terry fit, the bootstrap, length control, and the human-anchor workflow in full.

​
After the run

Arena keeps its results in the three local files rather than writing them back to Orq.ai. Candidates and judges share one router client, so every call a tournament makes is recorded in Traces alongside the workspace’s other AI Gateway traffic, with its own latency, token usage, and cost. Share the HTML report, then act on the ranking: pin the winner in a Deployment, or put the strongest and most economical models into a Smart Router pool and let it choose per request.

​
Going further

Getting started

Install, credentials, first tournament, and troubleshooting.

CLI reference

Every command and flag with its expected output.

Configuration

Every orq_arena.yaml key, its type and default, plus the prompts file format.

Source on GitHub

Example runs, preset model pools, and the implementation. MIT licensed.