> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. # Align an Evaluator with Human Judgement > Find where an LLM-as-a-judge Evaluator disagrees with human reviewers, and rewrite its prompt so its verdicts match. An Evaluator used to measure how a system performs is only as trustworthy as the Evaluator itself. An **LLM-as-a-judge Evaluator** grades agent output at a scale no human can match, but it is still an LLM, so it fails in two ways: it returns different verdicts for the same input (it is unstable), and it applies a standard that is not quite the intended one (it is misaligned). Inconsistent verdicts are a warning sign. When an Evaluator disagrees with itself on the same input, its scores cannot be trusted to track real quality, and every metric built on top of them inherits that noise. This cookbook measures an Evaluator's stability, compares its verdicts against human judgement on real **Traces**, and rewrites its prompt so the two agree. It uses the **Orq.ai** MCP throughout, so no local setup is required. **TL;DR** * **Measure self-consistency**: re-run the Evaluator on the same inputs and count how often it contradicts itself * **Collect human verdicts**: record ground truth on **Traces** with **Annotations** * **Compare and rewrite**: turn the disagreements into a revised Evaluator prompt **What this cannot do**: prove the Evaluator is correct. See [Limits](#limits). ## What you'll build A revised **Evaluator** prompt whose Pass/Fail verdicts match human judgement on real **Traces**, produced by measuring the original Evaluator's self-consistency and comparing its verdicts against recorded human labels. ## What you'll learn * Measure an **Evaluator**'s self-consistency by re-running it on the same inputs * Record human verdicts on **Traces** as **Annotations** * Compare Evaluator and human verdicts, then rewrite the prompt to close the gaps ## Prerequisites * The **Orq.ai** MCP connected to a coding assistant. See [Orq MCP](/docs/ai-studio/integrations/code-assistants/orq-mcp) * A boolean Pass/Fail Evaluator attached to an **Agent**, with at least 10 scored **Traces**. See [Evaluators](/docs/ai-studio/optimize/evaluators) * Access to create an **Annotation** in the project, under **Optimization** in the sidebar. See [Annotation Queues](/docs/ai-studio/observability/annotation-queues) This cookbook uses boolean Pass/Fail Evaluators. Rating-scale Evaluators follow the same principle but are not covered here. ## Install the skill The workflow is packaged as a skill. Save the file below as `.claude/skills/align-evaluator/SKILL.md` in the project where the coding assistant runs. The assistant then drives all five steps, pausing for approval before the expensive run in Step 2 and before anything is created or changed. The steps that follow show what the skill does on a real dataset. The prompts in each step also work without the skill installed. ````markdown theme={"theme":{"light":"github-light","dark":"github-dark"}} --- name: align-evaluator description: Align a binary Pass/Fail LLM-as-a-judge evaluator in orq.ai with human judgement, using only the orq MCP. Use when a boolean evaluator disagrees with human verdicts or gives different answers on the same input. Measures the evaluator's self-consistency by re-running it, collects human labels through orq Annotations, and proposes a rewritten evaluator prompt. Requires no extra scripts and no local Python. --- # Align Evaluator Make a Pass/Fail LLM-as-a-judge evaluator agree with a human. Everything runs through the orq MCP. **Scope**: this skill aligns boolean Pass/Fail evaluators. A 1-5 rating scale follows the same principle but is not covered here. **Hard rule**: the human supplies every label. Never label traces yourself, and never infer what the user "would have said". Aligning an LLM evaluator to another LLM's opinion produces a confident result that means nothing. ## What you need - The orq MCP connected (`claude mcp list`) - The evaluator's name, or its id (open it in orq, **View code**, copy `id="01..."`) - A project with traces the evaluator has already scored No local setup. ## The flow Five steps. Three are human gates (Steps 2, 3, and 5). Do not skip a gate. ### 1. Confirm the evaluator and pull its traces If the user gave a name rather than an id, resolve it to an id first, then: ``` get_llm_eval(id) → prompt, model, output type list_traces(project_id: ..., evaluator_key: ...) → traces it has scored ``` Stop if `output_type` is not boolean. Read the evaluator's prompt back in one sentence so the user can confirm it is the right evaluator. Each trace carries an `evaluations[]` array whose entries have an `annotator.kind` of `llm`, `code`, or `human`, so evaluator verdicts and human labels arrive from the same call. Report how many scored traces exist. Fewer than 10 is thin; say so. ### 2. Measure self-consistency ⟵ GATE Re-run the evaluator on the same traces to see whether it holds its verdict. A trace *flips* when the repeats do not all agree. A single pass reveals nothing, so repeats are the method rather than an optimisation. **Before running, state plainly: `traces × repeats` calls, and roughly what that costs.** Derive the per-call cost from the evaluator spans already present in the traces rather than quoting a published rate. Then wait for explicit approval. This is the most expensive step and users are routinely surprised by it. A reasonable default is **10 traces × 5 repeats = 50 calls**. Below 5 repeats the result is noise. Offer 8 repeats if the user wants a firmer result and accepts the extra calls. Re-run the evaluator with `invoke_model`, using its own prompt with its template variables filled from each trace. **Temperature must be above zero** or there is no variance to observe and every trace will look stable. Then report: - how many traces flipped, out of how many - the Pass/Fail split for each flipped trace - which traces were unanimous **State the limits every time, without softening them:** - A flip means the case is genuinely ambiguous. It does **not** mean the evaluator is wrong. - A stable trace is **not** evidence of correctness. An evaluator can be confidently and consistently wrong, and this method cannot see that by construction: a consistently wrong evaluator never flips. - **Do not rank by flip count.** How *often* something flipped does not track how ambiguous it is. Treat flipping as binary: it flipped, or it did not. - Many genuinely contested cases never flip at all, so this finds some problems and misses others. Never present a low flip-rate as a clean bill of health. A zero-flip run still proceeds to human labels. ### 3. Record human verdicts ⟵ GATE The user labels in the orq UI. If the project has no Annotation defined, they create one first: **sidebar → Optimization → Annotations → + Annotation**, type **Categorical**, with options matching the evaluator's verdict space (`Pass` / `Fail`). Once created it appears on every chat completion and responses span in the project automatically. They then open each trace and record their own verdict. Tell them to answer the same question the evaluator was asked and to ignore what the evaluator decided. Agreement is signal too. Ask them to label a spread of traces, not only the ones that flipped. ### 4. Compare Read the labels back with `list_traces(evaluator_key: "")` and compare the `annotator.kind: "human"` entries against the evaluator's. Present it as a table: evaluator verdict, human verdict, agree or disagree. Call out every disagreement, and note whether the evaluator was stable on those traces. A stable verdict that disagrees with the human is a prompt problem, not an instability problem. ### 5. Propose a rewrite ⟵ GATE Look at the disagreements and work out what rule would have produced the human's answers. Usually the prompt is silent on a distinction the human is applying: for example, whether a reply that is correct but does not actually help the customer counts as resolving their question. Write a revised prompt that: - keeps every template variable exactly as-is - states the criterion the disagreements revealed - remains a single Pass/Fail question Show the user the old prompt, the new prompt, and one line per change naming the disagreement that drove it. **Then stop.** Only after they approve, and only in the form they choose: ``` create_llm_eval(...) → a new evaluator, original untouched update_llm_eval(id) → edits the existing one in place ``` Default to creating a new one. Never call either without explicit approval. ## Final summary Tell the user: - what changed in the prompt and why, tied to specific disagreements - how many traces this was based on - **the blind spot**: alignment was measured only on traces that flipped or were labelled, so a consistently-wrong evaluator would not have surfaced. Suggest re-running periodically and labelling a few stable traces each time ## Practical notes - **Human labels are not perfectly stable either.** The same person can label the same reply differently on different days. When that happens, treat it as evidence the criterion is genuinely ambiguous and make the rewrite address it, rather than treating one label as a mistake. - **Model slugs take the form `/`**, for example `anthropic/claude-haiku-4-5`. - **PII guardrails can replace identifiers before the model sees them.** If a workspace redacts PII, order numbers and similar identifiers are swapped for placeholders, and any tool looking them up returns nothing. Check for a `pii.redact` span if replies look inexplicably empty. - **Reasoning models may ignore the requested temperature**, which removes the variance this method depends on. Prefer a non-reasoning model for the evaluator. ```` ## Step 1: Confirm the Evaluator Ask the assistant to fetch the Evaluator and the **Traces** it has scored. Refer to the Evaluator by name; the assistant resolves it to an id. Pass the id directly (`01ABC...`) only if the name is ambiguous. ```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}} Fetch the "cookbook-answer-quality" evaluator and list the traces it has scored in this project. ``` The assistant looks the Evaluator up by name, then calls `get_llm_eval` and `list_traces`. Every trace carries an `evaluations[]` array, and each entry records who produced it: ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "evaluation_type": "evaluator", "annotator": { "kind": "llm" }, "passed": false } ``` The `annotator.kind` field distinguishes `llm`, `code`, and `human` verdicts, so Evaluator scores and human labels arrive together. ## Step 2: Measure self-consistency A trustworthy Evaluator returns the same verdict every time it grades the same input. This step tests that by grading each trace several times and watching for **flips**. A trace *flips* when the repeats disagree with each other: some come back Pass and others Fail. A flip means the Evaluator cannot make up its mind on that case. ```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}} Re-run this evaluator 5 times on each of 10 of those traces (50 calls total) and report which traces flipped. ``` Five repeats per trace is the floor; fewer cannot reveal an inconsistency. Ten traces is a teaching-sized sample, so treat any result as illustrative. On a real Evaluator, use 30 traces or more. This is the expensive step. Cost is `traces × repeats`: 10 traces at 5 repeats is 50 Evaluator calls, and 200 traces at 8 repeats is 1,600. Confirm the projected call count before running. Set the temperature above zero. At temperature 0 the Evaluator has no room to vary, so every trace looks stable whether it is or not. Reasoning models may override the temperature, so prefer a non-reasoning model for the Evaluator. The result is a count of how many traces flipped, for example: ``` 0 of 10 traces flipped ``` No flips means the Evaluator was consistent: it returned the same verdict on all five repeats of every trace. That is useful to know, but it is not proof the Evaluator is correct. A consistent Evaluator can still apply the wrong standard, and it will apply it the same way every time. Whatever the flip count, the next step is to compare the Evaluator against a human. ## Step 3: Record human verdicts Create an **Annotation** so reviewers can record their own verdict next to the Evaluator's. In the sidebar, go to **Optimization** **Annotations** and click **+ Annotation**: * **Key**: for example `human_verdict` * **Title**: for example `human_verdict` * **Type**: Categorical * **Options**: `Pass` and `Fail`, matching the Evaluator's verdict space Add annotation dialog with Key and Title set to human_verdict, Type set to Categorical, and Pass and Fail listed as the two options. Once created, the **Annotation** appears on every chat completion and responses span in the project. No filtering or per-trace configuration is needed. Open the **Traces** view, filter to the **Agent** under review, and record a verdict on each trace from the review panel. Answer the same question the Evaluator was asked, and ignore what the Evaluator decided. Agreement is signal too. Traces list filtered to the acme-support-bot agent, then a Trace opened and given a Pass verdict in the Review panel. Label a spread of traces, not only the ones that looked suspicious. The clearest disagreements often appear on traces the Evaluator was completely consistent about. A stable verdict is not a correct one. ## Step 4: Compare against human verdicts Ask the assistant to read the human labels back and line them up against the Evaluator's own verdicts. ```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}} Compare the human_verdict annotations against the evaluator verdicts on those traces. ``` Human labels return through the same `evaluations[]` array, marked `"annotator": { "kind": "human" }`, so one call returns both sides. Laid out as a table, a result might look like this: | Customer question | Evaluator | Human | | ----------------------------------------------------- | ---------- | -------- | | "Is my winter jacket order still on track?" | Fail (5/5) | **Pass** | | "My running shoes say delivered but I never got them" | Pass (5/5) | **Fail** | | "Is my refund on the espresso machine approved yet?" | Pass (5/5) | Pass | Two things stand out, and both point to the prompt rather than to instability: * **The Evaluator is wrong in both directions.** It fails a reply that was correct and complete, and passes one that resolved nothing. An Evaluator that was merely too strict could be fixed by loosening it; one that errs both ways is applying a vague criterion. * **The repeats missed both cases.** Every verdict above was unanimous across all five repeats, so Step 2 never flagged them. These cases are not ambiguous to the Evaluator, they are simply graded against the wrong standard. This is why self-consistency alone is never enough. ## Step 5: Rewrite the prompt Ask for a revision grounded in the disagreements. ```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}} Based on those disagreements, propose a revised evaluator prompt. Keep the template variables unchanged and show me a diff. ``` Disagreements usually trace back to a distinction the prompt never made. In the example above, the original asked only: ``` Did the reply resolve the customer's question? Answer true for Pass, false for Fail. ``` It said nothing about replies that are policy-correct but leave the customer no better off, which is exactly where the Evaluator and human parted ways. A revision that names that distinction explicitly closes both disagreements: ``` Did the reply resolve the customer's question? A reply passes only if it is correct and leaves the customer able to act on it. A reply that is factually or policy-correct but leaves the customer no better off is a Fail. Answer true for Pass, false for Fail. ``` The added rule flips the winter jacket reply to Pass (correct and complete) and the running shoes reply to Fail (accurate but resolved nothing), matching the human verdicts. Review the proposed prompt, then create a new Evaluator rather than editing the original. The original stays available for comparison. Approve the diff before anything is created. Creating a new Evaluator leaves the original untouched; updating one in place does not. ## Limits This method has a blind spot that follows from its construction. * **A consistently wrong Evaluator never flips.** Self-consistency measures stability, not correctness. An Evaluator that applies the wrong standard every single time is perfectly stable and completely wrong, and no number of repeats will reveal it. * **Flip count is not an ambiguity ranking.** How often an item flipped does not track how contested it is. Treat flipping as binary and do not sort by it. * **Many contested items never flip.** This finds some problems and misses others. A low flip rate means the Evaluator is consistent, nothing more. * **Human labels are not fixed either.** The same reviewer can label the same reply differently on different days. When that happens, the criterion itself is ambiguous, and the rewrite should resolve it rather than treating one label as an error. Re-run periodically, and label a few stable traces each time as a check on the traces the flip signal cannot reach. ## Next steps Configure LLM-as-a-judge and Python Evaluators. Route Traces to reviewers at scale.