Align an Evaluator with Human Judgement

---
name: align-evaluator
description: Align a binary Pass/Fail LLM-as-a-judge evaluator in orq.ai with human judgement, using only the orq MCP. Use when a boolean evaluator disagrees with human verdicts or gives different answers on the same input. Measures the evaluator's self-consistency by re-running it, collects human labels through orq Annotations, and proposes a rewritten evaluator prompt. Requires no extra scripts and no local Python.
---

# Align Evaluator

Make a Pass/Fail LLM-as-a-judge evaluator agree with a human. Everything runs
through the orq MCP.

**Scope**: this skill aligns boolean Pass/Fail evaluators. A 1-5 rating scale
follows the same principle but is not covered here.

**Hard rule**: the human supplies every label. Never label traces yourself,
and never infer what the user "would have said". Aligning an LLM evaluator to
another LLM's opinion produces a confident result that means nothing.

## What you need

- The orq MCP connected (`claude mcp list`)
- The evaluator's name, or its id (open it in orq, **View code**, copy `id="01..."`)
- A project with traces the evaluator has already scored

No local setup.

## The flow

Five steps. Three are human gates (Steps 2, 3, and 5). Do not skip a gate.

### 1. Confirm the evaluator and pull its traces

If the user gave a name rather than an id, resolve it to an id first, then:

```
get_llm_eval(id)                                   → prompt, model, output type
list_traces(project_id: ..., evaluator_key: ...)   → traces it has scored
```

Stop if `output_type` is not boolean. Read the evaluator's prompt back in one
sentence so the user can confirm it is the right evaluator.

Each trace carries an `evaluations[]` array whose entries have an
`annotator.kind` of `llm`, `code`, or `human`, so evaluator verdicts and human
labels arrive from the same call.

Report how many scored traces exist. Fewer than 10 is thin; say so.

### 2. Measure self-consistency  ⟵ GATE

Re-run the evaluator on the same traces to see whether it holds its verdict. A
trace *flips* when the repeats do not all agree. A single pass reveals nothing,
so repeats are the method rather than an optimisation.

**Before running, state plainly: `traces × repeats` calls, and roughly what that
costs.** Derive the per-call cost from the evaluator spans already present in the
traces rather than quoting a published rate. Then wait for explicit approval.
This is the most expensive step and users are routinely surprised by it.

A reasonable default is **10 traces × 5 repeats = 50 calls**. Below 5 repeats the
result is noise. Offer 8 repeats if the user wants a firmer result and accepts
the extra calls.

Re-run the evaluator with `invoke_model`, using its own prompt with its template
variables filled from each trace. **Temperature must be above zero** or there is
no variance to observe and every trace will look stable.

Then report:

- how many traces flipped, out of how many
- the Pass/Fail split for each flipped trace
- which traces were unanimous

**State the limits every time, without softening them:**

- A flip means the case is genuinely ambiguous. It does **not** mean the
  evaluator is wrong.
- A stable trace is **not** evidence of correctness. An evaluator can be
  confidently and consistently wrong, and this method cannot see that by
  construction: a consistently wrong evaluator never flips.
- **Do not rank by flip count.** How *often* something flipped does not track how
  ambiguous it is. Treat flipping as binary: it flipped, or it did not.
- Many genuinely contested cases never flip at all, so this finds some problems
  and misses others.

Never present a low flip-rate as a clean bill of health. A zero-flip run still
proceeds to human labels.

### 3. Record human verdicts  ⟵ GATE

The user labels in the orq UI.

If the project has no Annotation defined, they create one first:
**sidebar → Optimization → Annotations → + Annotation**, type **Categorical**,
with options matching the evaluator's verdict space (`Pass` / `Fail`). Once
created it appears on every chat completion and responses span in the project
automatically.

They then open each trace and record their own verdict. Tell them to answer the
same question the evaluator was asked and to ignore what the evaluator decided.
Agreement is signal too. Ask them to label a spread of traces, not only the ones
that flipped.

### 4. Compare

Read the labels back with `list_traces(evaluator_key: "<annotation key>")` and
compare the `annotator.kind: "human"` entries against the evaluator's.

Present it as a table: evaluator verdict, human verdict, agree or disagree. Call
out every disagreement, and note whether the evaluator was stable on those
traces. A stable verdict that disagrees with the human is a prompt problem, not
an instability problem.

### 5. Propose a rewrite  ⟵ GATE

Look at the disagreements and work out what rule would have produced the human's
answers. Usually the prompt is silent on a distinction the human is applying: for
example, whether a reply that is correct but does not actually help the customer
counts as resolving their question.

Write a revised prompt that:

- keeps every template variable exactly as-is
- states the criterion the disagreements revealed
- remains a single Pass/Fail question

Show the user the old prompt, the new prompt, and one line per change naming the
disagreement that drove it. **Then stop.**

Only after they approve, and only in the form they choose:

```
create_llm_eval(...)   → a new evaluator, original untouched
update_llm_eval(id)    → edits the existing one in place
```

Default to creating a new one. Never call either without explicit approval.

## Final summary

Tell the user:

- what changed in the prompt and why, tied to specific disagreements
- how many traces this was based on
- **the blind spot**: alignment was measured only on traces that flipped or were
  labelled, so a consistently-wrong evaluator would not have surfaced. Suggest
  re-running periodically and labelling a few stable traces each time

## Practical notes

- **Human labels are not perfectly stable either.** The same person can label the
  same reply differently on different days. When that happens, treat it as
  evidence the criterion is genuinely ambiguous and make the rewrite address it,
  rather than treating one label as a mistake.
- **Model slugs take the form `<provider>/<model>`**, for example
  `anthropic/claude-haiku-4-5`.
- **PII guardrails can replace identifiers before the model sees them.** If a
  workspace redacts PII, order numbers and similar identifiers are swapped for
  placeholders, and any tool looking them up returns nothing. Check for a
  `pii.redact` span if replies look inexplicably empty.
- **Reasoning models may ignore the requested temperature**, which removes the
  variance this method depends on. Prefer a non-reasoning model for the evaluator.