LLM as Judge: How to Use a Model to Evaluate Your Agents Without Fooling Yourself

Why do enterprises need model-based evaluation at all?

Because agent quality degrades quietly and in ways a user does not report. A change to a chunk size, a new tool implementation, a different model, an upstream data change — each can shift output quality without changing anything obvious in the interface. The customer notices three months later, and by then the cause has been lost.

Human review does not scale to this. In a Singapore or Hong Kong SAR deployment with thousands of agent runs a week, a subject expert can look at perhaps a few dozen — enough to form an impression, not enough to detect a regression, because a few percent regression is invisible in a sample of thirty.

Assessor type	What it is good at	Where it is weak	Typical use
Deterministic check	Format, schema validity, required field presence, exact-match facts, latency budget	Anything requiring interpretation	Run on every single run; cheap and always on
Ground-truth verification	Answers where the correct value is known independently	Judgement, tone, strategy	Maths, extraction, lookups, contract clause matching
Human expert review	Anything consequential or ambiguous; calibration of everything else	Volume, consistency, reproducibility	Calibration sample, escalation cases, new task types
Model-based judge	Volume, consistency, regression detection over long periods	Strategic quality, novel task types, anything it was not calibrated on	Every run in regression plus a share of live traffic

The useful architecture layers the four by cost and severity. Deterministic checks and ground-truth verification run first, on every run, and are cheap. The judge runs on everything that survives them, and on a sampled share of live traffic, to catch drift that no fixed set would reveal. The human expert looks at what the judge flags, and at a fixed calibration sample that never changes, so the two stay comparable over time.

When should you use pairwise comparison instead of absolute scoring?

Use pairwise comparison when you are choosing between two candidate approaches — two prompts, two retrieval configurations, two agent architectures. Use absolute rubric scoring when you need a number you can track over time or gate a release on.

Relative judgement is easier and more consistent than absolute judgement, for both models and humans. Asked to score a response on a one-to-five scale, two reviewers will disagree about whether it is a three or a four. Asked which of two responses is better, they usually agree, and the disagreement concentrates on genuinely close cases. Judges behave the same way.

Pairwise comparison has a specific implementation requirement: you must randomise the position of the candidates and, where the ordering is decided by the model rather than fixed, you must run both orderings and require agreement. A judge that prefers the first candidate will produce a biased result unless you correct for it, and the bias is large enough to reverse a conclusion in close cases.

The second requirement is a reference anchor. Pure pairwise comparison gives you an ordering, not an absolute position, which means it cannot tell you whether the winner is good. In practice you keep a small set of reference examples with known human scores and include them in the comparison, so the ordering is anchored to something external.

The third requirement is consistency across runs. Because pairwise judgements are sensitive to small changes in presentation, the comparison protocol — the instruction, the order, the anchor set — must be frozen and versioned. Changing the protocol invalidates previous comparisons even if the underlying candidates did not change.

How do you calibrate a judge against human labels?

Calibration is defined as the process of measuring the agreement between a model-based judge and independent human labels on a labelled sample, and re-running that measurement whenever the judge, the rubric or the underlying model changes.

The procedure is straightforward and almost nobody does it.

1. Build a labelled sample. A few hundred cases is enough to start; stratify across task types and across quality levels, including deliberately weak cases, because a judge that only sees good outputs will look better than it is.
2. Measure agreement, and report it as part of the result. Report per-criterion agreement, not just one aggregate. A judge with strong agreement on factual criteria and weak agreement on tone is a useful judge for the first and not for the second.
3. Set the scope of what the judge may be used for. Write down the task types it is calibrated on and treat everything else as uncalibrated. This prevents the most damaging misuse: a judge applied to a new task type with no recalibration.
4. Re-run on a schedule and on change. After any change to the rubric, the judge model, the candidate model or the retrieval configuration.

Report the calibration figure alongside every evaluation result. An evaluation that publishes a score without an agreement rate asks the reader to treat the number as more precise than it is, and in an enterprise setting that number ends up in a board paper.

Where does model-based judging stop being the right tool?

Four cases.

First, when the task is strategic rather than operational. Whether a negotiation opening is the right one, whether a risk escalation was proportionate — these are judgements where disagreement is a matter of perspective, and a rubric-trained judge systematically favours the mainstream. Those need human review.

Second, when the correct answer is unknown and the stakes are high. If nobody can say what the right answer is, a judge's confidence is not evidence of anything.

Third, when the evaluated model produces text the judge is unusually fluent about. Judges are more reliable on structured, fact-shaped work — extraction, summarisation against a source, compliance checks, classification — and less reliable on long free-form prose.

Fourth, when the volume does not justify it. If you run forty agent tasks a week, a human expert can read all of them, and the judge adds cost, calibration work and a second thing to maintain. The honest recommendation for a low-volume deployment is a good regression set and a person reading it.

Where we draw the same boundary in delivery: we do not sell a judge as a compliance attestation. It is a measurement instrument, and an auditable statement about an agent's behaviour needs deterministic checks, traceable evidence and human sign-off. A model-based score is one input to that statement.