Most enterprises treat evaluation as a gate. It has to be a system.
A gate is passed once and then forgotten, which is why the same bad answers keep coming back and the new ones arrive unannounced. This page covers the method we run: a defined target, a managed test set, chosen assessors, and a pipeline that executes continuously rather than before a launch.
Four steps, in order
Target, test set, assessors, execution — each one is a decision, and skipping the first two makes the rest theatre
Four kinds, with edges
LLM-as-a-Judge, classic NLP metrics, code scripts, human annotation — and knowing where each stops
Every conversation
Full-volume quality inspection, not a sample, producing an exception list with a type on every item
Trace down, export out
Visual reports with drill-down to the trace and data export for whoever has to sign it off
Evaluation is a production system, not a checkpoint
The enterprise that only evaluates before launch spends its whole quality budget chasing the failures it already knows about — and never finds the ones it has not met yet.
That is the shape of most evaluation practice we inherit. A test set is written before launch, the results are discussed once, and then the agent goes into production alone. Six months later nobody can say whether it is better or worse than it was, because the measurement stopped when the project stopped.
The alternative is to treat an agent the way a serious business treats a person in a regulated role: it has a job description, a defined standard, a periodic assessment against a maintained set of questions, a record of its failures, and a trigger for escalation when it falls below the line. That is not exotic. It is what "quality assurance" was always supposed to mean before the subject of the assessment started generating its own answers.
Everything below this point is the machinery for that.
What "first-class" means in practice
A named owner
Evaluation is somebody's job, with a schedule and a budget, not a task that appears when something goes wrong.
A maintained test set
The set grows from production reality, and the growth is the point — a static set stops telling you the truth.
A pipeline, not a spreadsheet
Scheduled and repeatable execution with concurrency, throttling, timeout and retry, so results are comparable over time.
A link back to production
When aggregate performance falls below the threshold you set, the system raises the alarm. Evaluation that cannot page anyone is a report.
Evaluation in numbers
Not sampled — sampling misses exactly the tail that matters
Each with a stated boundary, including where a model should not judge
Test set → assess → run → regression
The threshold is yours to set, and we make you justify it
Evaluation is a production system, not a checkpoint
Four steps, in order, every time. Teams that do only steps 3 and 4 find out they cannot tell whether step 1 was ever built properly.
Build the test set
From your own historical cases — the ones that took longest, errored most, or escalated. Not a synthetic set written to be passable.
Choose the assessors
Rule-based where a rule exists, model-as-judge where judgement is required, human where the stakes justify it. Each with a declared boundary.
Run against live traffic
Every conversation assessed, results aggregated, failures clustered by cause rather than counted.
Regression on every change
Prompt, context, model or tool change re-runs the suite before traffic moves. No change ships on a hunch.
Four steps, in order, every time
The value of a method is that it removes the argument about what to measure. The steps are not administrative overhead — each one prevents a different category of wasted evaluation.
Define the effectiveness target
Before any test set is written, you decide what "good" means for this agent, across four dimensions: business outcome, safety and compliance, performance, and cost. An agent serving a regulatory question and an agent summarising news have almost nothing in common, and evaluating them against one definition of correctness is the most common reason evaluation results are disputed. Cost belongs in this list from the start — a quality target that ignores cost produces an estate nobody can afford to run.
Build the test set
The questions become the examination. They must cover the effect targets you just defined, and they must include multi-modal and multi-turn conversation cases, because those are where agents break in ways single-turn questions never reveal. A set of easy questions produces a flattering result and no information.
Design the assessment method
For each question, choose how it will be judged: an LLM-as-a-Judge for subjective or creative answers, traditional NLP metrics where a reference answer exists, a code script where the rule is fixed, or a human annotator where neither a machine nor a metric can be trusted. The rules are configured separately from the test set, which means one set of rules can be applied across several evaluation subjects for comparison.
Execute and analyse
Run the tasks on a schedule, with concurrency, throttling, scheduling strategy and timeout retry configured, then read the report: aggregate statistics, comparison across subjects, drill-down to the trace of any individual case, and export for whoever needs the evidence. A run that cannot be repeated produces a claim rather than a measurement.
Evaluation tasks are reusable objects. A task is named, described, and may carry several evaluation subjects, and tasks can be built from templates so that a standard baseline is available to every team while each business unit keeps its own focus. That is what stops evaluation from being reinvented badly in four different departments.
The test set is the exam, and it is the part teams get wrong
An assessment measures the questions you ask. In most estates the test set is drawn from what the team found easy to write down rather than from what the business actually depends on, which is why scores are high and incidents continue.
What a test set contains
- The data content — the questions and cases themselves, drawn from real usage rather than from a scenario workshop.
- A custom structure — metadata and fields your business cares about, so results can be sliced by region, customer tier, product or risk category.
- Multi-modal cases — images, documents and mixed inputs, because a text-only test set will flatter an agent that has to read a table or a scanned page.
- Multi-turn cases — conversations, not just questions, since most production failures are context loss across turns rather than a wrong first answer.
- Reference answers — where a correct answer exists, so that the traditional metric assessors have something to compare against.
Where the cases come from
- Human construction — by domain experts, slow and expensive, but authoritative on the cases that matter.
- Data synthesis — generated to fill a coverage gap you already know exists.
- Data feedback from production — real conversations, and the source most often overlooked.
- Open evaluation sets — useful as a baseline, rarely sufficient alone.
The most underestimated source
Production feedback is the only source that is guaranteed to stay current with how users actually behave, including the requests nobody in the business thought to write down. It also tells you which questions are being asked repeatedly, which is where retrieval quality and cost both show up most clearly.
The workflow we recommend is unglamorous: take a sample of real conversations each cycle, add the ones that failed or that represent an important capability, keep a reference answer where a correct one exists, and grow the set. It also gives you the regression set for free, because the same set does both jobs.
A note on size. There is no official sample-size standard, and anyone quoting one is quoting their own project. What matters is coverage of the scenarios you care about and enough cases per scenario to notice a change. A small set covering the right cases beats a large set of near-duplicates.
Four kinds of assessor, and the boundary of each
No single judging method is sufficient. The practical question is not which one is best, but which one you trust for which part of the test set — and where the result needs a second opinion before it counts.
| Assessor | Suited to | Boundary — where it stops being trustworthy |
|---|---|---|
| LLM-as-a-Judge | Subjective, creative and open-ended answers where there is no single correct response — tone, relevance, completeness, appropriateness. | The judge is a model, and it inherits the family's blind spots. It should not be the only assessor on a safety or compliance question, and its prompt and version should be pinned like any other production dependency. |
| Traditional NLP metrics | Questions that have a reference answer — similarity, overlap, exactness, faithfulness against a source. | Requires a reference answer, so it cannot assess open-ended judgement. A similarity score on a creative task is a measurement of the wrong thing, and it is easy to optimise against. |
| Code script evaluation | Scenarios with a fixed rule — does the output contain the required field, did the tool call use the right arguments, did the response respect a length or format constraint. | Only sees what it was written to see. The script is a specification, and it can only test the cases its author anticipated — which is why it sits alongside the other three rather than replacing them. |
| Human annotation | Final judgement on contested cases, on the questions that matter most, and on establishing the reference standard for the rest. | Slow and expensive, so it is a scarce resource and should be spent on calibration and on a sample of the automated results, not on the whole set. |
Going past the final answer
For complex business tasks, a correct final answer is not sufficient evidence that the agent is reliable. A task can reach the right destination by accident, or through a sequence of steps that would break the moment the input changed slightly.
Advanced process evaluation therefore looks inside the run: the quality of the task plan, the quality of intermediate outputs rather than only the conclusion, and whether the tool calls were reasonable — the right tool, the right arguments, in a sensible order.
This is where agents most often fail silently. An agent that ends with a correct answer via an unsound path will fail on a different day, and the end-to-end score will not have warned you.
Rules live outside the test set
Evaluation rules are configured independently of the test set, and several evaluation subjects can be assessed in one run. Two practical consequences: the same question set can be used to compare an agent against a different model or against a different prompt template, and a rule set can be applied to a new agent without being rewritten.
A typical release is a comparison, not a number: the candidate configuration against the current production one, on the same set, with the same assessors, so that any difference is attributable to the change rather than to a different day.
The test set is the exam, and it is the part teams get wrong
A test set built by the same person who built the agent, in the same week, is not an exam. It is a rehearsal.
- Every case is a real historical instance, not something written today to be passable.
- Hard cases are over-represented — the distribution of your failures, not of your volume.
- Cases include the ones that should be refused, not only the ones that should be answered.
- Cases carry the expected outcome and the reason it is expected.
- The set is versioned, and adding a case is a reviewed change.
- A case cannot be removed without a recorded reason — otherwise the suite quietly shrinks to what the agent can do.
- The failure rate on your own live traffic is measured, so you know what the set is not covering.
Four kinds of assessor, and the boundary of each
Every assessor has a domain where it is trustworthy. Declaring that boundary up front is what stops model-as-judge quietly grading its own homework.
| Assessor | Use it when | Boundary — where it must not judge |
|---|---|---|
| Rule-based | There is a correct answer and it is machine-checkable: format, field presence, arithmetic, a threshold | Never for quality, tone, adequacy, or anything requiring judgement |
| Model-as-judge | The question is whether an answer is adequate, relevant, well-grounded and appropriately refused | Never as the sole assessor on high-stakes decisions, and never on its own outputs without a reference |
| Human review | The decision is material, the population is small enough to read, and the cost of being wrong is high | Never at full volume — humans cannot sustain it and will degrade into sampling the easy ones |
| Reference comparison | A known-correct answer exists and near-miss matters | Never where there are multiple valid answers; it will reward one phrasing over a better one |
Scanning every conversation, not sampling some of them
This is the capability that most distinguishes a platform built for production from one built for demonstration. Sampling tells you about the conversations you chose to look at. Full inspection tells you what actually happened.
What the inspection does
- Scans the full volume of conversation data — every recorded interaction is checked against the configured rules, not a statistically interesting subset.
- Identifies every non-conforming conversation — each one that fails a criterion is captured, not aggregated away into an average.
- Produces an exception list with a type — every item in the list is annotated with the category of failure, which is what makes the list actionable rather than merely alarming.
- Runs on a configurable cadence — down to hourly scanning, so a regression introduced by a morning deployment is visible the same day.
Bad case classification
Failure categories are what turn an exception list into a work queue. The recurring causes we see, and the work each one implies:
Unreasonable knowledge base segmentation
The retrieved passage was the wrong slice of the right document. This is a knowledge engineering problem, not a model problem, and no amount of prompt work will fix it.
Prompt logic problems
The instructions produce the wrong behaviour — conflicting rules, missing steps, or a system prompt that contradicts the business policy it is meant to carry.
Insufficient intent recognition samples
The utterance never matched a trained intent, so the agent guessed. The remedy is training data, gathered from the conversations that did not match.
Why this is worth building
Users do not file tickets about bad answers. They ask the question again, rephrase it, or give up and phone a colleague. The conversation is still in the record, so the failure is visible to you and invisible to the user — which is why bad-case collection is a function of instrumentation, not of complaint volume.
Once the list exists and the categories are assigned, three things become possible that were not before: the exception list becomes the backlog for knowledge engineering, the intent-recognition gaps become the training-data backlog, and the counts over time become a quality trend you can put in front of a steering committee.
Prompt efficiency as a by-product. Because the inspection sees the actual instructions that produced a failure, it also surfaces where a more efficient instruction would have worked. That feedback is precise enough to be useful to a fine-tuning effort, rather than a general impression that the prompts could be better.
Scanning every conversation, not sampling some of them
Sampling is the right answer for web-scale consumer products. It is the wrong answer for an estate a regulator might ask about. The cost difference is a routing problem, not a tooling problem.
- Scales to any volume
- Cheap to run
- Misses the tail — and the tail is where the incidents are
- Cannot give a rate you can put in a report
- Statistical, which sounds rigorous and is not informative here
- Cost scales with volume, so it needs the routing work as well
- Every failure is caught, including one-in-ten-thousand cases
- Produces a real rate, which is what an auditor asks for
- Enables clustering by cause, not just counting
- Feeds directly back into the test set
Regression, alerting, and the evidence trail
Evaluation earns its cost in the second month, when the release train is running and every change is a candidate regression. These are the mechanisms that make the month survivable.
Regression testing
After a change to the prompt, the model or the retrieval strategy, the same batch of questions is run again. The purpose is not to confirm that the change was an improvement — it is to find what the change broke while nobody was watching.
- Same set, same assessors — otherwise the comparison is meaningless.
- Multiple subjects compared — a configuration change can be assessed against the current production one directly.
- Trace-level drill-down — a failed case leads to the trace, so you can see what changed rather than inferring it from a score.
- Rules held separately — the same rules apply across subjects, so a difference is attributable to the subject.
Linkage to live observation
Evaluation is not sealed off from production. Aggregate accuracy falling below a threshold raises an alert, and an agent performing below expectations triggers a real-time warning rather than being discovered in the next quarterly review.
- Threshold-based alerting — you set the line; the system watches it.
- Per-business pipelines — different teams weight the evaluation criteria differently, and a single global standard is rarely right for all of them.
- Template reuse — recurring evaluation tasks are templated, so a new agent inherits a sensible baseline.
- Pipeline controls — scheduled and repeatable execution, with concurrency, throttling, scheduling strategy and timeout retry.
What the report has to contain
A report that only shows a score is not evidence; it is a claim. The deliverable for an enterprise has to survive a review, so it carries four things:
- Data statistics — counts, distributions, per-category performance, and the size of the set behind each figure.
- Comparative analysis — subjects side by side, so the change is visible rather than asserted.
- Trace drill-down — from a failed case to the individual execution that produced it.
- Data export — so the result can be reconciled, archived and independently reviewed.
The deliverable, in plain terms
Somebody in a risk or audit function should be able to open a report, see what was tested, see the results with the underlying cases attached, and reach a defensible conclusion without asking the engineering team to explain what the numbers mean. That is the bar. If a report only works when the person who built it is in the room, it is not yet a deliverable.
Regression, alerting and the evidence trail
The archive at the end is what turns evaluation from a quality tool into a governance artefact. Six months later, it is the answer to "who approved this and why".
A prompt edit, a context change, a model swap, a new tool version. All four are changes, and all four need the same gate.
The full test set runs against the candidate. Results compared against the live baseline, per case rather than in aggregate.
New passes are noted. New failures are the ones that matter. A change that improves 40 cases and breaks 1 badly is usually a rejection.
Somebody with authority accepts the regression risk, by name, in a record.
A small share of live traffic first, with automatic rollback if the live rate breaches the threshold.
Post-release observation, then the version and its results archived as evidence.
There is no industry pass mark. The line is yours to set.
This is the part of evaluation where we have to decline the easy answer.
No vendor can give you a universal threshold. A useful accuracy figure for a media summarisation agent and one for a benefits-entitlement agent are not comparable, and a number that looks impressive in a vendor comparison is a number that was chosen for a different business. The threshold has to come from your risk appetite, and it has to be calibrated against your own historical data and your own human judgement.
How we do it instead. We establish the line empirically: run the first evaluation against the current production behaviour, inspect the failures with your domain experts, and set the threshold where the failure modes you are willing to accept end and the ones you are not begin. That baseline then travels with every future release. It is slower to establish and it is the only version that means anything.
Other things we will not overstate
- No pre-set industry default. Any default we offered would be a number we invented, and it would quietly become your standard.
- LLM-as-a-Judge needs a control. We recommend running a human-annotated sample against the judge to measure its agreement, because a judge that is systematically lenient will raise every score you report.
- Full-volume inspection is not full understanding. Rules catch what they were written to catch. The categories reduce an exception list to a work queue; they do not diagnose the root cause on their own.
- Cost belongs in the evaluation, but it is not a quality metric. It is a dimension to be made explicit, and a trade-off to be decided deliberately rather than absorbed silently.
- A high score is not a claim about the world. It is a claim about the questions you asked. Which is why the test set is the part we argue about most.
What ninety days of evaluation actually changes
Illustrative, from our own delivery experience. Grounding and refusal discipline improve most — because those are the two dimensions nobody measures without trying.
Start with the failures you already know about
There is no industry pass mark. An agent that answers 96% of support questions correctly and escalates the rest is not the same as one that answers 96% correctly and confidently invents the other 4%. The threshold is yours to set, and we will make you justify it against the cost of being wrong in your specific workflow.
Five questions that settle whether you need this
Five yes-or-no answers. If you cannot answer the first three, the capability is not your bottleneck yet.
- Can you name the specific workflow this serves — not the department?
- Can you name the person accountable when it is wrong?
- Can you show me historical cases where a human did this task, so we can build an evaluation set?
- Can you tell me what this capability explicitly will not do?
- Can you tell me what it costs per task, and how you would reduce that cost?
What this page answers
Every topic below is covered in the sections above, and each links to the page that goes deeper on it.
Start with the failures you already know about.
A working session where we take a sample of your real conversations, agree the first evaluation criteria, and run one baseline evaluation against current production behaviour. You leave with a first test set and a threshold you can defend.