You cannot manage what you cannot see, and you cannot budget what you cannot attribute.
A model call is a black box, and a black box cannot be debugged, defended or costed. This page covers the instrumentation that makes agents legible: full-chain tracing, operational dashboards, Bad case attribution, cost accounting down to the workflow, and the data loop that turns production traffic into your next improvement.
Metrics, logs, traces
Classical observability, extended for a process that was never transparent to begin with
Bad case, categorised
Error samples classified by cause, so a failure becomes an owned task rather than a complaint
Attributed to the workflow
Token consumption, call counts and quotas — allocated to the work, not to a single total
Usage into training
Production traffic becomes evaluation and training data, which is how the agent gets better
Observability, extended for a process that was never transparent
Your existing observability stack was built for services that are deterministic and whose failure is unambiguous. An agent violates both assumptions: the same input can produce two legitimate answers, and "wrong" is often a matter of policy rather than of an exception. The three pillars are the same; the instrumentation is different.
Metrics
The numbers that describe how the estate is behaving: completion, latency, call volume, error rates, and the quality measures the evaluation system produces. Read at two levels — the agent's own performance, and the model layer beneath it, so you can tell a bad prompt from a bad model.
Logging
The record of what was said, requested and returned. In an agent estate the log is not just for debugging — it is the raw material for quality inspection, for cost attribution, and for the training data you will need later.
Tracing
The sequence of steps inside a single run: what was retrieved, which tools were called, in what order, with what arguments, and what each step produced. This is the pillar that does not exist classically, because a function call is not the unit of work here.
Why this is a new requirement rather than an extension. In a conventional service, the call graph shows you how the request moved through known code paths. In an agent, the path is chosen at runtime by a model, so the interesting question is not "did a component fail" but "why did it decide to do that". Tracing is how the second question becomes answerable at all.
Observability in numbers
Not a sample — an estate you cannot fully see is an estate you cannot defend
Who, what, why, on what evidence, with what authority
Cost per task, not a monthly token bill
The model call itself is observable, not opaque
Turning the model call from a black box into a white box
Six stages, all recorded. This is what lets you answer "why did the agent do that" six months later, which is the only question that really matters.
Request
The full prompt as sent, with the model, version, parameters and cost estimate.
Context assembly
Which sources were retrieved, in what order, with what scores. This is where grounding questions are answered.
Response
The full output as received, plus token counts and latency.
Tool calls
Each call with arguments and result, in order.
Assessment
The evaluation verdict, the assessor used, and the failure cluster if it failed.
Outcome
Accepted, overridden, escalated — and by whom.
Turning the model call from a black box into a white box
Link tracing reconstructs what actually happened during a model call, and — the part that matters more — reconstructs the real user scenario behind it. Effect becomes traceable, and problems become attributable to a specific cause rather than to a general feeling that the agent has been off lately.
Reasoning made visible
The chain of thought within a run can be visualised step by step, so a reviewer can see where a plan went wrong — a misread intent at the start, a poor retrieval in the middle, a tool chosen for the wrong reason at the end. Without that sequence, all you have is the answer and a hunch.
This is also the only way to tell a genuinely hard problem from an avoidable one. A large fraction of what looks like a model limitation turns out, once traced, to be a retrieval failure or an instruction conflict.
Attribution, not symptom-watching
Because the trace is a structured sequence rather than a wall of text, it can be attributed: to the knowledge base, to the prompt, to the tool, to the model, or to the request itself. Each of those has a different owner and a different remedy, and confusing them is the most expensive mistake in agent operations.
Session management sits alongside this, supporting both troubleshooting a specific conversation and the broader analysis of how the estate is used in practice.
What tracing is for, in order of frequency. First, answering "why did it say that" for a specific user. Second, finding the class of failures behind a rising complaint rate. Third, giving a knowledge engineer the exact passages that were retrieved for a failing question. Fourth, producing the evidence an audit asks for. A capability that only serves the fourth purpose is a compliance feature; a capability that serves the first three is an operational one.
What the estate looks like, and why it went wrong
Two different questions need two different surfaces: how the estate is performing right now, and why a particular conversation failed. Boards answer the first; the Bad case pool answers the second.
The observation board
Each agent has a view of its own working performance, with performance, effect and user feedback monitored together rather than in separate tools. The board is configurable: metric cards you define, trend lines over the period that matters to you, and alert thresholds set against the measures you actually care about.
- Custom metric cards — the measures your business uses, not only the ones the platform ships with.
- Trend lines — over a period chosen by the reader, so an operations lead and a finance lead can look at the same data and each get the view they need.
- Alert thresholds — configured once, then enforced consistently, including notification into your enterprise collaboration platform.
- Runtime reporting switch — telemetry reporting is configurable per agent in its deployment settings, so a deployment with data-residency constraints can be handled deliberately rather than by exception.
- Session management — conversation-level inspection for troubleshooting and for the operational analysis of how the estate is actually used.
Bad case attribution
A list of error samples, a count per category, and a classification by cause. The three recurring causes each point at a different owner, which is the reason the classification is worth the effort of maintaining:
Unreasonable knowledge base segmentation
Owned by knowledge engineering. The document was right and the slice of it that was retrieved was not.
Prompt logic problems
Owned by the agent designer. The instructions conflict, omit a step, or contradict the policy they carry.
Insufficient intent recognition samples
Owned by the training data owner. The utterance never matched, so the agent inferred its intent from nothing.
Prompt efficiency tracking. Because the platform can see which instructions produced which outcomes, it can also identify where a more efficient instruction would have produced the same result. That is targeted feedback for a fine-tuning effort — not a general impression that the prompts could be tidier.
Five dimensions in every decision log entry
A transcript tells you what was said. A decision log tells you what was decided, on what basis, and by whose authority. Only the second one survives an audit.
| Dimension | What is recorded | Who asks for it |
|---|---|---|
| Who | The agent identity and the human who caused the action | Audit, and any incident review |
| What | The action taken, the arguments, and the result | Operations, and the system owner |
| Why | The reasoning summary and the sources it relied on | Supervisors, and anyone challenging a decision |
| On what evidence | The exact context retrieved — which documents, which records, which version | Risk, and anyone checking whether the answer was grounded |
| With what authority | The approval point, the confidence level, and whether it escalated | Risk and compliance — this is the row that makes it defensible |
| What it cost | Tokens, tool calls, latency, and the per-task total | Finance, every single month |
Cost you can take to a finance conversation
Agent spend behaves unlike infrastructure spend. It scales with how the business uses the agent, it changes when a prompt changes, and it is invisible until someone asks for a number and nobody has one. Without attribution to a workflow, cost is a complaint rather than a management input.
What is measured
Token consumption, model invocation counts and tool invocation counts. The tool count matters as much as the token count: a workflow that calls eleven tools to answer a question is an integration design problem wearing a cost problem's clothing.
What the board shows
Task completion rate, average task duration, token consumption per task, output consistency attainment, and tool call correctness. The last two are the most useful, because they connect spend to quality: a workflow that is cheap and wrong is not efficient.
What is enforced
Quota and allowance management: free allowance separated from packaged allowance, configurable warning thresholds as consumption approaches the limit, and per-sub-account call ceilings so one team's runaway traffic cannot consume the whole estate's allowance.
Cost is a dimension of evaluation, not a separate report
The evaluation system treats cost explicitly as one of the dimensions it measures alongside business effect, safety and performance. That is a structural decision with a practical consequence: when you compare two candidate configurations on the same test set, you see what each one costs as well as what each one achieves. Cost becomes an input to a decision rather than a surprise discovered on the invoice.
It also prevents a familiar failure mode in which teams optimise for quality alone, produce an excellent agent, and then cannot afford to run it. In an enterprise the acceptable point on that trade-off is a business decision, and the platform should at least make the trade visible.
Attribution is the whole point
A single monthly total tells you nothing you can act on. Cost attributed to a specific workflow, a specific agent, or a specific business function tells you where to look — and in a well-run estate, it is what a finance business partner will accept as a figure rather than dispute as an estimate.
It also makes prioritisation honest. When a workflow costs more than it saves, you can see that immediately, and the conversation about whether to keep it becomes an operating-model discussion rather than a technical one.
Cost you can take to a finance conversation
A token bill cannot be defended because nobody can explain what it bought. This split can — and the bottom band is usually where the recoverable money is.
The loop that makes the agent better the more it is used
The most valuable asset an agent estate accumulates is not its prompts. It is the record of what real people asked and how the agent handled it — and that record is the raw material for every subsequent improvement, if the estate is instrumented to use it.
Use
Production conversations accumulate as the estate is used. Nothing extra is required of the teams building agents — the data is a by-product of normal operation, provided the reporting switch is on for that agent.
Feedback
The platform runs quality inspection over the traffic, classifies the failures, and captures explicit user feedback — whether something helped, did not resolve the issue, or cited something incorrectly — directly in the front end, where the user is already present.
Optimise
Production data is cleaned and annotated, and becomes both an expanded evaluation set and a training set. A dedicated data engineering layer handles integration, cleaning and synthesis, so the raw traffic is not merely accumulated but made usable.
Use again
A fine-tuning dataset can be generated from the cleaned data and fed back into the estate. The new configuration is evaluated against the expanded set, and the loop closes. The data module can also generate fresh examination questions from the latest enterprise knowledge, so the evaluation set keeps pace with what the business now knows.
The loop, stated plainly. Use it, collect the feedback, improve the configuration, and use it again — with the improvement measured rather than assumed. What makes this a flywheel rather than a slogan is that each turn produces an artefact: a test set that grew, a labelled training set, a knowledge base with fewer segmentation defects, a prompt that has been shown to be equivalent or better on a comparable set. Without those artefacts you have an anecdote about improvement. With them, you have a trajectory.
There is a second, quieter benefit. As the estate accumulates production data, it becomes a record of how the business actually works — the questions people ask, the decisions they need help with, the policies they look up. That record is a management asset, and it is also the clearest possible statement of what the next generation of agents should be built to do.
The loop that makes the agent better the more it is used
This is the difference between an agent that degrades under load and one that improves. It only works if the whole estate is traced.
Every conversation traced. No sampling.
Failures grouped by cause, not counted. A hundred retrieval misses are one problem, not a hundred.
For each cluster, the stage it failed at: retrieval, ranking, reasoning, tool, or specification.
A targeted change to context, prompt, tool, or scope.
Regression suite re-run. The cluster must reduce, not merely move.
The cluster becomes permanent test cases. The agent cannot regress on a failure we have already paid for.
What the instrumentation will not do for you
Observability tools have a habit of implying more certainty than they deliver. These are the limits we state up front.
Boards refresh on a delay. Dashboard data is refreshed on an interval rather than continuously, which means boards are the right instrument for trends and for operational review, and the wrong instrument for second-by-second alerting. If you need second-level detection, that belongs in your own infrastructure monitoring in front of the platform, not in the observation board.
Turning reporting off costs you observation. The runtime reporting switch is a genuine data-residency and privacy control, and we encourage you to use it deliberately. But with reporting disabled for an agent, the parts of the observation stack that depend on that traffic — quality inspection, Bad case collection, cost attribution, and the data flywheel for that agent — do not function. You are trading observability for residency, deliberately, and it should be a decision rather than a default.
And to be plain about the rest
- Tracing shows the run, not the truth. A reasoning trace records what the agent did. Whether its answer was correct is a separate question, answered by evaluation and by human review.
- Attribution is a first cut. Classifying a failure by cause helps enormously, and it is not a substitute for diagnosis. A large share of real failures sit across categories rather than neatly inside one.
- Cost attribution depends on the workflow being labelled. If a workflow has no owner and no name, the money still appears on the invoice — just not anywhere useful.
- No availability or refresh claims on this page. The specific figures depend on version and configuration, and we confirm them per project rather than publishing numbers we would have to qualify.
What the instrumentation will not do for you
Instrumentation will not tell you whether the workflow was worth automating, or whether the business process behind it is sound. It also will not tell you that an agent's answer is right — it tells you what happened, not whether it was correct. That is what evaluation is for, and the two are not interchangeable.
Five questions that settle whether you need this
Five yes-or-no answers. If you cannot answer the first three, the capability is not your bottleneck yet.
- Can you name the specific workflow this serves — not the department?
- Can you name the person accountable when it is wrong?
- Can you show me historical cases where a human did this task, so we can build an evaluation set?
- Can you tell me what this capability explicitly will not do?
- Can you tell me what it costs per task, and how you would reduce that cost?
What the first ninety days look like
The same shape on every capability. The content differs; the checkpoints do not.
| By day 30 | By day 60 | By day 90 |
|---|---|---|
| Baseline captured on your metric | Workflow running with a named approver | Metric moved, against the baseline you did not choose |
| Scope boundary documented — including what it may not do | Evaluation harness re-running on every change | Decision log reviewed by your risk function |
| Historical cases assembled for testing | Training delivered, completion in the LMS | Next workflow scoped from what we learned |
| Data lineage for every source the agent reads | Cost per task measured and modelled | Evidence pack complete and accepted |
What this page answers
Every topic below is covered in the sections above, and each links to the page that goes deeper on it.
Instrument one agent, and find out what it is really costing you.
A working session on a live agent: trace a handful of real conversations, categorise the failures, and attribute the spend to the workflow. You will leave with a cost figure and a quality baseline you did not have before.