Which model answers is an engineering decision, not a procurement one.
The platform manages multiple model families, cloud-hosted and self-owned compute side by side, and routes each request to the tier that actually suits it. It also gives you the parts that decide whether the arrangement survives a year of production: routing, degradation, recovery, and pinned versions.
Many models, one surface
Open and hosted models managed together, so switching is a configuration change rather than a re-integration
Work goes to the right tier
Simple questions do not need your most expensive model, and the platform knows the difference
Inference inside your network
Private inference with data that does not leave the domain, alongside hybrid deployment
Versions held still
Production pins its model version, and upgrades are rehearsed in a test environment first
One management surface over many model families
The platform accepts open-source models, third-party hosted models, your own private inference endpoints and models you host on your own compute. They are addressed through the same interface, so an architecture decision about one of them does not ripple through the rest of the estate.
What you can configure per model node
- Model selection — which model serves this node, chosen from what the deployment has registered.
- Temperature — the sampling setting, set deliberately per task rather than left at a default across every prompt in the company.
- Maximum output tokens — a hard ceiling on generation length, which is a cost control as much as a behaviour control.
- System prompt — the instruction frame for the node, held separately from the runtime so it can be reviewed and versioned like any other configuration.
Comparing models before you commit
The evaluation system supports selecting multiple evaluation subjects in a single run — an agent, an LLM, or a prompt template — so the comparison happens inside the same harness, against the same test set, with the same assessors. That matters more than it sounds: a model benchmark run in isolation answers a different question from the one your users will actually ask.
Where the models come from
Hosted model families
The platform's own model family, available in tiered variants, including deeper reasoning and multimodal capability.
Third-party hosted models
Connected by API key, including providers that speak a widely used compatible interface.
Open-weight models
Run on your own infrastructure, on shared or dedicated compute.
Your own fine-tuned models
Produced on the platform's fine-tuning path and served like any other model.
How to choose. Start with the workloads you can measure — high-volume, low-complexity traffic first. The expensive question is not which model is best, but which questions you were paying an expensive model to answer.
Models and compute in numbers
Cheap, standard, and reserved for the hard tail
Over many model families — not a commitment to one
Private, hybrid, and your own silicon
The stated goal of the operational discipline
Show us your traffic mix, and we will show you the routing
Most estates are paying premium prices for work a small model does perfectly. The routing is where the cost conversation actually happens.
One model for everything is the wrong answer in an enterprise
Quality, latency and cost pull against each other, and a single-model strategy forces you to pay for the best answer on every request — including the ones where the best answer is a lookup. Routing and dual-mode thinking are how you stop doing that.
| Workload | Tier it should reach | Why |
|---|---|---|
| High-frequency simple questions | Lightweight model | Most traffic is this. Serving it on a heavyweight model is the single largest source of avoidable spend, and it also inflates latency for the users who just wanted a fact. |
| Tool-calling and structured action | Standard tier | Reliable function selection and parameter formatting matter more than deep reasoning, so a mid tier is usually the better trade. |
| Complex multi-step reasoning | Heavyweight tier | The cases where quality genuinely compounds — long-horizon analysis, ambiguous policy interpretation, difficult synthesis — and where the marginal cost is worth paying. |
| Recognised, simple intent | Fast thinking | A known intent does not benefit from extended deliberation. Skipping it removes the wait that makes agents feel slow. |
| Complex or ambiguous intent | Slow thinking | The cases where deliberation changes the answer, and where the extra latency is understood and acceptable. |
Cost
A routed estate can spend far less than a single-model estate of equivalent quality, because the expensive tier is used only where it changes the outcome.
Latency
Fast-path handling of common requests is what makes an assistant feel immediate. Extended deliberation on every turn is the usual cause of a sluggish agent.
Quality
Quality is not a single dial. Setting one global tier is a compromise on every workload at once, and the compromise is usually wrong in the expensive direction.
The honest caveat. Routing rules are not a fully self-service configuration surface in every product version, and the mapping of request type to model tier is something we design with you during the architecture phase rather than something you will tune alone on day one. We would rather say that than ship you a diagram that does not exist.
Where the money actually goes
None of these are model price problems. They are routing problems, and fixing them typically halves the bill without touching a single capability.
| Cost driver | How it behaves | What usually fixes it |
|---|---|---|
| Unnecessary model tier | 60–80% of calls are on an expensive model doing classification work | Route by task type, not by agent |
| Unbounded agent loops | A reasoning loop with no step budget runs until it decides it is finished | Set a hard step budget and an escalation trigger |
| Context carried too far | Full document history resent on every turn in a multi-turn conversation | Summarise and drop, rather than resending |
| Redundant parallel attempts | Fan-out with adjudication for tasks that never needed adjudication | Only adjudicate where the decision is material |
| No cache on stable content | Governed reference text re-retrieved and re-sent thousands of times a day | Cache by content hash with an explicit invalidation rule |
| Silent fallback to premium | A rate limit quietly downgrades to a more expensive model and nobody sees it | Alert on tier changes, not just on failures |
Private inference, hybrid deployment, and your own silicon
Model choice and data residency are the same conversation once the models are on your infrastructure. The platform supports the full range: a private inference service inside your network, a hybrid topology that uses both, and hosting your own compute for inference and training.
Private inference, in-network
A private inference service embedded within your internal network, so prompts, retrieved context and model outputs never leave the domain. For defence, banking and public sector, this is the configuration that makes the platform deployable at all.
Hybrid deployment
Cloud-hosted capacity and your own compute running side by side. The split is normally made by data classification rather than by preference: regulated content stays inside, everything else uses managed capacity.
Your compute, hosted for you
Your own hardware can be managed to run both inference and training, so a capital investment in accelerators becomes usable capacity for the agent estate rather than an underused cluster.
One service for inference, data and fine-tuning
Training and serving are treated as one path rather than three projects that hand off to each other. Data preparation, fine-tuning and the serving of the resulting model sit in the same platform, which removes the most common cause of failure in enterprise model programmes: the gap between what the team trained and what production actually runs.
For organisations with specific hardware or domestic-compatibility requirements, the platform is also adapted to mainstream domestic CPU and GPU architectures, operating systems and databases.
Fine-tuning on your own data
Fine-tuning is built on two inputs that only an enterprise has: production conversation data flowing back from the live estate, and your core knowledge base. Both are cleaned and annotated into a high-quality dataset, and used to build a model specific to your business.
A reinforcement-learning path is also integrated, working from production dialogue data to refine an agent's decision logic — reducing the annotation effort that usually dominates the cost of this work.
Private, hybrid, or your own silicon
The honest version: private inference is a data decision first and a cost decision second. If your data can leave, hybrid will usually serve you better.
- Your data does not leave your boundary
- Predictable cost at volume
- You own the hardware refresh cycle
- Capability tracks whatever hardware you bought
- Strong choice for regulated data at steady volume
- Sensitive classification and extraction stay private
- Hard reasoning goes to the best available model
- Cost follows the routing, not the hardware
- You can move either side without a re-architecture
- This is what most regulated clients end up with
The estate has to behave the same on a Tuesday afternoon as on quarter close
Agents fail in a specific way: not by crashing, but by taking too long, queueing behind each other, or answering slightly differently because the model behind them moved. These are the mechanisms that address the first two, and that hold the third steady.
Compute elasticity
- Automatic tier switching — the scheduler moves work between lightweight and heavyweight models according to task weight, rather than committing one tier for the whole estate.
- Idle capacity reclaimed — compute sitting unused between peaks is returned rather than held, which is what makes elastic economics work on your own hardware as well as on hosted capacity.
- Framework images pre-loaded — inference runtimes are preloaded into their images so the model starts fast rather than paying a cold-start cost on the first request.
- Prefill and decode separation — separating the two stages of inference, with high-speed interconnect, reduces time to first token, which is the latency users actually perceive.
- Elastic KV cache — cache capacity scales with the load, again targeting time to first token rather than throughput in the abstract.
Failure handling
- Circuit breaking and degradation — a single request that times out trips the breaker and the call is served by a degraded model, so one slow response does not become a user-visible failure.
- Scale on a concurrency threshold — capacity expands against measured concurrency rather than a fixed schedule, and contracts again when the load falls.
- Per-instance rate limiting and queueing — requests beyond an instance's capacity are queued rather than rejected, which protects the instances that are working.
- Self-healing and instance failover — an agent instance that fails is replaced by a standby automatically, and long-running tasks resume from a checkpoint instead of restarting.
- Confidence-based fallback — a request whose confidence falls below threshold is routed to a human queue and the responsible owner is notified, rather than answered badly.
One caveat worth stating. Idempotency is not automatic. If a retried request writes to a system of record, it needs a unique request identifier to prevent a duplicate write — that is your engineering responsibility, and we will design it in with you rather than let you assume the platform handles it.
The estate must behave the same on a Tuesday afternoon as on quarter close
Seven properties. If any of them is missing, quarter close will find out before your customers do.
- Capacity is reserved for the predictable peak, not provisioned on average.
- A rate limit degrades to a smaller model, never to an outage and never silently upward.
- Every tier change is logged and alerted on, so finance can reconcile the bill.
- Cost per task is a monitored metric with a budget, not a monthly surprise.
- Latency percentiles are tracked per task type, not just as an average.
- A model deprecation is a planned migration with a named owner and a date — never a surprise.
- The evaluation harness runs against the candidate replacement before traffic moves.
The operational discipline that stops an upgrade becoming an incident
This is the least glamorous section on the page and the one that decides whether the estate is still trusted a year later. Model behaviour changes without notice, and every change is a candidate regression.
Four rules we enforce in design review
- Pin the model version in production. A floating reference means behaviour can change under you between a Tuesday and a Wednesday, with no deployment to point at. The version in the configuration is part of the release record.
- Rehearse every upgrade in the test environment. The upgrade is a change, and it gets validated against your own test set before it reaches a user.
- Expect a quarterly minor and an annual major release. Within a contracted term, minor versions arrive quarterly and major versions annually. That rhythm is your maintenance calendar, not a surprise — and it is the reason the regression set matters.
- Schedule canary releases across a business peak. Releasing a small share of traffic first and observing it is the standard practice, and a canary window that does not cover your busiest hours has not really tested anything.
We also work with multiple environments — development, test and production — where data and variables are maintained independently while the workflow and plugin configuration is synchronised across them. That separation is what makes rehearsal meaningful rather than cosmetic.
The upgrade checklist we run with clients
Pin first
Confirm every production model reference resolves to an explicit version before anything is upgraded.
Run the regression set
The same test set, in the test environment, against the new version and the old one, side by side.
Canary a small traffic share
Observe real conversations and the bad-case queue before the change reaches everyone.
Promote or hold
A decision with a named owner, and a written reason either way.
What we will not put in a proposal
Model and compute commitments are where procurement language tends to drift furthest from engineering reality. Here is where we hold the line.
There is no clear productised interface for model allow and deny lists. If your estate requires a hard policy of which models may be used, that policy has to be enforced at the network and credential layer — a key distribution control, a gateway rule, or a configuration review. We will help you design it, but we will not claim a switch exists if it does not.
Hardware specifications and service commitments vary by version and by contract. Minimum cluster shapes, availability commitments and support terms are not the same across hosted, dedicated and on-premise topologies. We confirm them against your specific version and the signed agreement. We will not put a number in a document that is not in the contract.
And to be explicit about this page
- No price appears here. Model compute, fine-tuning and private deployment are all quoted per project, because the honest number depends on your volume, your hardware position and your support requirement.
- No model version numbers appear here either. Version pinning is a discipline we describe, not a specific release we are promoting.
- Availability and throughput figures we do quote are marketing claims. We will not put them in a contract, and we will tell you which ones we are excluding and why.
- Fine-tuning is not a substitute for retrieval. If your problem is stale or missing knowledge, tuning the model will not fix it, and we will say so rather than sell you a training project.
What we will not put in a proposal
We will not put a token commitment in a proposal with no baseline. We will not promise a price reduction that depends on volume you have not demonstrated. And we will not quote a per-seat cost for a workload whose consumption is per task — those are different units and confusing them is how budgets break.
Five questions that settle whether you need this
Five yes-or-no answers. If you cannot answer the first three, the capability is not your bottleneck yet.
- Can you name the specific workflow this serves — not the department?
- Can you name the person accountable when it is wrong?
- Can you show me historical cases where a human did this task, so we can build an evaluation set?
- Can you tell me what this capability explicitly will not do?
- Can you tell me what it costs per task, and how you would reduce that cost?
What the first ninety days look like
The same shape on every capability. The content differs; the checkpoints do not.
| By day 30 | By day 60 | By day 90 |
|---|---|---|
| Baseline captured on your metric | Workflow running with a named approver | Metric moved, against the baseline you did not choose |
| Scope boundary documented — including what it may not do | Evaluation harness re-running on every change | Decision log reviewed by your risk function |
| Historical cases assembled for testing | Training delivered, completion in the LMS | Next workflow scoped from what we learned |
| Data lineage for every source the agent reads | Cost per task measured and modelled | Evidence pack complete and accepted |
What this page answers
Every topic below is covered in the sections above, and each links to the page that goes deeper on it.
Show us your traffic mix, and we will show you the routing.
A working session on the questions your agents actually answer, split by complexity, with the model tier and the cost implication for each. You will leave with a routing proposal and a deployment topology recommendation.