The gap between what the models can do and what most production systems are actually built to handle is still wide, and it is not closing at the rate the model releases suggest.
The reason is a difference in how fast two things move. Model capability arrives in steps, from outside, on somebody else's release schedule. Instrumentation, evaluation layers and cost controls are built internally, by a team, against a specific organization's processes and risk tolerance. Those move slowly, and that is where most of the real deployment work lives.
This is why an organization can adopt a frontier model in a week and spend four quarters getting one workflow into production.
What the instrumentation layer actually contains
Three components. None of them is a model and all of them are prerequisites for using one in a process that matters.
| Component | What it does | Why it is slow to build |
|---|---|---|
| Evaluation | Determines whether an output is acceptable, on this organization's work | Requires a definition of correct for each task, which the organization has to supply |
| Cost control | Keeps spend predictable per unit of work | Requires a unit of work to exist, and a baseline cost for it |
| Observability | Shows what the system did, on what input, with what result | Requires a record of expected behaviour to compare against |
The shared property is visible in the right-hand column. All three depend on the organization describing its own work precisely enough to say what correct looks like, what a unit of it is, and what normally happens. That description is the slow part, and it is not a technical task.
Why does evaluation take so long?
Four reasons, and the first accounts for most of the delay.
Correct is organization-specific
A general benchmark says a model performs well at summarization. It does not say whether this summary, of this document type, for this reviewer, meets the standard this team applies. The only source for that standard is the team, and in most cases it has never been written down.
The test set has to come from real cases
An evaluation built on constructed examples measures performance on constructed examples. A usable set comes from actual historical cases including the awkward ones, which means somebody has to find them, label them and agree on the right answer.
The exceptions carry the risk
The normal path is usually straightforward to evaluate. The cases that determine whether a system is safe to deploy are the exceptions, and exception handling is typically undocumented because it was learned rather than designed.
Acceptable is a negotiated threshold
Ninety percent accuracy is adequate for some tasks and unacceptable for others. Setting the threshold requires knowing what happens downstream of an error and who absorbs it, which is a process question.
Published frameworks place this work before deployment rather than alongside it. The NIST AI Risk Management Framework defines Measure as a distinct function requiring metrics and test sets appropriate to the context, and it places Map, establishing that context, ahead of it. ISO/IEC 42001 specifies requirements for an AI management system, including the need to define objectives and assess the system against them, which is the same sequencing in management-system terms.
Why cost control is harder than it looks
Cost per request is simple to read from a dashboard. Cost per unit of business work is the number a budget needs, and it requires three things most organizations lack.
A defined unit. Per claim, per invoice, per ticket, per candidate. Without it, spend is reported per token and nobody can say whether it is reasonable.
A baseline for comparison. What the unit costs today, performed the current way. An AI cost of two dollars per case is either excellent or indefensible depending on a number most organizations have never measured.
A retry and escalation model. Real workflows retry, fall back to a larger model and escalate to a person. Those paths dominate the cost distribution and they are usually absent from the original estimate.
| Metric | Available on day one | Useful for a budget |
|---|---|---|
| Cost per request | Yes | No |
| Tokens per month | Yes | No |
| Cost per unit of work | Requires a defined unit | Yes |
| Cost per unit against the current method | Requires a baseline | Yes, this is the decision number |
| Fully loaded cost including retries and escalation | Requires an operating model | Yes |
The bottom three rows all depend on process description rather than on tooling, which is why cost control stalls in organizations that have the dashboards.
The slower layer
What each component needs from the organization before it can be built
| Component | Technical requirement | Organizational requirement |
|---|---|---|
| Evaluation | A test harness | A definition of correct, per task |
| Cost control | Usage metering | A unit of work and its current cost |
| Observability | Logging and tracing | A description of expected behaviour |
| Human escalation | A routing mechanism | The exception paths people already use |
The left column is a few weeks of engineering. The right column is the part that takes quarters, and none of it can be bought.
The sequencing problem
Most AI programmes build in the order the tooling suggests: model, prototype, then instrumentation when something is about to go live. That order puts the organizational work last, at the point where a date has already been committed.
The alternative order is uncomfortable because it looks slower at the start.
| Order | Step one | Where it stalls |
|---|---|---|
| Tooling-led | Pick a model, build a prototype | At production, when correct has to be defined under deadline |
| Process-led | Describe the work and define correct | Rarely, and the prototype is faster because the target is specified |
The practical consequence is visible in a specific pattern. A prototype demonstrates well and then spends two quarters in pre-production, with the work in those quarters consisting almost entirely of establishing what the system is supposed to do on edge cases, how much it is allowed to cost, and who handles the output when it fails. That work could have been done before the prototype existed.
What to establish before the first deployment
Six items. All of them are process artifacts rather than technical ones.
1. The unit of work
The thing being processed, named, with a volume per period. Everything in cost control and most of evaluation hangs off this.
2. The current cost and duration per unit
Measured, from the people doing the work. This is the baseline that turns a cost figure into a decision, and its absence is the most common reason an AI programme cannot demonstrate impact afterward.
3. A definition of correct, per task
Written in terms the team uses, with examples. Three to five real cases marked acceptable and three to five marked unacceptable is enough to start.
4. The exception inventory
What goes wrong, how often, and what people currently do about it. This is the content that determines deployment risk and it is the hardest to obtain, because it exists in practice rather than in documentation.
5. The escalation path
Who receives an output the system cannot handle, and what they do with it. A workflow with no named escalation target either blocks or proceeds incorrectly.
6. The threshold and who owns it
The accuracy or cost level at which the system is acceptable, and the person accountable for that judgment. An unowned threshold moves under schedule pressure.
Where Horizon fits
Horizon is an AI-powered continuous discovery platform. Its relevance to this problem is the right-hand column of the table above: the organizational inputs the instrumentation layer depends on, which cannot be bought as tooling and are slow to assemble by hand.
Discovery Cycles run AI-led interviews with the people performing the work, which produces the unit of work, the current duration per unit, the exception inventory and the escalation paths people already use. Those four are the inputs evaluation and cost control are waiting on, and they come from practice rather than from design documents.
The Process Library holds the described process including steps that leave no system record, which is where most exceptions live. The Insights Dashboard attaches hours and impact to findings with traceability to the inputs behind them, which supplies the baseline that makes a cost per unit interpretable. The Initiatives Dashboard carries each candidate into a business case with a named owner, which is where the threshold decision belongs.
At Trafilea, a cycle ran 43 interviews in two weeks and produced more than 10 findings, including 218 hours a month going into duplicated tracking work. The engagement reported 129 hours of discovery effort avoided, a 4.1x figure and a 310% return from day one, against a baseline of roughly a year for comparable work. A figure like 218 hours a month on a named activity is exactly the shape a cost baseline needs, since it gives a unit, a volume and a current cost in the same statement. These numbers describe one engagement and they are evidence rather than a guarantee.
The question worth asking before any deployment: do we have a measured current cost per unit of this work, and a written definition of what correct output looks like on our own cases?
Common mistakes
Treating instrumentation as an engineering task. The technical side is weeks. The organizational side, defining correct and establishing a baseline, is the part that takes quarters and it cannot be delegated to the platform team.
Evaluating against general benchmarks. A benchmark score describes performance on the benchmark. It says nothing about whether output meets this team's standard on this document type.
Building the test set from constructed examples. Performance on constructed cases predicts performance on constructed cases. Real historical cases, including awkward ones, are the only usable source.
Reporting cost per request to a budget holder. The number a budget needs is cost per unit of work against the current method, and both halves of that require process description.
Omitting retries and escalation from the cost model. Those paths dominate the distribution in production and are almost always absent from the initial estimate.
Deploying without a named escalation target. A workflow that cannot hand off either blocks or proceeds incorrectly, and both failures land on someone who was not expecting them.
Leaving the threshold unowned. An acceptance level without an accountable owner moves when the schedule tightens.
FAQ
What is AI production instrumentation?
The layer around a model that makes it usable in a real process: evaluation, which determines whether output is acceptable on this organization's work; cost control, which keeps spend predictable per unit of work; and observability, which records what the system did on which input with what result.
Why does instrumentation move slower than model capability?
Because capability arrives from outside on a vendor's release schedule, while instrumentation is built internally against a specific organization's processes and risk tolerance. All three components depend on the organization defining what correct means, what a unit of work is and what normally happens, and that definition is usually not written down.
Why can't you use general benchmarks for evaluation?
Because they measure performance on the benchmark rather than on your work. Whether a given output meets the standard a particular team applies to a particular document type is an organization-specific judgment, and the test set has to come from real historical cases including the exceptions.
What cost metric should an AI programme report?
Cost per unit of business work, compared against what that unit costs today performed the current way, with retries and escalation included. Cost per request and monthly token spend are available immediately and neither tells a budget holder whether the system is worth running.
What has to exist before a first deployment?
A named unit of work with its volume, a measured current cost and duration per unit, a written definition of correct with real examples, an inventory of exceptions and their frequency, a named escalation target, and an owned acceptance threshold. All six are process artifacts rather than technical ones.
The slow layer is the organizational one
Model capability is procured. Evaluation, cost control and observability are built, and each of them waits on the same input: a description of how the work actually runs, precise enough to say what correct looks like and what it currently costs.
That description is the real schedule of an AI programme. Produced first, it shortens everything after it. Produced under deadline in pre-production, it is the reason the prototype that demonstrated well in March is still not live in December.
See it. Fix it. Lead it.