There's a moment every serious agent operator hits: a model that looks perfect on the spec sheet — right size, right quantization, right benchmark scores — and then underperforms in the real workload. Not dramatically. Not with an error you can point at. Just slower than it should be, in a way that compounds across a fleet of agents running all day.
The instinct is to blame the model and swap it. That's usually wrong. The model isn't broken — it's misunderstood. And the fix is often a repair, not a replacement.
This note is about how we test models, why we stopped trusting spec sheets, and the time we had to open one up and stitch on a new part to get it running at full speed.
The trap: the spec sheet lies by omission
A model's spec sheet tells you what it is — parameter count, quantization, context window, benchmark scores. It doesn't tell you how it behaves in your workload. Those are different things.
We run a fleet of autonomous agents doing real work: legal and contract analysis, tool-calling, chain-of-thought reasoning, long-context retrieval. When we bring in a new model, we don't ask "is it good?" We ask "is it good at what we do, at the speed we need, reliably, all day?"
The model that bit us looked great on paper. Right class, right size, solid benchmarks. But in our battery it was consistently slower than its architecture said it should be. Not a little — enough to matter when a dozen agents are all making calls.
The diagnosis: measure, don't trust
We don't trust leaderboards. We don't trust the vendor's numbers. We run a deterministic battery — a fixed set of prompts across the workloads we actually run, scored for both accuracy and wall-clock throughput, repeated head-to-head against the models we already trust.
That battery is what caught it. The model answered correctly — accuracy was fine. But throughput was ~15% below where its architecture should have put it. That gap is the tell. A model that's accurate but slow is a model with a bottleneck, and a bottleneck is a part, not a personality.
We dug in. The culprit was the MTP head — the multi-token-prediction draft head that makes speculative decoding fast. Speculative decoding works by having a small, fast draft head predict several tokens ahead, then letting the big model verify them in parallel. When that draft head is broken, missing, or mismatched, the model falls back to generating one token at a time. It's correct — just slow. And nothing on the spec sheet tells you the draft head is broken.
The fix: stitch on a new head
The repair was to replace the MTP head — take a compatible draft head and splice it onto the model so speculative decoding works at full speed again. Not by swapping to a different model. By opening this one up and fixing the part that was broken.
Models are not monolithic. They're assemblies — a main body, an embedding layer, a draft head, a tokenizer. When one part is wrong, you can often repair it rather than discard the whole assembly. A model that's 95% right and has one broken part is a repair candidate, not a write-off.
The result: the same model, same accuracy, but at the throughput its architecture always promised. The fix wasn't a different model. It was understanding the one we had.
The guardrail: the battery is the floor
The reason we caught it is the reason it won't recur: we measure, we don't assume. The deterministic battery is the floor under every model we run. It scores accuracy and throughput, on our workload, head-to-head against known-good baselines. A model doesn't get into the fleet on its spec sheet. It gets in by passing the battery.
And when something underperforms, the battery tells us where — accuracy or speed — which tells us what kind of problem we're looking at. A speed gap points at a bottleneck. A bottleneck points at a part. A part can be repaired.
The takeaway
Benchmark on your workload, not the leaderboard. Measure throughput, not just accuracy. And when a model underperforms, don't reach for the replacement — reach for the diagnostic. Sometimes the model isn't the problem. Sometimes it's one broken part, and the fix is a repair, not a write-off.
The models that matter aren't the ones with the best spec sheets. They're the ones you understand well enough to fix.