Before you let an agent take over a real job — answer a phone, parse a radio call, draft a document that leaves your building — you have to know what you're actually working with. Not what the marketing copy claims. Not what a single cherry-picked demo shows. What the model can do under the exact conditions you'll put it in.
That's a different exercise than a leaderboard, and it's worth doing deliberately.
The trap: a leaderboard tells you almost nothing
Public model scores are measured on benchmark questions the model may have effectively memorized, under ideal sampling and unlimited time. They answer a different question than the one you actually care about, which is: will this model do the work reliably, the way I need it done, at the speed I need it done?
A benchmark ranks models against each other. A work evaluation tells you whether one model can do your job. Run your own test.
The method: a fixed, repeated, scored suite
We evaluate a candidate against a job by defining a suite of tasks that mirror the real work — then running every candidate through the same suite, under the same conditions, and scoring each answer against ground truth. Every candidate answers the exact same prompts. Nothing is skipped, nothing is improvised mid-run.
Three decisions make this worth doing:
- Keep the suite fixed. The moment you start adding or swapping prompts run-to-run, you can't compare anything.
- Score correctness, not completion. A model that produced an answer isn't the same as a model that produced the right answer. A wrong answer is a fail, and it gets no credit for having finished.
- Include the hard edge cases. The most instructive leg is the one where a reasonable-sounding answer is actually wrong, or where there are two defensible readings. That's where surface polish falls apart.
The part nobody talks about: the hardware is part of the test
Models run on something, and that something shapes what you observe. On a shared or constrained GPU, two models fighting for the same memory produce timings that are meaningless. A large model that spills past video memory runs at a fraction of the speed of one that fits. If you're not controlling for that, your speed comparison is fake.
Two practices keep the measurement clean:
- One model in memory at a time. Never run two candidates concurrently on the same card. The contention invalidates both sets of times.
- Measure cold start separately from warm decode. A model can load fast and then crawl the moment it starts generating. The two are different numbers and each matters.
What we actually find, run after run
The pattern repeats: the newest, fastest candidate almost always keeps up on correctness — and then reveals its character on one specific leg. Different builds handle the deliberately ambiguous case differently. Some refuse. Some over-think until they time out. Some answer cleanly because they don't notice the trap at all.
The one that catches it — that names the mechanism, walks the numbers, and flags the ambiguity — is the one you trust with real work. And the one that times out on that same leg is the one you don't, no matter how fast it looked elsewhere.
The model that looks best on the leaderboard is frequently not the model that wins your own fixed suite. It's the fit — model, quantization, sampling, hardware — that decides, and only your own test measures that.
The guardrail
The guardrail is the suite itself. It's a fixed fixture we re-run whenever we consider a change — a new model, a re-tuned quantization, a different card. Baseline it, then only change one variable at a time. If the numbers shift, you know what moved them.
The job doesn't change because a shiny new model shipped. The job changes only when your own testing says the new model does it measurably better.
The takeaway
The most capable-sounding model isn't the one you can ship. It's the one you validated — on your tasks, at your conditions, at your speed, scored for being right. The discipline of running that fixed, repeated, honestly-scored suite is what separates an agent you trust from a demo you're nervous about.
Benchmark the agent, not just the model — and let your own test, not the marketing, decide what gets a job.
← Back to Field Notes