Reading past the headline benchmarks

01 · The number

The headline compresses away the conditions that made it true.

A model launches with a table of scores. The highest cells become screenshots, then buying arguments. This is understandable: teams need a fast way to compare systems that change every few weeks. The mistake is not reading the score. The mistake is treating it as a portable property of the model.

Every score is conditional. It depends on the task set, the prompt or scaffold, the tools, the sampling settings, the token and time budgets, the number of attempts, and the grader. Change one of these and the ranking can change. In an agent evaluation, the model may be only one part of a much larger execution system.

A useful review therefore starts with the evaluation contract. What work was attempted? What counted as success? What information and actions were available? How many independent trials ran? Which costs were included? Only then does the final percentage become interpretable.

02 · Five questions

Interrogate the benchmark in a fixed order.

The order matters. Teams often begin with the model column and work backward. Begin with the work. If the work does not resemble your use case, greater precision in the remaining details will not make the result relevant.

03 · Capability and reliability

One successful run proves possibility. Repeated runs reveal a product.

A pass rate is more useful than a best run because models and agent loops are nondeterministic. Yet even a pass rate needs context. How many attempts were made per task? Were results averaged or was the best scaffold selected for each model? Are confidence intervals wide? Did any task fail because the infrastructure broke rather than because the system lacked capability?

METR's time-horizon work shows why labels require care. Its 50% time horizon is the human-expert task duration at which an agent is predicted to succeed half the time. It is not the length of time the agent can operate, and it does not mean the agent can complete every job of that duration. The evaluated tasks are mostly self-contained software, machine-learning, and cybersecurity work with clear automated scoring. Real work is often messier and depends on tacit context or other people.

For production, ask for the reliability you need at the unit that matters. A coding agent that succeeds 70% of the time on independent patches may still be useful if all changes receive strong tests and review. The same reliability may be unacceptable for an unattended database migration. Capability and acceptable risk meet in the workflow, not on the leaderboard.

04 · System effects

For agents, you are usually comparing systems—not bare models.

Terminal-Bench states the problem directly: model and agent performance are hard to decouple. Agent scaffolds are often tuned to the tendencies of particular models. In its 2.0 evaluation, the researchers ran multiple agents across multiple models and selected each model's best-performing scaffold for the headline chart. That is a legitimate capability view. It is not a controlled estimate of model quality alone.

The harness decides what the model sees, which tools it can call, how results return, when history is compacted, how errors are presented, and when the run stops. A change in any of these can improve the score without changing the model. That is good news for builders: product quality is not locked to model quality. It is bad news for casual comparisons: two numbers with the same benchmark name may represent different systems.

Record the complete configuration for every evaluation. Pin model identifiers and provider versions when possible. Save the system prompt, tool schemas, environment image, budgets, and grader. If the result cannot be reproduced, treat it as an observation, not a baseline.

05 · Economics

Cost per successful outcome is the unit that survives contact with production.

Token price is easy to compare and often incomplete. Agent costs also include repeated context, tool calls, retries, failed runs, sandboxes, storage, evaluation, and review. Latency may become the dominant constraint when a workflow performs dozens of serial turns. A cheaper token can produce a more expensive completed task if it requires more attempts or more supervision.

Terminal-Bench 2.0 found a wide cost range across model and agent configurations. It also reported almost no correlation between average turn count and success, and no general rule that more output tokens produce better performance. More reasoning activity is not evidence of better work.

Build a Pareto view with at least four axes: task success, severe-error rate, p95 completion time, and cost per accepted output. Add review minutes if a person must inspect the result. A model is dominated when another option is at least as good on every constraint and better on one. This removes many impressive but impractical configurations quickly.

06 · Decision process

Use a funnel from public evidence to live traffic.

Public benchmarks are valuable at the top of the funnel. They expose broad strengths, reveal failure modes, and help narrow a large market. They should not carry the final decision.

OpenAI's evaluation guidance recommends a defined objective, a dataset tied to that objective, explicit metrics, comparison runs, and continuous evaluation. It also recommends typical, edge, and adversarial cases, with production and expert-curated data where available. The same discipline applies regardless of provider.

1. Screen: use relevant public evaluations to choose a small candidate set.
2. Replay: run each candidate on versioned examples from your own historical work.
3. Stress: add rare, ambiguous, adversarial, and high-impact cases.
4. Shadow: run on live inputs without allowing external side effects.
5. Gate: allow reversible actions with independent verification and rollback.
6. Operate: monitor quality, latency, cost, and drift; add failures to the evaluation set.

The best model is the least expensive system that meets your required quality and risk target on your work.