What Terminal-Bench tells us—and what it cannot

01 · The benchmark

Terminal-Bench tests whether an agent can leave a working environment behind.

Terminal-Bench is both a task format and an evaluation framework for agents that operate in command-line environments. Version 2.0 contains 89 hard tasks inspired by real technical workflows. Examples include configuring legacy systems, training models, building software, reverse engineering files, and reproducing research results.

Each task supplies an instruction, a containerized environment, tests, a human-written reference solution, and a time limit. The agent can inspect and change the environment through a terminal or other tools exposed by its harness. The grader checks properties of the final container state. It does not reward a convincing explanation or a plausible command trace.

That design is important. It turns the benchmark away from text imitation and toward completed work. It also allows different approaches to pass if they produce an acceptable result.

02 · What it tells us

The score is evidence of integrated technical execution.

A strong result shows that a model and harness can navigate unfamiliar technical environments, choose commands or tools, preserve enough context across many steps, and satisfy programmatic tests. Because the tasks vary, the result is more informative than success on a single coding pattern.

The benchmark is also hard enough to separate current systems. The paper reports that the best evaluated model–agent pairing resolved 63% of tasks on average and that some tasks remained unsolved by every configuration. A benchmark with headroom can reveal progress and expose where systems still break.

Repeated trials strengthen the result. The paper reports at least five runs for each supported model–agent combination, with more than 32,000 task trials overall. This makes variation visible and allows confidence intervals. For nondeterministic agents, that is a material improvement over a one-run demonstration.

Finally, the benchmark publishes its task format, harness, configuration files, and data. Teams can inspect what ran, reproduce selected results, and adapt the machinery to internal tasks. Reproducibility is a practical feature, not an academic decoration.

03 · The unit under test

The score belongs to the model, agent, tools, and budget together.

Terminal-Bench's authors warn that model and agent performance are hard to decouple. A scaffold decides how the model receives the task, what tools it can use, how tool results are returned, how much context persists, and when the run ends. Commercial agents may include substantial software inside the container. Two runs with the same model can therefore measure different systems.

The headline chart uses the best-performing scaffold tested for each model. This answers a reasonable question: how capable can this model be with a compatible agent? It does not answer a different question: how much better is this model under a fixed scaffold? For that, inspect matched rows or run a controlled evaluation.

This coupling mirrors production. A model with a lower general score may outperform inside a harness designed for its tool-use patterns. A stronger model may lose value if its latency forces serial queues, its output format is unstable, or its provider does not support the controls your workflow needs.

Procurement should name the full configuration. Say model X with agent Y, tool set Z, a given token budget, a given timeout, and N attempts. Anything less precise will be hard to reproduce after the next release.

Do not buy the model named in the winning row. Recreate the row, then replace its tasks with yours.

04 · What it cannot tell us

A clean container is not a company.

Terminal-Bench tasks are intentionally self-contained and programmatically scorable. Company work is often neither. A real repository carries undocumented conventions, partial migrations, ownership boundaries, customer commitments, private dependencies, noisy telemetry, and several conflicting sources of truth. Correctness may include maintainability, security, rollout safety, and stakeholder intent—not only a passing test suite.

The benchmark also cannot tell you whether an agent works well with your engineers. It does not measure how clearly the agent explains uncertainty, how efficiently a reviewer can inspect a patch, whether it respects local design decisions, or whether it makes the same costly mistake repeatedly across related tickets.

A final-state test can miss process risks. An agent may reach a passing state after reading data it should not access, invoking an unapproved network service, or making an irreversible change that the isolated task permits. Production controls must evaluate the action path as well as the result.

Nor is the aggregate score a forecast for one task category. Software engineering is the largest category in the dataset, but it is not a majority. The mix also includes scientific computing, security, data work, and other terminal tasks. Inspect per-task results and the actual list before mapping the average to a coding workflow.

05 · Failure analysis

The taxonomy may be more useful than the leaderboard.

The paper classifies failures that appear across long agent runs. The categories include disobeying the specification, repeating steps, missing termination conditions, losing context, drifting from the task, mismatching reasoning and action, stopping too early, and verifying weakly or not at all.

These labels translate directly into harness requirements. Repetition calls for retry counters and progress checks. Context loss calls for structured task state. Premature termination calls for independent completion gates. Weak verification calls for a stronger test contract. Reasoning–action mismatch calls for logging the requested operation and validating it at the tool boundary.

Use failure categories as product metrics. A single pass rate says whether the system completed the task. A distribution of failure modes says what to build next. If most failures come from missing dependencies, improve environment setup. If they come from weak verification, a larger model may not be the first fix.

06 · How to use it

Turn a public benchmark into an internal evaluation program.

First, use Terminal-Bench to identify candidate models and harnesses that can sustain multi-step terminal work. Inspect task-level results, cost, latency, and failure traces. Select two or three configurations, not one winner.

Next, reproduce a small public slice. This validates that your provider, settings, container runtime, and harness match the claimed setup. If the result diverges, investigate before you add private complexity.

Then create internal tasks in the same outcome-oriented style. Each task should have an isolated starting state, a clear instruction, an objective verifier, and a reference path. Start with recent resolved work that your engineers consider representative. Remove secrets, future information, and any shortcut the agent would not have in production.

1. Include routine work, edge cases, and incidents—not only showcase problems.
2. Run several independent attempts per task and report uncertainty.
3. Keep model, harness, tools, budgets, and environment versions fixed during comparisons.
4. Track severe failures separately from ordinary non-completion.
5. Measure accepted outcome cost and review time, not only API spend.
6. Promote real production failures into the next evaluation release.