The Score Changed Because the Test Changed: What Terminal-Bench 4.0 Actually Did
If an AI agent's benchmark score moved this summer, the test moved first. Terminal-Bench 4.0, released on 28 August 2026, did not rank new models against old rules; it changed the rules, and every entry on the leaderboard had to be run again. A benchmark score is only comparable to another score from the same benchmark version, on the same date, under the same rules, and most quotes you see carry none of that context. That makes a falling number ambiguous on its face: it can mean a model got worse, or it can mean the measurement got stricter.
What Terminal-Bench 4.0 Actually Changed
Terminal-Bench 4.0 is a maintenance release, not a new test: it removed eight tasks, fixed nineteen, and set a flat eight-hour agent timeout on every task, so any score from version 3.0 or earlier describes a different measurement and cannot be compared to a 4.0 score.
The maintainers' own announcement sets out the detail. Eight tasks left the set for four separate reasons: two were saturated, two produced refusals, two had public solutions circulating, and two had unresolved quality or platform-compatibility problems. A task counts as saturated when every class in every family of the latest model generation solves it five times out of five, at which point it no longer separates models. Nineteen tasks were fixed, updating instructions, environments and verifiers to address flakiness or misspecification found in leaderboard runs or reported by users.
According to the Terminal-Bench announcement, the resource calibration followed a method published by Anthropic engineering for quantifying infrastructure noise in agentic coding evaluations: run the tasks with generous resource headroom, then select settings that do not distort the final metrics. The announcement reports that frontier models now never or rarely hit the timeout.
One entry on the board illustrates why the cleanup matters. According to the same announcement, Claude Sonnet 5 consumed 21.6 billion tokens on its leaderboard run against 6.5 billion for Claude Opus 5, and sometimes hit timeouts and output-token limits. A scoring condition that punishes one model's verbosity rather than its capability is measurement noise, and the release exists partly to strip that kind of noise out.
The maintainers also confirmed something structural: Terminal-Bench is now a continuous benchmark with semantic versioning, so the major number carries meaning rather than marketing. That is what forced every leaderboard entry to be re-run, and it is the reason a score is only readable alongside the version that produced it.
Why the Same Benchmark Name Gives Incomparable Numbers
Each major version of Terminal-Bench has measured a different task set under different rules, so the top score of roughly 83% in May 2026, 34% in July and 52% in August are three separate measurements that happen to share a name, not a rise and fall in machine capability.
That version history is the arithmetic of this story. The board's top score on version 2.1, with 89 tasks, was about 83%. Version 3.0 replaced the task set with 74 new, harder tasks, and the top score fell to about 34%. Version 4.0 trimmed to 66 tasks, calibrated resources, and the top score recovered to around 52% on the August snapshot, with the latest entries reaching 58%. According to the analysis site Capital and Compute, which tracks these releases, none of those three numbers is comparable to either of the others: they are separate measurements that share a name.
Version | Released | Tasks | Top score |
|---|---|---|---|
2.1 | May 2026 | 89 | ~83% |
3.0 | July 2026 | 74 | ~34% |
4.0 | August 2026 | 66 | ~52% and rising |
Scores on the current board also carry confidence intervals, and the intervals are wide relative to the gaps. The August snapshot reported Claude Opus 5 at 51.82% and Claude Fable 5 at 44.55%, with confidence half-widths of several points on every row. Two entries separated by less than their combined intervals are not reliably separated, however tidy the ranking looks.
The reason a benchmark decays at all is that agentic tasks live in the world. A task that installs a package inherits that package's release schedule; a task that touches a website inherits its markup; a task with a public solution stops measuring problem-solving. Saturated, leaked and broken tasks do not fail loudly, they fail quietly, by scoring models on the wrong thing. Maintenance releases are how a benchmark stays honest, and the honest consequence is that old numbers expire.
The Bill Now Sits Next to the Score
Terminal-Bench 4.0's leaderboard publishes the total dollar cost of every run alongside the score, which turns model comparison from a capability question into a value question, and the value question has a much wider spread.
Each leaderboard entry is 330 trials: 66 tasks, attempted five times each, with every entry's full-precision total spend printed beside its accuracy. The board itself does not publish a per-task price, but dividing published cost by published solved trials is simple arithmetic, and according to Capital and Compute's derivation the cost of one solved task on the current board runs from $6.08 to $234.24, a spread of roughly 38 times. The most expensive entry per solved task is also tied for last place on accuracy.
The token figures in the announcement point the same way. A model that burns 21.6 billion tokens to score lower than a rival using 6.5 billion is not merely less capable, it is less capable per pound spent, and that difference is invisible in a score-only summary. Both figures come from the maintainers' own leaderboard runs.
Read the derived figures as ratios, not quotes. A benchmark trial is a worst-case unit of work: a hard, unfamiliar task, an eight-hour budget, no repository context. Nobody pays benchmark rates for production work. What the cost column buys is the ability to see that two models with similar scores can differ by an order of magnitude in what each solved task cost to produce.
What to Do With Any Benchmark Number From Now On
A benchmark figure is only usable if you can state its version, its snapshot date and its trial count in the same sentence; if any of the three is missing, the number is decoration, not evidence, and should be left out of the decision.
That check takes under a minute and it changes what you read. Version first: a score from Terminal-Bench 3.0 says nothing about 4.0, and the maintainers have said version bumps will now be frequent, so the half-life of any given number is short. Date second: leaderboard snapshots move as new entries are re-run, which is why this post pins its figures to the late-August snapshot and says when they were read. Trials third: 330 attempts per entry is what makes the confidence intervals calculable, and a score quoted without its uncertainty is a score inviting you to over-read it.
The habit generalises past this one benchmark. Any vendor slide, launch keynote or trade press article that quotes an AI score can be tested with the same three questions, and the ones that fail the test are telling you something: a number whose provenance cannot be stated is a number chosen for its size. Terminal-Bench earned attention this month precisely because it did the opposite, publishing its removals, its fixes and its reasoning in full. That is what a measurement built to be checked looks like, and it is a reasonable standard to hold every other score to before it influences a decision.
Sources
Tier A sources supporting figures in this piece:
- Terminal-Bench team, "Terminal-Bench 4.0" announcement, tbench.ai, read 10 September 2026: task removals (8, with the 2+2+2+2 breakdown), 19 task fixes, flat 8-hour timeout, saturation definition (5/5 across the latest generation), Sonnet 5 token figure (21.6B vs 6.5B for Opus 5), continuous-benchmark versioning. https://www.tbench.ai/news/terminal-bench-4-0
- Terminal-Bench 4.0 leaderboard (subject's own published data; mirrored because the live board is JavaScript-rendered): 66 tasks, 5 trials, 330 trials per entry, top scores of 51.82% and 58.18% on successive snapshots, confidence intervals. https://www.tbench.ai/leaderboard/terminal-bench/4.0
- GitHub release v4.0.0, harbor-framework/terminal-bench. https://github.com/harbor-framework/terminal-bench/releases/tag/v4.0.0
Background reading (qualitative guidance and derived figures, no independently verified claims):
- Capital and Compute, "Terminal-Bench 4.0: The $234 Solved Task": version history table (83% / 34% / 52% top scores) and the derived cost-per-solved-task column ($6.08 to $234.24). Both are that site's own arithmetic on published figures and are attributed to it in the text. https://capitalandcompute.net/blog/terminal-bench-4-0/
