Every benchmark we threw away, and why

We started this site with a list of about forty benchmarks and an assumption that the job was ingesting them.

It was not. Most of the work turned out to be deciding which ones to leave out, and by the time we had finished the list had roughly halved. This is the record of what went and on what grounds, because a ranking is only as honest as the set it refuses to use, and nobody ever publishes the refusals.

Three grounds, and they are independent. A benchmark only has to fail one.

One: it is saturated

A benchmark works by separating things. Once the models you care about are all clustered at the top of its range, it has stopped separating them, and the ordering that is left is noise wearing a leaderboard’s clothing.

AIME 2025 is the clearest example. It is a genuinely good set of problems and it did real work for a couple of years; there is simply nothing left of it at the frontier, and the models at the top are tied. A one-place difference there tells you about run variance, not ability. The same has happened to GPQA Diamond, where the leaders sit inside the spread between reruns. Fiction.LiveBench and SimpleQA are done too.

MathArena is the interesting case, because they tried to fix it and said publicly that they could not. They went to build a fresh set for this year, lined up a batch of candidate problems, and found a frontier model solved every one of them at least once. So they did not ship it. They wrote up why instead.1

I would rather read that than another leaderboard. It is a maintainer telling you the ground has moved under their own project, at some cost to the project, and it is the clearest single piece of evidence that this category of benchmark has run out of road.

Two: the tasks themselves are broken

This one surprised us, and it is the reason we now read a benchmark’s own audits before we read its scores.

Somebody also demonstrated that a fake curl wrapper, which does no real work at all, scored full marks on the original Terminal-Bench harness.2 An uncorrected gap of a few points on MMLU is not interpretable either; the error bar from the questions alone is wider than most of the differences being reported.

BrowseComp produced the strangest finding of the lot. Anthropic disclosed nine ordinary leaks in it, plus two cases where Claude worked out it was being evaluated, went looking for the answer key, and decrypted it.3 They published that. It took some nerve, and this field needs more of it.

What all of these have in common is that the defect was found by the people running the thing, or by a lab acting against its own interest, and published. That is the good outcome. The bad outcome is the benchmark where nobody has looked, and there are more of those than there are of these.

Three: we are not allowed to republish it

The least interesting reason and the most absolute one. Some benchmark data is published under terms that do not permit commercial redistribution. MathArena’s sets and AIME sit there. We can read them. We cannot put their figures on this site, and we do not.

That is not a complaint about the licence. The people who built those sets chose their terms and the terms are clear; the failure would be ours if we quoted the numbers anyway and hoped nobody minded. So the rule is enforced in the query layer rather than left to whoever is building a page that week. A restricted source cannot reach a rendered page by someone forgetting a condition, because there is no condition to forget.

There is a second tier here that is messier: benchmarks published with no licence statement at all. SWE-bench Verified is one. LiveBench and LiveCodeBench are others. Silence is not permission, so those get treated as restricted until somebody tells us otherwise. If you maintain one of them and that is wrong, we would genuinely like to hear it; being able to use them would improve this site.

The test we actually apply

Those three grounds sound tidier written down than they are in practice, so here is the working version. Four questions, in order, and a benchmark has to survive all of them.

• Can the models we care about still be separated by it? Not "is it hard", which is a different question. A set can be brutally hard and still useless if everything at the frontier fails it equally. What we want is spread, and enough headroom that next quarter’s models will not erase it.
• Do we know how the questions were made, and can they be remade? A fixed set ages from the day it ships. A set with a pipeline behind it, so fresh questions can be generated on a cadence, ages much more slowly.
• Has anyone looked for defects, and did they publish what they found? An audited benchmark with a known defect rate is more useful to us than an unaudited one with an unknown rate, because the first can be reasoned about and the second cannot. Absence of bad news is usually absence of looking.
• Are we permitted to quote it and credit the authors? Yes or no. There is no partial credit here and no interpretation to do.

Nothing in that list is about a benchmark’s reputation, and that is deliberate. The most-quoted set in the industry fails the third question badly, and the best-behaved one on our list is one most people have not heard of.

What survived

For coding: SWE-rebench, Terminal-Bench 2.0 after its audit, Aider Polyglot, tau²-bench, and SWE-bench Pro with its defect estimate attached. For general capability: LMArena, HLE, MMLU-Pro, BFCL, MCP-Universe and OSWorld.4

The pattern in that list is not that they are the hardest or the newest. It is that each one either refreshes its questions on a schedule, or audits itself and publishes the corrections, or does both, and all of them carry a licence that lets us quote them and credit the people who did the work.

Genuinely unsaturated is a shorter list than you would hope. Most of the field is somewhere above eighty and closing.

It is worth saying what this costs us. Dropping half the field means our coverage is thinner than a site that ingests everything, and there are models we can say less about than we would like, particularly newer ones with only vendor numbers behind them. We would rather have a narrow set we can defend than a wide one we cannot. But it is a trade, and you should know we made it.

The uncomfortable bit

Everything above is a judgement, and judgements are where a site like this can quietly go wrong. We have tried to make ours checkable rather than asking you to trust them. Every benchmark we dropped is in the register with the reason attached and a link to the primary source. The incident log is there whether or not the incident is convenient for us. When we are wrong about one, the correction goes on the page and stays on it.

What we cannot offer is a set that is clean. There is not one. The best available position is a set whose weaknesses are documented by the people maintaining it, ranked by someone who will tell you which weaknesses they have decided to live with. This is that list. It will be wrong in a year, and the interesting question then will be which of the survivors went first.