OpenAI quietly stopped using SWE-bench Verified. So did we.

SWE-bench Verified was meant to be the clean one.

It exists because people didn't trust the original SWE-bench. A panel went through the tasks, threw out the ones that were unsolvable or badly specified, and published a subset everybody could point at.1 For about eighteen months it was the number you quoted when you wanted to say a model could do real engineering work. It is in launch posts, in pitch decks, in the paragraph of a model card where the lab tells you it can handle your codebase.

That is the whole story, and I think it is the most useful thing anyone published about benchmarks last year. It is also the reason this site does not rank anything on SWE-bench Verified.

What a defective task actually does

A SWE-bench task is meant to be a small, honest exam. You get a real issue from a real repository, the repository at the commit before the fix, and a set of tests that pass once the issue is genuinely resolved.2 The model writes a patch. The tests run. You get a pass or a fail.

That only works if the tests are actually testing the thing the issue describes. When they are not, the score stops measuring what you think it measures, and it does so quietly.

The audit did not break out a share for the third category1, so we leave the cell empty rather than inferring a number. An absent measurement is not a small one.

None of these is exotic. Anyone who has written tests for a real codebase has shipped all three at some point. The problem is not that defects exist; it is that a benchmark converts them into a number with two decimal places, and the decimal places survive the journey into the slide deck long after the caveat has fallen off.

Every frontier model they tested showed evidence of prior exposure to some of the tasks. Not one clean model in the set.

Why exposure is the worse half of it

A defect rate is annoying but survivable. If the errors are random, they add noise, everyone’s score moves down a bit, and the ranking roughly holds.

Exposure does not behave like that. It is not noise; it is an advantage distributed unevenly across the models being compared, according to what happened to be in each one’s training data. Two models with the same underlying ability can land points apart because one of them saw more of the exam. Under those conditions the gap between two models stops being a measurement of anything, and there is no correction you can apply afterwards to recover it.

So the honest read of that defect rate is not "the scores are wrong by that much". It is "we can no longer say what the scores mean, and we cannot bound the error either".

What this is not

That distinction is the editorial line of this whole site, and it is not squeamishness. A statement about a measurement is defensible from disclosed method. A statement about intent is a different kind of claim entirely, and we do not have the evidence to make it. Anyone who tells you they can read a lab’s intentions off a leaderboard is selling you something.

What we replaced it with

We rank coding work on SWE-rebench instead.3

Three properties earn it that spot, and they are worth spelling out because they are the properties any replacement would need.

• The tasks are written after the models are. It builds from issues raised in the months just gone, on a rolling monthly cadence. A model cannot have seen a problem that did not exist when it was trained. That does not make exposure impossible forever, but it puts a clock on it, and the clock is visible.
• It runs each task five times. One run tells you what a model did once. Five tells you whether it can do it reliably, and reliability is most of what you actually care about when you are deciding whether to let something touch your repository.
• It flags its own suspect rows. When it thinks a task may be contaminated, it says so in the data rather than in a footnote. A benchmark that publishes its own weak spots is doing the thing we are asking every benchmark to do.

It is licensed so that we can republish the numbers and credit the people who produced them. That matters more than it sounds. A good benchmark we cannot lawfully quote is a benchmark this site cannot use, and there are several of those.

SWE-rebench is also harder, and the leaderboard is lower and messier as a result. That is the point. Anything sitting near the top of its range has stopped telling you much.

What this changes here

• SWE-bench Verified appears in our benchmark register with its incident log attached, and nowhere else. You can read what happened to it. You cannot get a model ranked by it.
• Any coding score we show carries the effort setting it was produced at, because a score without its effort is unusable, and because the setting a lab benchmarks at is often not the one your subscription gives you.
• Where we have no independent number for a model on a task, the page says so and leaves the space visibly empty. It does not render a zero, and it does not quietly fall back to the vendor’s own figure with the label filed off.

None of that makes our numbers right. It makes them checkable, which is the only property we can actually offer.

The uncomfortable version of all this: the benchmark everyone used to quote was the one that had been examined most carefully, by people acting in good faith, and it still degraded quietly enough that it took a first-party audit to notice. Whatever replaces it will degrade too. The useful question is not which leaderboard is clean; it is which ones tell you when they have stopped being.