Loupe / Benchmarks / SWE-bench Verified

SWE-bench Verified

Resolving real Python repository issues by generating patches that must pass repository tests.

May republish2 incidents recorded

Text equivalentSWE-bench Verified: best reproduced result 83.5 points. Remaining headroom has not been measured, so how much room is left is unknown — not zero.

Best reproduced: 0.8347107438016529Claude Opus 4.7max effortIndependentepoch-evaluated
Observed 2026-04-20 · source
Saturation: Remaining headroom has not been measured. Unknown, not zero.
Contamination controls: independently run under a fixed harness
restricted problem content on the FrontierMath tiers
Licence: CC BY 4.0 Recorded as republishable, so its figures appear on this site.
Update cadence: Maintainer-run; models are added when Epoch evaluates them
Access path: CSV inside the Epoch benchmark data archive

Incident log

Each entry states what was found, by whom, and when. The kind describes the finding, not anybody’s conduct. An entry without a primary source and a date is not published here.

2026-02-23Withdrawal
OpenAI stopped evaluating on SWE-bench Verified. It audited 138 hard tasks and reported that 59.4 per cent carried material test or prompt defects, and that every frontier model it tested could reproduce some gold patch or prompt detail.
openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
2025-06-14Leak
A forensic probe reported recovering file paths from issue text alone at up to 76 per cent on this set, against up to 53 per cent elsewhere, and reproducing 5-gram functions at 35 per cent against 18 per cent. The probe measures recovery; it does not establish how the material was learned.
arxiv.org/abs/2506.12286