Coding-agent issue resolution on continuously collected recent GitHub issue/PR pairs under a fixed scaffold, repeated five times.
May republish1 incident recorded
Text equivalentSWE-rebench: no reproduced result. Remaining headroom has not been measured, so how much room is left is unknown — not zero.
Best reported
No result recorded in this register.
Saturation
Remaining headroom has not been measured. Unknown, not zero.
Contamination controls
issues collected after model cutoffs
date-window filtering
rows whose task dates precede model release are flagged as possibly contaminated
Licence
CC BY 4.0 Recorded as republishable, so its figures appear on this site.
Update cadence
Monthly rolling splits
Access path
Task instances via Hugging Face; scores are website-only
Incident log
Each entry states what was found, by whom, and when. The kind describes the finding, not anybody’s conduct. An entry without a primary source and a date is not published here.
2026-04-22Governance
The project reports finding contamination in some models and marks the affected rows on its own leaderboard, flagging evaluations whose task dates precede a model release. It is recorded here because a maintainer publishing its own doubt is a governance control, not a defect.