SWE-bench and Coding Agent Benchmarks

SWE-bench was introduced by Carlos Jimenez and colleagues at Princeton in 2023 and published at ICLR 2024. It contains 2,294 task instances drawn from 12 widely used Python repositories, including Django, scikit-learn, sympy, matplotlib and requests. Each instance pairs a real GitHub issue with the pull request that fixed it. The system under test receives the issue text and the repository as it was before the fix, and must produce a patch. Scoring uses the tests from the original pull request. FAIL_TO_PASS tests failed before the human fix and passed after it, so they check that the issue is solved. PASS_TO_PASS tests passed both before and after, so they check that nothing else broke. An instance counts as resolved only if all of these pass, and the headline number is the percentage resolved. At launch, the best models resolved only a few percent of tasks. Two subsets matter. SWE-bench Lite has 300 instances chosen to be cheaper and more self-contained. SWE-bench Verified, released in 2024 by OpenAI with the SWE-bench authors, has 500 instances that human engineers screened to remove vague issue descriptions or tests that would reject valid solutions. Verified is now the most commonly quoted split. Scores reflect a model plus its scaffold, the program that lets it browse files, run commands and edit code. SWE-agent from Princeton and the Agentless pipeline, which localizes, repairs and validates without an open-ended agent loop, are well-known examples. The same model can score very differently in different scaffolds. Common misreadings: the original set is Python-only and covers a dozen repositories; passing tests does not guarantee maintainable code; the repositories and fixes are public, so contamination is possible; and multi-attempt results are not comparable to single-attempt ones. Related benchmarks such as SWE-bench Multilingual, SWE-Lancer and Terminal-Bench probe other languages, freelance tasks and command-line work.