GUIDE Technique
Benchmarks SWE et agents de codage
SWE-bench is a benchmark that tests whether an AI system can resolve real GitHub issues by editing a Python codebase so that the project's own tests pass.
Sur cette page4 minutes de lecture
Aperçu
It matters because it is one of the most cited measures of coding agents, and its scores are easy to over-read without knowing how tasks, agent scaffolds and test checks work.
Plongée profonde
SWE-bench was introduced by Carlos Jimenez and colleagues at Princeton in 2023 and published at ICLR 2024. It contains 2,294 task instances drawn from 12 widely used Python repositories, including Django, scikit-learn, sympy, matplotlib and requests. Each instance pairs a real GitHub issue with the pull request that fixed it. The system under test receives the issue text and the repository as it was before the fix, and must produce a patch. Scoring uses the tests from the original pull request. FAIL_TO_PASS tests failed before the human fix and passed after it, so they check that the issue is solved. PASS_TO_PASS tests passed both before and after, so they check that nothing else broke. An instance counts as resolved only if all of these pass, and the headline number is the percentage resolved. At launch, the best models resolved only a few percent of tasks. Two subsets matter. SWE-bench Lite has 300 instances chosen to be cheaper and more self-contained. SWE-bench Verified, released in 2024 by OpenAI with the SWE-bench authors, has 500 instances that human engineers screened to remove vague issue descriptions or tests that would reject valid solutions. Verified is now the most commonly quoted split. Scores reflect a model plus its scaffold, the program that lets it browse files, run commands and edit code. SWE-agent from Princeton and the Agentless pipeline, which localizes, repairs and validates without an open-ended agent loop, are well-known examples. The same model can score very differently in different scaffolds. Common misreadings: the original set is Python-only and covers a dozen repositories; passing tests does not guarantee maintainable code; the repositories and fixes are public, so contamination is possible; and multi-attempt results are not comparable to single-attempt ones. Related benchmarks such as SWE-bench Multilingual, SWE-Lancer and Terminal-Bench probe other languages, freelance tasks and command-line work.
Impact stratégique
Coût et budget
Les décisions en matière d'architecture déterminent les performances et les coûts d'exploitation pendant des années.
Décisions plus claires
La formation technique aide les équipes à choisir la bonne pile, pas seulement la plus récente.
Contrôle qualité
De meilleurs choix d’ingénierie réduisent les incidents de fiabilité en production.
The Future of SWE-bench and Coding Agent Benchmarks
As top scores on SWE-bench Verified rise, the benchmark separates leading systems less clearly, and attention is moving toward newer task sets with fresher issues, more programming languages, longer multi-step work and reporting of cost and reliability. Contamination is a lasting concern because the source repositories are public, which favors benchmarks that add recent tasks. For teams choosing tools, public leaderboards are a starting point; testing agents on their own codebases and tickets remains the most reliable signal.
Mise en œuvre dans le monde réel
A task hands an agent a Django issue describing incorrect behavior plus a checkout of the repository at the commit before the fix; the agent must write a patch, which is then judged by tests that the original human fix made pass.
A vendor reports a SWE-bench Verified score using its own agent harness and several attempts per task; a buyer should confirm whether it is a single-attempt result before comparing it with another vendor's number.
An engineering team builds an internal SWE-bench-style set from its own private repositories and resolved tickets to check whether an agent that ranks well publicly actually works on their code.
Researchers use SWE-bench Multimodal, where issues come from JavaScript front-end projects and often include screenshots of visual bugs, to test skills the Python-only original cannot measure.
Risques et garde-fous
L’optimisation d’un benchmark peut masquer des faiblesses plus larges du système.
Les coûts d’infrastructure et de maintenance sont souvent sous-estimés.
Les lacunes en matière de sécurité et d’observabilité peuvent se creuser à mesure que les systèmes deviennent plus complexes.
Feuille de route de mise en œuvre
Définissez les objectifs de latence, de qualité et de coût avant la mise en œuvre.
Benchmark dans des conditions de charge et de données réalistes.
Surveillance des instruments pour détecter les erreurs, la dérive et l'impact sur l'utilisateur.
Préparez les chemins de restauration et de réponse aux incidents avant la mise à l’échelle.
Continuez à explorer
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the SWE-bench and Coding Agent Benchmarks quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Questions fréquemment posées
What is SWE-bench and Coding Agent Benchmarks?
SWE-bench is a benchmark that tests whether an AI system can resolve real GitHub issues by editing a Python codebase so that the project's own tests pass. It matters because it is one of the most cited measures of coding agents, and its scores are easy to over-read without knowing how tasks, agent scaffolds and test checks work.
What does a SWE-bench task ask the system to do?
Each instance pairs a real issue with a repository snapshot, and the system must generate a code patch that fixes it.
What are FAIL_TO_PASS tests?
They confirm the issue is actually resolved, since they only pass when the fix is in place.
What is the role of PASS_TO_PASS tests?
They passed before and after the human fix, so they guard against regressions introduced by the agent's patch.
What is SWE-bench Verified?
Verified was released in 2024 by OpenAI with the SWE-bench authors, filtering out tasks that were underspecified or had tests rejecting valid solutions.
How many instances are in SWE-bench Lite?
Lite is a 300-instance subset chosen to be cheaper and more self-contained.
Continuez à apprendre
Guides associés
Plus de guides sélectionnés pour ce sujet