SWE-bench Verified: what the benchmark actually measures
A practical guide to swe-bench-verified, using data from the AI & LLM Index. The goal is a shortlist and a decision, not a tour of every option.
What SWE-bench Verified measures
SWE-bench Verified is a coding benchmark. Real GitHub issues resolved by an AI coding agent, verified by human reviewers.
The index scores all 600 models on a 0–100 scale. The average is 57.5 and the top score is 81.7.
How to read the scores
A benchmark score is a compressed signal. It tells you how a model performed on a specific test distribution, not how it will perform on your data. Read the leaderboard as a way to form a shortlist, then validate the shortlist with your own examples.
On SWE-bench Verified, the gap between the leader and the median is 24.2 points. That gap is the useful part of the chart: it shows how much separation the benchmark creates.
Leaders
GPT-5.2 Pro leads at 81.7, followed by Claude Opus 4.5 Pro at 81.1 and GPT-5.2 Max at 80.5.
Limitations
- Scores in this index are modelled estimates, not official results.
- Benchmark contamination can inflate scores for models trained on similar data.
- A high score does not guarantee good performance on a different task shape.
- Prompt format and evaluation harness can move results by several points.
Top 5 on SWE-bench Verified
| Model | Provider | Score | Input / 1M |
|---|---|---|---|
| GPT-5.2 Pro | OpenAI | 81.7 | $21.32 |
| Claude Opus 4.5 Pro | Anthropic | 81.1 | $20.98 |
| GPT-5.2 Max | OpenAI | 80.5 | $21.39 |
| Claude Sonnet 4.5 Max | Anthropic | 80.2 | $19.90 |
| GPT-5.2 | OpenAI | 80.2 | $19.71 |
Related
FAQ
Who leads SWE-bench Verified?
GPT-5.2 Pro leads this index at 81.7.
What is the most important factor in swe-bench-verified?
Fit to the task beats raw benchmark rank. Define the failure modes, pick the benchmarks that resemble them, then test a small shortlist on your own examples.
How many models should I evaluate?
Three is usually enough: a quality candidate, a value candidate and a fallback. More candidates add work without changing the decision.
How do I control cost?
Compare blended 3:1 price, watch output tokens, use caching where available and route simple requests to a cheaper model.