MMLU-Pro: what the benchmark actually measures
A practical guide to mmlu-pro, using data from the AI & LLM Index. The goal is a shortlist and a decision, not a tour of every option.
What MMLU-Pro measures
MMLU-Pro is a knowledge benchmark. A harder version of MMLU with ten answer options and more reasoning-heavy questions.
The index scores all 600 models on a 0–100 scale. The average is 73.5 and the top score is 91.6.
How to read the scores
A benchmark score is a compressed signal. It tells you how a model performed on a specific test distribution, not how it will perform on your data. Read the leaderboard as a way to form a shortlist, then validate the shortlist with your own examples.
On MMLU-Pro, the gap between the leader and the median is 18.1 points. That gap is the useful part of the chart: it shows how much separation the benchmark creates.
Leaders
GPT-5.2 Pro leads at 91.6, followed by GPT-5.1 Max at 91.3 and Claude Opus 4.5 Pro at 91.2.
Limitations
- Scores in this index are modelled estimates, not official results.
- Benchmark contamination can inflate scores for models trained on similar data.
- A high score does not guarantee good performance on a different task shape.
- Prompt format and evaluation harness can move results by several points.
Top 5 on MMLU-Pro
| Model | Provider | Score | Input / 1M |
|---|---|---|---|
| GPT-5.2 Pro | OpenAI | 91.6 | $21.32 |
| GPT-5.1 Max | OpenAI | 91.3 | $20.50 |
| Claude Opus 4.5 Pro | Anthropic | 91.2 | $20.98 |
| Claude Opus 4.5 Reasoning | Anthropic | 90.7 | $20.91 |
| Gemini 3 Pro Reasoning | Google DeepMind | 90.1 | $19.69 |
Related
FAQ
Who leads MMLU-Pro?
GPT-5.2 Pro leads this index at 91.6.
What is the most important factor in mmlu-pro?
Fit to the task beats raw benchmark rank. Define the failure modes, pick the benchmarks that resemble them, then test a small shortlist on your own examples.
How many models should I evaluate?
Three is usually enough: a quality candidate, a value candidate and a fallback. More candidates add work without changing the decision.
How do I control cost?
Compare blended 3:1 price, watch output tokens, use caching where available and route simple requests to a cheaper model.