Benchmark Contamination: a practical guide
A practical guide to benchmark contamination, using data from the AI & LLM Index. The goal is a shortlist and a decision, not a tour of every option.
What benchmark contamination is
benchmark contamination is a concept that shows up when teams design, evaluate or operate language model systems. The short definition matters less than the decisions it changes: what data goes into the prompt, how results are checked and where cost accumulates.
In the index, benchmark contamination is best understood through the models and benchmarks it touches. A concept is useful only if it helps you ask a sharper question about price, context, latency or quality.
Why it matters
Most production failures are not caused by a model being generally weak. They are caused by a mismatch between the task and the system around the model. benchmark contamination is one of the places where that mismatch shows up, which is why it deserves a explicit place in your design review.
How to evaluate it
- Define the behaviour you expect in one sentence.
- Build a small set of examples that should pass and examples that should fail.
- Measure both quality and cost per successful task.
- Re-run the set when you change model, prompt or retrieval.
Where it shows up in the index
You can see related signals on model pages and benchmark leaderboards. For a quick comparison, GPT-5.2 Pro and GPT-5.2 Max sit at different price and capability points, which makes them a useful pair for a small pilot.
Models worth testing
| Model | Provider | Input / 1M | Context | Avg score |
|---|---|---|---|---|
| GPT-5.2 Pro | OpenAI | $21.32 | 2,000,000 | 89.6 |
| GPT-5.2 Max | OpenAI | $21.39 | 1,000,000 | 89.2 |
| Claude Opus 4.5 Pro | Anthropic | $20.98 | 200,000 | 89.1 |
| GPT-5.1 Reasoning | OpenAI | $20.27 | 400,000 | 88.3 |
| Claude Opus 4.5 Reasoning | Anthropic | $20.91 | 2,000,000 | 88.2 |
Related
FAQ
What is the most important factor in benchmark contamination?
Fit to the task beats raw benchmark rank. Define the failure modes, pick the benchmarks that resemble them, then test a small shortlist on your own examples.
How many models should I evaluate?
Three is usually enough: a quality candidate, a value candidate and a fallback. More candidates add work without changing the decision.
How do I control cost?
Compare blended 3:1 price, watch output tokens, use caching where available and route simple requests to a cheaper model.