Benchmark
Also known as: benchmarks · eval
A benchmark is a standardized test with known answers, used to compare models fairly. MMLU, HumanEval (code), and various reasoning suites are the usual suspects.
They let us say “model A beats model B on coding” with some basis.
- Strength: apples-to-apples comparison.
- Weakness: a model can be trained on the benchmark, inflating the score — and no benchmark covers everything you’ll actually use.
Read scores skeptically: a high benchmark number is necessary but not sufficient. What a model does on *your* task is the only number that really counts.