MMLU (Massive Multitask Language Understanding) is a multiple-choice benchmark spanning 57 subjects from law to physics, at high-school to professional difficulty. The score is the percentage of questions answered correctly; random guessing scores 25%, and frontier models now score around 90%.

What do AI coding benchmarks measure?

Coding benchmarks test whether generated code actually runs and passes tests — from short function-completion problems (HumanEval style) to fixing real GitHub issues in large codebases (SWE-bench style), with fresh contest problems (LiveCodeBench style) used to resist memorization.

Why should benchmark scores be read with caution?

Because of contamination (test questions leaking into training data), saturation (differences above ~90% are mostly noise), and the proxy gap — a benchmark approximates an ability rather than measuring your actual task. Use leaderboards to shortlist models, then test on your own workload.

The Benchmarks, Explained

The AI model leaderboard ranks models by a composite index averaging three scores: general knowledge (MMLU), coding, and math. This page explains what each category actually measures, where the numbers come from, and — just as important — what they don't tell you.

General knowledge `MMLU`

MMLU — Massive Multitask Language Understanding — is the closest thing AI has to a general exam. It's a collection of multiple-choice questions spanning 57 subjects: law, medicine, history, physics, accounting, ethics, and more, at difficulties from high school to professional level. A model answers thousands of questions and the score is simply the percentage it gets right.

Random guessing scores 25%. An expert human in a single field scores around 90% in that field; frontier models now score around 90% across all fields at once, which is why newer, harder variants (like MMLU-Pro, with ten answer choices and more reasoning-heavy questions) are increasingly used alongside it.

Coding

Coding benchmarks test whether a model can write programs that actually run and pass tests — not whether the code merely looks plausible. The family ranges from short, self-contained problems to realistic engineering work:

Function-level tests (the HumanEval style): implement a described function; unit tests decide pass or fail.
Competitive-programming tests (LiveCodeBench and similar): fresh contest problems, which resist memorization because they're published after a model's training data was collected.
Real-repository tests (the SWE-bench style): fix an actual GitHub issue inside a large codebase, where success means the project's own test suite passes.

A high coding score means the model reliably turns a clear specification into working code. It says less about vaguer real-world skills: navigating a messy legacy codebase, asking the right clarifying question, or knowing when not to write code.

Math

Math benchmarks measure multi-step reasoning with a verifiable answer — the model either derives the correct number or it doesn't, which makes math the cleanest signal of raw reasoning ability. Difficulty tiers run from grade-school word problems (GSM8K), through competition mathematics (the MATH dataset), up to olympiad-qualifier problems (AIME), where even strong human students miss most questions. Frontier models jumped from single-digit AIME scores to near-perfect within roughly two years, which is why math benchmarks saturate fastest and get replaced most often.

How the index works

The leaderboard's index is the plain average of the three category scores, equally weighted. No secret sauce — the goal is a single sortable number that's easy to reason about, with the per-category columns right there when your use case leans one way (a coding assistant should be picked by the coding column, not the index).

Read benchmarks with care. Three standing caveats: contamination — benchmark questions can leak into training data, inflating scores (fresh-problem benchmarks exist precisely to counter this); saturation — once top models cluster above ~90%, a benchmark stops discriminating and differences within a point or two are noise; and proxy gap — a benchmark is a proxy for ability, not the ability itself. The best model for you is the one that performs on your actual task — use the leaderboard to build a shortlist, then test.

Full leaderboard → Head-to-head comparisons → What do they cost? →