12 models. 9 suites.
Know what each model is good at.

Leaderboard

All model runs, ranked

• The rest
• AnthropicClaude models, one suite per effort
• OpenAIGPT models, one suite per effort

Scores are percentages, 0–100, graded by code against seeded database state. ▫ marks a row whose eight non-restraint benches ran on the previous fixture: rank it, but do not read a single cell against a ▪ row. Costs come from three different bases — see the run economics below. Output tokens are observed completion tokens for that row's nine-bench run: a workload and verbosity signal, not a quality score.

Weighting

Not every bench is worth the same

The observed range is the lowest and highest score any configuration reached on that bench. A tenth bench, context scaling, was excluded for every model: it never produced a graded case, so it is not on this page and not in any score.

Per-bench breakdown

What drives the ranking

Cell shading is the score itself, one hue, no thresholds. Bold is the column best. Every value is a single measurement: the same bench has swung more than thirty points between identical runs elsewhere in this campaign, so treat close cells as a tie. ▫ rows had their eight non-restraint benches measured on the previous fixture — the restraint column was re-measured on the current fixture for everyone, so that one column is comparable across the whole table.

Reasoning effort

More thinking costs time, not much score

Thick bar: weighted score on a 0–100 scale. Thin bar underneath: wall clock, scaled against the longest run on the board (48.5 minutes). The footer compares the lowest effort with the highest. Read only ▪-to-▪ pairs as a clean effort effect — a ▫ row had its eight non-restraint benches measured on the previous fixture. Wall clock is operational time, not model latency: the hosted model group is throttled and Anthropic/OpenAI runs pay process-spawn overhead per call.

Max output

The best team mixes strengths

This is a ceiling, not a result. It assumes a router that always picks correctly — nothing here demonstrates such a router, and building one is its own hard problem. Every cell behind it is a single measurement with real run-to-run variance. Treat 89% as the headroom a routing strategy could chase, not a score any deployment has achieved.

Cost vs quality

185× the cost buys 6.4 more points

• The rest
• AnthropicClaude models, one suite per effort
• OpenAIGPT models, one suite per effort

Dot size is reasoning effort — smallest low, largest xhigh; the hosted provider models ran at their provider's default and appear once. Costs are shown as the dollar total for each suite. OpenAI's models actually bill to a subscription, so their dollars answer what the same work would have cost at published usage rates, not what was charged.

Run economics

What the whole campaign cost

• The rest
• AnthropicClaude models, one suite per effort
• OpenAIGPT models, one suite per effort

Wall clock is scored-bench time, an operational number rather than model latency: the hosted model group is throttled to 15 requests a minute, while Anthropic and OpenAI runs are unthrottled but pay process-spawn overhead on every call. One row to read with care — DeepSeek V4 Flash 0731 is the cheapest full suite at $0.09 and scores 72.4%; low cost is not the same as broad capability.

Why a separate benchmark

We went looking for a benchmark that measured this. There wasn't one.

Nothing on this page was graded by another model. Each bench seeds its own workspace in an isolated database, runs against a fingerprinted fixture, and is scored by plain code — set overlap, precision and recall, or simply reading the project afterwards to see whether the change is there. A grader that cannot be charmed is a harsh one, which is why the scores are lower than you might expect.