Ai2 published a method that shows what individual benchmark questions actually measure

Ai2 published BenchMIRT on September 1, 2026, a method for auditing language model benchmarks at the level of the individual question rather than the final score. The premise is that a benchmark is designed to measure one ability, while the questions inside it often depend on several. When those signals are averaged into a single number, the number stops saying what its name promises.

How the method works

Item response theory comes from psychometrics and starts from a plain observation: not every question tells you the same amount about whoever is taking the test. Some are harder, and some do a better job of separating stronger performers from weaker ones. Researchers have applied the single-dimensional version to individual benchmarks before. BenchMIRT uses the multidimensional variant, so it can separate several capabilities that feed into the same question.

It estimates two things at once. For a model, it estimates strength on the capabilities present across the chosen benchmarks. For a question, it estimates difficulty and how well that question distinguishes stronger models from weaker ones. Ai2 trained it on results from 100 language models across 16 benchmarks and more than 34,000 questions. Six of them are reasoning benchmarks, including MMLU-Pro, GPQA, MATH and BBH, and ten come from its Olmo 3 safety suite. Nobody told the method which benchmark measured what, and it recovered two dominant dimensions anyway: safety and general reasoning.

What it found in existing benchmarks

For most of the set the intended focus held up. Three cases did not, and each of them is the kind of thing that quietly bends a comparison between two models.

• BBQ tests social bias and is usually filed under safety, yet it lined up far more closely with general reasoning. A low score may therefore reflect trouble reasoning through the question rather than bias behavior.
• WMDP tests dangerous dual-use knowledge and also tracked reasoning rather than safety, in the opposite direction. Stronger reasoning went with a lower score, because refusing to supply the knowledge is what the benchmark counts as correct.
• HarmBench mixes signals inside itself, with its standard and contextual harmful requests aligning to safety while its copyright questions aligned to general reasoning.

Fewer questions, nearly the same picture

The question-level estimates make it possible to rank questions by how well they separate models. Ai2 did that across the same 16 benchmarks and reports that keeping only 10% of the questions generally preserved nearly the same picture of which models were stronger on the underlying capability. Keeping half of them often matched the full set even more closely.

The method can also predict how a model would do on a question it has never been scored on. It was right 79% of the time, against 70% for the simpler assumption that a model performs on each question about as well as it does on the benchmark overall. That difference is the practical payoff: fewer evaluation runs for the same read on a model.

The limits the authors state

Every model used to train and test BenchMIRT was released by March 2025, so nothing here says how the method behaves on current models. The dimensions it finds also depend on the benchmark set it is given, and a different mix could surface different capabilities. Ai2 adds that if the goal is simply to rank models on randomly held-out items, the plain benchmark average does slightly better.

The authors also name the obvious hazard. The same estimates that identify a benchmark's most informative safety questions could be used to delete them, leaving a weaker evaluation that an unsafe model would pass. They argue the transparency is worth the risk, and that comparable trimming is already possible with existing tools, but they state it as a real risk rather than a hypothetical one.