bench-lint
A static linter for multiple-choice benchmark files. Ten checks flag duplicates, leaky options and broken fields, and every check is scored against expert relabels of MMLU.
- Year
- 2026
- Kind
- Evaluation · Python
- Status
- Shipped
Evaluation pipelines lint their code and rarely lint their data. bench-lint reads a benchmark file and reports duplicates, answer-key skew, options that leak the key, near-identical options, broken encodings and few-shot template drift. Each flag carries its evidence in words. It needs only the file: no model predictions, no network, only the standard library.
I scored every check against the MMLU-Redux 2.0 expert relabels, 5,668 items. Most checks are at chance. Across all checks, 714 items were flagged and 34 of them were faulted by the experts, a lift of 0.80 over the base rate. One pair stands out: near-identical options against items with more than one correct answer, lift 10.57, but its Holm-adjusted p is 0.052.
The honest limit is wrong keys. 94 of the 106 wrong answer keys go unflagged, because a wrong key in a well-formed item looks like a correct one without domain knowledge. The rules were also tuned on the same labels they are scored on, so the table is in-sample. It is a hygiene step before expert relabelling, not a replacement for it.
