judgecheck

Many eval pipelines put a language model in the scoring loop, and few report how reliable that judge is. judgecheck measures it from ratings you already have: agreement with human labels and with other judges, calibration, run-to-run stability, and five bias probes such as answer position and verbosity. One command turns a ratings file into a report.

One number is not enough. On 400 simulated allow-or-block cases rated by a judge and a human, raw agreement is 96%, and three chance-corrected coefficients on the same data run from 0.60 to 0.96. In another, a judge agrees with one human at an alpha of 0.74 while the two humans agree with each other at 0.61, so the labels are too noisy to certify the judge.

Every number in the repo comes from data generated inside the scripts. No judge was called, and there is no live-judge mode. Reliability is not validity: two judges can agree perfectly and both be wrong. A probe that stays quiet has only shown it did not detect a bias at that sample size, so each null result is printed with the smallest effect it could have seen.