guardparity
Runs each pair of published files for three open guard classifiers over 4,723 public prompts and counts disagreements. One pair disagreed on 19.4% of prompts: two classifiers under one name.
- Year
- 2026
- Kind
- Security · Python
- Status
- Shipped
Open guard classifiers ship as several files: a pickle and an ONNX export, a checkpoint and an int8 version. Users pick whichever their runtime supports and assume it is the same classifier. guardparity runs both files of each pair over one pinned set of 4,723 public prompts and counts the prompts where their verdicts differ. A pair fails above 0.5%.
NemoGuard-JailbreakDetect's pickle and ONNX files disagreed on 917 prompts, 19.4% of the set, given the same embedding for every prompt. The ONNX file flagged 399 benign prompts the pickle passed. Their tree ensembles differ in size, 615 against 11,157 nodes per tree, so this is not rounding. Two ONNX exports matched their checkpoints on every verdict; an int8 version flipped 5 prompts.
A disagreement says the files are not the same classifier. It does not say which is right, and the cause is not diagnosed. The prompts are English-heavy. Loading a pickle runs code, so it loads only after its hash matches, in a subprocess locked down with Landlock and seccomp, and the tests attack that loader with malicious pickles.
