bm25-diff
Runs one corpus and one query set through four Python BM25 libraries, measures how far their rankings disagree, and traces each gap to its cause one setting at a time.
- Year
- 2026
- Kind
- Evaluation · Python
- Status
- Shipped
Four libraries in the Python retrieval stack say "BM25": rank_bm25, bm25s, SQLite's FTS5 and Tantivy. They do not compute the same function. bm25-diff runs one corpus and one query set through six configurations of them and measures how far the rankings disagree. Then it holds one thing fixed at a time: the tokeniser, then k1 and b, then the IDF form. Whatever is left is published as unexplained.
At their defaults, rank_bm25 and bm25s put a different document first on 39 of 64 English queries. Giving both the same tokens cuts that to 4. The tokeniser is the credited cause in 11 of the 12 English pairs it separates. The last gap between those two libraries is how they floor negative IDF: with the same IDF form and rank_bm25's epsilon set to 0, they agree on every query.
There are no relevance judgements, so it says whether the engines agree, not which one is right. The corpus is small: 1,416 passages from Conan Doyle and Ibn Khaldun, and 86 queries written for the repo. Six small k1 and b credits do not survive a correction for multiple intervals. Lucene is missing: it needs a JDK, which the project does not install.
