rag-lab
A search system measured one design choice at a time on 94 questions over a public standard. Most ideas did not help. Every number regenerates offline, and CI fails if one moves.
- Year
- 2026
- Kind
- Agents & RAG · Python
- Status
- Shipped
I built a search system over NIST's digital identity guidelines and measured each choice on its own: chunking, keyword settings, fusion, reranking and query rewriting. The rule for picking a winner was written down before each sweep ran. Every results page regenerates from the repo, offline, and CI fails if a number moves.
Most ideas lost. Keyword search beat embedding search on ordering. Semantic chunking did not beat a fixed 128-word window. None of nine re-scoring methods helped. A failure taxonomy found that 7 of 23 misses were label errors, not retrieval errors. The questions were produced with a language model and checked only for consistency, so recall numbers are lower bounds.
The second half is a sandbox for untrusted Python. Each of its ten escape tests runs the attack twice: with that one boundary removed, where it must succeed, and under the strict policy, where it must fail. The no-network claim was false until a working payload showed io_uring slipping past the syscall filter. That payload is now a test.
