ZacharyZhang-NY
a4953335e7
The merged database names a source by content hash (`11089_4f2a...`) while the frozen sets name the same file by its corpus id, so every arm scored 0.000 until load_atoms joined atoms through sources.path. It now returns corpus ids and refuses an atom it cannot trace to a corpus file. Also fixes found by running the harness: - merge rebuilt atom_fts from the deduped atoms table; two attempts on one file left 288,266 fts rows for 165,973 atoms. - merge accepts the two honest reasons a corpus file has no product source (500 byte-identical LongMemEval sessions collapse under content hashing, and 12 photos that ONNX Runtime cannot decode) and reports them. - --arms used action=extend against a non-empty default, so naming two arms ran seven. - the dense cache is keyed by the atom set, so a scale-curve subset cannot be scored with full-corpus vectors. - write_markdown crashed on a run that covered only some modalities.
1.1 KiB
1.1 KiB
BASELINE: what the product retrieves today
- generated 2026-10-06T20:42:42+00:00 at commit
fc0698c242 - corpus 50278 sources, 10 queries (8 answerable)
- models: dense
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, rerankBAAI/bge-reranker-base
| arm | recall@1 | recall@5 | recall@10 | nDCG@10 | MRR | hit@10 (95% lower) | p50 s | p95 s |
|---|---|---|---|---|---|---|---|---|
| chance (random ranking) | 0.000 | 0.000 | 0.000 | - | - | - | - | - |
| bm25_only | 0.625 | 0.875 | 1.000 | 0.788 | 0.725 | 1.000 (0.676) | 0.881 | 1.207 |
recall@10 by modality
| arm | |
|---|---|
| bm25_only | 1.000 |
unanswerable queries (44.8% of the suite)
- bm25_only: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 100% of them
reading
- recall counts a source as found only when it is the source a third party marked correct.
- abstention is the only honest number for unanswerable queries: a retriever that always returns ten results never says it does not know.
- chance is the floor any of these arms must beat; a number near chance is not a capability.