Every number in CHECKLIST p2-1 and p0-3.7 came out of a scratch script: the
kind of artifact a reader cannot re-run, and the kind that quietly drifts
away from the code it describes. These reports are generated by committed
harnesses and each one stamps the commit that carries the code it measures.
DENSE.md 165,973 atoms / 153,857 vectors: build 1421.0s, search p50 75.6ms
p95 93.0ms, 0.998 agreement with a brute-force scan, one write into
a full index p50 518ms, 1,729 bytes per stored vector.
QUANT.md twelve encodings measured on the same vectors: int8 keeps 0.977 of
the ranking for a quarter of the bytes and no speed, bit is 14x
faster and keeps 0.728, so bit is a coarse filter and nothing more.
TERMS.md the stop-word bound swept, not guessed: 0.05 is the measured peak
(recall@10 0.803 at p50 69.5ms) and also the fastest of the rows
that keep recall.
SCALE.md the product's own search at 165,973 atoms: p50 71.7ms against the
frozen baseline's 20.16s.
LEGS.md what each retrieval leg is worth, per modality, with the page size
the product actually asks for in the table.
The acceptance bar the operator set is recall@10 >= 0.90 at n>=300. It is
not met. The best measured row is 0.858 (dense leg on, page widened to 30,
p50 414ms); the shipped configuration gives 0.848 at p50 72.5ms; the
in-repo counterpart at holder scope gives 0.866. The shortfall is
concentrated in picture golds (video 0.583-0.708, ui_screenshot 0.500-0.583,
image 0.750) while the word-shaped ones already pass (code 0.953, pdf 0.919).
Latency passes by three hundred fold; recall does not, and the gap is four
points of cross-modal retrieval, not index scale.
The frozen baseline claimed the product's top-10 shared 0.0% with the fast
BM25 path. That was a harness bug: gold names a corpus source
(mmlongbench:PH_2016.06.08_Economy-Final.pdf) while the store names a content
hash (PH_2016_06_08_Economy_Final_10e57152f44d1a2b), so the two id spaces never
met and the right document, ranked first, scored as a miss.
- bench/systems.py: `_corpus_ids` joins store sources through the
`source_map()` the counterpart already uses, and refuses a store whose
sources are not corpus files instead of silently scoring zero.
- bench/systems.py: `live_product_rankings(..., as_holders=...)` asks the
product as the query's own holder, which is how the counterpart is scored.
- bench/run.py: the live section now reports recall@10 / MRR / hit@10 with a
95% lower bound, and probes 300 queries instead of 40 — the SQL path answers
in well under a second, so the old cap was a cost estimate that expired.
- bench/run.py: `--name CURRENT` writes a post-P0 report beside the frozen
BASELINE without touching it, stamped with the commit and the store schema
version the pass started on (the live pass migrates the store mid-run, so a
plain stamp would be a lie).
- bench/stamp_holders.py: the pre-P0 store holds 165,973 memories nobody owns.
The corpus index states the owner, so holders, scenarios and event times are
stamped in place — 165,973 atoms across 50,278 sources, 0 left unowned.
Numbers, same machine, load 1.0-1.7:
- CURRENT (post-P0, schema v5): no_rerank recall@10 0.565 p50 0.643s;
fusion+rerank 0.453 p50 7.000s; graph 0.001.
- Product as its holder: recall@10 0.870, MRR 0.740, n=173.
- Counterpart: mem0_flat holder scope 0.866, whole corpus 0.412; the holder
boundary is worth +0.454 with an identical ranker and identical items.
Verification: pytest backend/tests bench/tests -q -> 173 passed; ruff and mypy
(fresh cache) clean on every touched file.
ZacharyZhang-NY