Score the product path against corpus ids, not storage ids

The frozen baseline claimed the product's top-10 shared 0.0% with the fast
BM25 path. That was a harness bug: gold names a corpus source
(mmlongbench:PH_2016.06.08_Economy-Final.pdf) while the store names a content
hash (PH_2016_06_08_Economy_Final_10e57152f44d1a2b), so the two id spaces never
met and the right document, ranked first, scored as a miss.

- bench/systems.py: `_corpus_ids` joins store sources through the
  `source_map()` the counterpart already uses, and refuses a store whose
  sources are not corpus files instead of silently scoring zero.
- bench/systems.py: `live_product_rankings(..., as_holders=...)` asks the
  product as the query's own holder, which is how the counterpart is scored.
- bench/run.py: the live section now reports recall@10 / MRR / hit@10 with a
  95% lower bound, and probes 300 queries instead of 40 — the SQL path answers
  in well under a second, so the old cap was a cost estimate that expired.
- bench/run.py: `--name CURRENT` writes a post-P0 report beside the frozen
  BASELINE without touching it, stamped with the commit and the store schema
  version the pass started on (the live pass migrates the store mid-run, so a
  plain stamp would be a lie).
- bench/stamp_holders.py: the pre-P0 store holds 165,973 memories nobody owns.
  The corpus index states the owner, so holders, scenarios and event times are
  stamped in place — 165,973 atoms across 50,278 sources, 0 left unowned.

Numbers, same machine, load 1.0-1.7:
- CURRENT (post-P0, schema v5): no_rerank recall@10 0.565 p50 0.643s;
  fusion+rerank 0.453 p50 7.000s; graph 0.001.
- Product as its holder: recall@10 0.870, MRR 0.740, n=173.
- Counterpart: mem0_flat holder scope 0.866, whole corpus 0.412; the holder
  boundary is worth +0.454 with an identical ranker and identical items.

Verification: pytest backend/tests bench/tests -q -> 173 passed; ruff and mypy
(fresh cache) clean on every touched file.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 19:32:32 -04:00
1 parent 4ab36cd5eb
commit 6de7439db6
8 files changed
+975 -68

No files matched your search

+6
View File
@@ -181,7 +181,12 @@ def _report() -> dict:
"live_product": {
"queries": 2,
"top_k": 10,
"recall_at_10": 0.87,
"mrr": 0.74,
"hit_at_10": 0.87,
"hit_at_10_wilson_lower": 0.82,
"p50": 1.0,
"p95": 1.0,
"mean": 1.0,
"max": 1.0,
"top10_overlap_with_bm25_only": 0.0,
@@ -202,6 +207,7 @@ def test_the_report_writer_survives_a_scale_note(tmp_path: Path) -> None:
assert "_latency only_" in text
assert "load average 1.0 on 32 cores" in text
assert "## unanswerable queries (50.0% of the suite)" in text
assert "recall@10 0.870" in text
def test_scale_arm_names_reads_only_latency_dicts() -> None: