Score the product path against corpus ids, not storage ids

The frozen baseline claimed the product's top-10 shared 0.0% with the fast
BM25 path. That was a harness bug: gold names a corpus source
(mmlongbench:PH_2016.06.08_Economy-Final.pdf) while the store names a content
hash (PH_2016_06_08_Economy_Final_10e57152f44d1a2b), so the two id spaces never
met and the right document, ranked first, scored as a miss.

- bench/systems.py: `_corpus_ids` joins store sources through the
  `source_map()` the counterpart already uses, and refuses a store whose
  sources are not corpus files instead of silently scoring zero.
- bench/systems.py: `live_product_rankings(..., as_holders=...)` asks the
  product as the query's own holder, which is how the counterpart is scored.
- bench/run.py: the live section now reports recall@10 / MRR / hit@10 with a
  95% lower bound, and probes 300 queries instead of 40 — the SQL path answers
  in well under a second, so the old cap was a cost estimate that expired.
- bench/run.py: `--name CURRENT` writes a post-P0 report beside the frozen
  BASELINE without touching it, stamped with the commit and the store schema
  version the pass started on (the live pass migrates the store mid-run, so a
  plain stamp would be a lie).
- bench/stamp_holders.py: the pre-P0 store holds 165,973 memories nobody owns.
  The corpus index states the owner, so holders, scenarios and event times are
  stamped in place — 165,973 atoms across 50,278 sources, 0 left unowned.

Numbers, same machine, load 1.0-1.7:
- CURRENT (post-P0, schema v5): no_rerank recall@10 0.565 p50 0.643s;
  fusion+rerank 0.453 p50 7.000s; graph 0.001.
- Product as its holder: recall@10 0.870, MRR 0.740, n=173.
- Counterpart: mem0_flat holder scope 0.866, whole corpus 0.412; the holder
  boundary is worth +0.454 with an identical ranker and identical items.

Verification: pytest backend/tests bench/tests -q -> 173 passed; ruff and mypy
(fresh cache) clean on every touched file.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 19:32:32 -04:00
1 parent 4ab36cd5eb
commit 6de7439db6
8 files changed
+975 -68

No files matched your search

+29 -22
View File
@@ -1,19 +1,26 @@
{
"chance": {
"median_holder_sources": 43.0,
"note": "expected recall@10 of a random ranking inside the candidate set each row is allowed to see",
"recall_at_10": 0.00019689302801787788,
"recall_at_10_holder_scope": 0.3685528769188164
},
"meta": {
"answerable": 2928,
"extraction": "lexical (see bench/counterpart.py); LLM extraction deferred with the generator",
"generated_at": "2026-10-07T17:26:07+00:00",
"git_commit": "c4e2982506faaf350a4b06738923899fc44bdedd",
"generated_at": "2026-10-07T19:16:08+00:00",
"git_commit": "3d792417cfd0554cbe2bae0cfc098bb5bb07e28c",
"limit_per_modality": null,
"machine": {
"cpus": 32,
"load_average_1_5_15": [
1.47,
2.81,
10.33
2.85,
10.0,
14.84
],
"release": "6.18.40.1-microsoft-standard-WSL2",
"system": "Linux"
"system": "Linux",
"threads_per_query": 4
},
"queries": 5308,
"queries_by_modality": {
@@ -118,10 +125,10 @@
"closed_items": 0,
"items": 403859,
"latency_seconds": {
"max": 7.547149860001809,
"mean": 0.09187759584445908,
"p50": 0.08432659949903609,
"p95": 0.13331470334669573,
"max": 5.168096734996652,
"mean": 0.08254052409894046,
"p50": 0.08343072250136174,
"p95": 0.10547906984866127,
"queries": 5308
},
"overall": {
@@ -230,10 +237,10 @@
"closed_items": 0,
"items": 403859,
"latency_seconds": {
"max": 2.318251260003308,
"mean": 0.10140667538014644,
"p50": 0.09026374550012406,
"p95": 0.17443375029943126,
"max": 0.1391175699973246,
"mean": 0.08744079960348271,
"p50": 0.0859174314973643,
"p95": 0.11227948825326166,
"queries": 5308
},
"overall": {
@@ -342,10 +349,10 @@
"closed_items": 27072,
"items": 403859,
"latency_seconds": {
"max": 18.72793256899604,
"mean": 0.09218696485981677,
"p50": 0.08754261549984221,
"p95": 0.11125843500121846,
"max": 1.9491578790039057,
"mean": 0.08603176019042821,
"p50": 0.0841971809968527,
"p95": 0.11250105440230979,
"queries": 5308
},
"overall": {
@@ -454,10 +461,10 @@
"closed_items": 27072,
"items": 403859,
"latency_seconds": {
"max": 0.2064202700057649,
"mean": 0.08776056289499023,
"p50": 0.08638653549860464,
"p95": 0.10662773229960291,
"max": 0.17060414699517423,
"mean": 0.0951456186994907,
"p50": 0.09420693750143982,
"p95": 0.12111507409863406,
"queries": 5308
},
"overall": {