Score the product path against corpus ids, not storage ids
The frozen baseline claimed the product's top-10 shared 0.0% with the fast BM25 path. That was a harness bug: gold names a corpus source (mmlongbench:PH_2016.06.08_Economy-Final.pdf) while the store names a content hash (PH_2016_06_08_Economy_Final_10e57152f44d1a2b), so the two id spaces never met and the right document, ranked first, scored as a miss. - bench/systems.py: `_corpus_ids` joins store sources through the `source_map()` the counterpart already uses, and refuses a store whose sources are not corpus files instead of silently scoring zero. - bench/systems.py: `live_product_rankings(..., as_holders=...)` asks the product as the query's own holder, which is how the counterpart is scored. - bench/run.py: the live section now reports recall@10 / MRR / hit@10 with a 95% lower bound, and probes 300 queries instead of 40 — the SQL path answers in well under a second, so the old cap was a cost estimate that expired. - bench/run.py: `--name CURRENT` writes a post-P0 report beside the frozen BASELINE without touching it, stamped with the commit and the store schema version the pass started on (the live pass migrates the store mid-run, so a plain stamp would be a lie). - bench/stamp_holders.py: the pre-P0 store holds 165,973 memories nobody owns. The corpus index states the owner, so holders, scenarios and event times are stamped in place — 165,973 atoms across 50,278 sources, 0 left unowned. Numbers, same machine, load 1.0-1.7: - CURRENT (post-P0, schema v5): no_rerank recall@10 0.565 p50 0.643s; fusion+rerank 0.453 p50 7.000s; graph 0.001. - Product as its holder: recall@10 0.870, MRR 0.740, n=173. - Counterpart: mem0_flat holder scope 0.866, whole corpus 0.412; the holder boundary is worth +0.454 with an identical ranker and identical items. Verification: pytest backend/tests bench/tests -q -> 173 passed; ruff and mypy (fresh cache) clean on every touched file.
This commit is contained in:
1 parent
4ab36cd5eb
commit
6de7439db6
8 files changed
+975
-68
No files matched your search
@@ -1,19 +1,26 @@
|
||||
{
|
||||
"chance": {
|
||||
"median_holder_sources": 43.0,
|
||||
"note": "expected recall@10 of a random ranking inside the candidate set each row is allowed to see",
|
||||
"recall_at_10": 0.00019689302801787788,
|
||||
"recall_at_10_holder_scope": 0.3685528769188164
|
||||
},
|
||||
"meta": {
|
||||
"answerable": 2928,
|
||||
"extraction": "lexical (see bench/counterpart.py); LLM extraction deferred with the generator",
|
||||
"generated_at": "2026-10-07T17:26:07+00:00",
|
||||
"git_commit": "c4e2982506faaf350a4b06738923899fc44bdedd",
|
||||
"generated_at": "2026-10-07T19:16:08+00:00",
|
||||
"git_commit": "3d792417cfd0554cbe2bae0cfc098bb5bb07e28c",
|
||||
"limit_per_modality": null,
|
||||
"machine": {
|
||||
"cpus": 32,
|
||||
"load_average_1_5_15": [
|
||||
1.47,
|
||||
2.81,
|
||||
10.33
|
||||
2.85,
|
||||
10.0,
|
||||
14.84
|
||||
],
|
||||
"release": "6.18.40.1-microsoft-standard-WSL2",
|
||||
"system": "Linux"
|
||||
"system": "Linux",
|
||||
"threads_per_query": 4
|
||||
},
|
||||
"queries": 5308,
|
||||
"queries_by_modality": {
|
||||
@@ -118,10 +125,10 @@
|
||||
"closed_items": 0,
|
||||
"items": 403859,
|
||||
"latency_seconds": {
|
||||
"max": 7.547149860001809,
|
||||
"mean": 0.09187759584445908,
|
||||
"p50": 0.08432659949903609,
|
||||
"p95": 0.13331470334669573,
|
||||
"max": 5.168096734996652,
|
||||
"mean": 0.08254052409894046,
|
||||
"p50": 0.08343072250136174,
|
||||
"p95": 0.10547906984866127,
|
||||
"queries": 5308
|
||||
},
|
||||
"overall": {
|
||||
@@ -230,10 +237,10 @@
|
||||
"closed_items": 0,
|
||||
"items": 403859,
|
||||
"latency_seconds": {
|
||||
"max": 2.318251260003308,
|
||||
"mean": 0.10140667538014644,
|
||||
"p50": 0.09026374550012406,
|
||||
"p95": 0.17443375029943126,
|
||||
"max": 0.1391175699973246,
|
||||
"mean": 0.08744079960348271,
|
||||
"p50": 0.0859174314973643,
|
||||
"p95": 0.11227948825326166,
|
||||
"queries": 5308
|
||||
},
|
||||
"overall": {
|
||||
@@ -342,10 +349,10 @@
|
||||
"closed_items": 27072,
|
||||
"items": 403859,
|
||||
"latency_seconds": {
|
||||
"max": 18.72793256899604,
|
||||
"mean": 0.09218696485981677,
|
||||
"p50": 0.08754261549984221,
|
||||
"p95": 0.11125843500121846,
|
||||
"max": 1.9491578790039057,
|
||||
"mean": 0.08603176019042821,
|
||||
"p50": 0.0841971809968527,
|
||||
"p95": 0.11250105440230979,
|
||||
"queries": 5308
|
||||
},
|
||||
"overall": {
|
||||
@@ -454,10 +461,10 @@
|
||||
"closed_items": 27072,
|
||||
"items": 403859,
|
||||
"latency_seconds": {
|
||||
"max": 0.2064202700057649,
|
||||
"mean": 0.08776056289499023,
|
||||
"p50": 0.08638653549860464,
|
||||
"p95": 0.10662773229960291,
|
||||
"max": 0.17060414699517423,
|
||||
"mean": 0.0951456186994907,
|
||||
"p50": 0.09420693750143982,
|
||||
"p95": 0.12111507409863406,
|
||||
"queries": 5308
|
||||
},
|
||||
"overall": {
|
||||
|
||||
Reference in new issue
Block a user