Score the product path against corpus ids, not storage ids
The frozen baseline claimed the product's top-10 shared 0.0% with the fast BM25 path. That was a harness bug: gold names a corpus source (mmlongbench:PH_2016.06.08_Economy-Final.pdf) while the store names a content hash (PH_2016_06_08_Economy_Final_10e57152f44d1a2b), so the two id spaces never met and the right document, ranked first, scored as a miss. - bench/systems.py: `_corpus_ids` joins store sources through the `source_map()` the counterpart already uses, and refuses a store whose sources are not corpus files instead of silently scoring zero. - bench/systems.py: `live_product_rankings(..., as_holders=...)` asks the product as the query's own holder, which is how the counterpart is scored. - bench/run.py: the live section now reports recall@10 / MRR / hit@10 with a 95% lower bound, and probes 300 queries instead of 40 — the SQL path answers in well under a second, so the old cap was a cost estimate that expired. - bench/run.py: `--name CURRENT` writes a post-P0 report beside the frozen BASELINE without touching it, stamped with the commit and the store schema version the pass started on (the live pass migrates the store mid-run, so a plain stamp would be a lie). - bench/stamp_holders.py: the pre-P0 store holds 165,973 memories nobody owns. The corpus index states the owner, so holders, scenarios and event times are stamped in place — 165,973 atoms across 50,278 sources, 0 left unowned. Numbers, same machine, load 1.0-1.7: - CURRENT (post-P0, schema v5): no_rerank recall@10 0.565 p50 0.643s; fusion+rerank 0.453 p50 7.000s; graph 0.001. - Product as its holder: recall@10 0.870, MRR 0.740, n=173. - Counterpart: mem0_flat holder scope 0.866, whole corpus 0.412; the holder boundary is worth +0.454 with an identical ranker and identical items. Verification: pytest backend/tests bench/tests -q -> 173 passed; ruff and mypy (fresh cache) clean on every touched file.
This commit is contained in:
1 parent
4ab36cd5eb
commit
6de7439db6
8 files changed
+975
-68
No files matched your search
@@ -181,7 +181,12 @@ def _report() -> dict:
|
||||
"live_product": {
|
||||
"queries": 2,
|
||||
"top_k": 10,
|
||||
"recall_at_10": 0.87,
|
||||
"mrr": 0.74,
|
||||
"hit_at_10": 0.87,
|
||||
"hit_at_10_wilson_lower": 0.82,
|
||||
"p50": 1.0,
|
||||
"p95": 1.0,
|
||||
"mean": 1.0,
|
||||
"max": 1.0,
|
||||
"top10_overlap_with_bm25_only": 0.0,
|
||||
@@ -202,6 +207,7 @@ def test_the_report_writer_survives_a_scale_note(tmp_path: Path) -> None:
|
||||
assert "_latency only_" in text
|
||||
assert "load average 1.0 on 32 cores" in text
|
||||
assert "## unanswerable queries (50.0% of the suite)" in text
|
||||
assert "recall@10 0.870" in text
|
||||
|
||||
|
||||
def test_scale_arm_names_reads_only_latency_dicts() -> None:
|
||||
|
||||
Reference in new issue
Block a user