ZacharyZhang-NY
6de7439db6
The frozen baseline claimed the product's top-10 shared 0.0% with the fast BM25 path. That was a harness bug: gold names a corpus source (mmlongbench:PH_2016.06.08_Economy-Final.pdf) while the store names a content hash (PH_2016_06_08_Economy_Final_10e57152f44d1a2b), so the two id spaces never met and the right document, ranked first, scored as a miss. - bench/systems.py: `_corpus_ids` joins store sources through the `source_map()` the counterpart already uses, and refuses a store whose sources are not corpus files instead of silently scoring zero. - bench/systems.py: `live_product_rankings(..., as_holders=...)` asks the product as the query's own holder, which is how the counterpart is scored. - bench/run.py: the live section now reports recall@10 / MRR / hit@10 with a 95% lower bound, and probes 300 queries instead of 40 — the SQL path answers in well under a second, so the old cap was a cost estimate that expired. - bench/run.py: `--name CURRENT` writes a post-P0 report beside the frozen BASELINE without touching it, stamped with the commit and the store schema version the pass started on (the live pass migrates the store mid-run, so a plain stamp would be a lie). - bench/stamp_holders.py: the pre-P0 store holds 165,973 memories nobody owns. The corpus index states the owner, so holders, scenarios and event times are stamped in place — 165,973 atoms across 50,278 sources, 0 left unowned. Numbers, same machine, load 1.0-1.7: - CURRENT (post-P0, schema v5): no_rerank recall@10 0.565 p50 0.643s; fusion+rerank 0.453 p50 7.000s; graph 0.001. - Product as its holder: recall@10 0.870, MRR 0.740, n=173. - Counterpart: mem0_flat holder scope 0.866, whole corpus 0.412; the holder boundary is worth +0.454 with an identical ranker and identical items. Verification: pytest backend/tests bench/tests -q -> 173 passed; ruff and mypy (fresh cache) clean on every touched file.
66 lines
3.3 KiB
Markdown
66 lines
3.3 KiB
Markdown
# BASELINE_EXTERNAL: memory counterparts on the same corpus
|
|
|
|
- generated 2026-10-07T19:16:08+00:00 at commit 3d792417cfd0
|
|
- 5308 queries (2928 answerable) against 50789 sources
|
|
- machine: 32 cores, load average 2.9 when the report was written
|
|
- extraction is lexical for both stores; Mem0's real add() and Zep's graph builder call a generator model, which this round does not have
|
|
- chance: random guessing scores recall@10 0.00020 over the whole corpus and 0.3686 inside one holder's memory (median 43 sources per holder); read each row against its own floor
|
|
|
|
| store | items | closed | recall@10 | hit@10 | hit@10 Wilson 95% lower | MRR | abstain (answerable) | abstain (unanswerable) | p50 s | p95 s |
|
|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
|
| mem0_flat [holder scope] | 403859 | 0 | 0.866 | 0.867 | 0.855 | 0.629 | 0.000 | 0.000 | 0.083 | 0.105 |
|
|
| mem0_flat [whole corpus] | 403859 | 0 | 0.412 | 0.416 | 0.398 | 0.283 | 0.000 | 0.000 | 0.086 | 0.112 |
|
|
| zep_temporal [holder scope] | 403859 | 27072 | 0.866 | 0.867 | 0.854 | 0.627 | 0.000 | 0.058 | 0.084 | 0.113 |
|
|
| zep_temporal [whole corpus] | 403859 | 27072 | 0.429 | 0.433 | 0.415 | 0.297 | 0.000 | 0.000 | 0.094 | 0.121 |
|
|
|
|
n per modality: {"code": 2474, "conversation": 300, "image": 480, "pdf": 984, "ui_screenshot": 640, "video": 430}
|
|
|
|
|
|
`[holder scope]` answers only from the asking holder's memory; `[whole corpus]` searches every
|
|
holder's memory, which is what the product's arms are shown. The gap between the two rows is the
|
|
value of the holder boundary, not of a better ranker.
|
|
|
|
## mem0_flat [holder scope] by modality
|
|
|
|
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| code | 1237 | 0.952 | 0.952 | 0.939 | 0.695 |
|
|
| conversation | 150 | 0.993 | 1.000 | 0.975 | 0.927 |
|
|
| image | 240 | 0.646 | 0.654 | 0.592 | 0.505 |
|
|
| pdf | 766 | 0.916 | 0.916 | 0.895 | 0.692 |
|
|
| ui_screenshot | 320 | 0.622 | 0.622 | 0.568 | 0.330 |
|
|
| video | 215 | 0.716 | 0.716 | 0.653 | 0.396 |
|
|
|
|
## mem0_flat [whole corpus] by modality
|
|
|
|
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| code | 1237 | 0.349 | 0.349 | 0.323 | 0.216 |
|
|
| conversation | 150 | 0.500 | 0.573 | 0.493 | 0.373 |
|
|
| image | 240 | 0.525 | 0.533 | 0.470 | 0.362 |
|
|
| pdf | 766 | 0.683 | 0.683 | 0.649 | 0.515 |
|
|
| ui_screenshot | 320 | 0.075 | 0.075 | 0.051 | 0.033 |
|
|
| video | 215 | 0.116 | 0.116 | 0.080 | 0.069 |
|
|
|
|
## zep_temporal [holder scope] by modality
|
|
|
|
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| code | 1237 | 0.952 | 0.952 | 0.939 | 0.695 |
|
|
| conversation | 150 | 0.978 | 0.987 | 0.953 | 0.905 |
|
|
| image | 240 | 0.646 | 0.654 | 0.592 | 0.505 |
|
|
| pdf | 766 | 0.916 | 0.916 | 0.895 | 0.692 |
|
|
| ui_screenshot | 320 | 0.622 | 0.622 | 0.568 | 0.330 |
|
|
| video | 215 | 0.716 | 0.716 | 0.653 | 0.396 |
|
|
|
|
## zep_temporal [whole corpus] by modality
|
|
|
|
| modality | n answerable | recall@10 | hit@10 | Wilson lower | MRR |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| code | 1237 | 0.350 | 0.350 | 0.324 | 0.216 |
|
|
| conversation | 150 | 0.591 | 0.667 | 0.588 | 0.429 |
|
|
| image | 240 | 0.562 | 0.571 | 0.508 | 0.426 |
|
|
| pdf | 766 | 0.685 | 0.685 | 0.652 | 0.519 |
|
|
| ui_screenshot | 320 | 0.119 | 0.119 | 0.088 | 0.057 |
|
|
| video | 215 | 0.163 | 0.163 | 0.119 | 0.087 |
|