ZacharyZhang-NY
29163632af
The per-arm runner recomputed BM25 and the dense matmul for every arm, which put the full ablation at roughly 16 hours and left no way to survive a kill. JointRunner computes each signal once per query, attributes latency to the components each arm uses, and checkpoints every query to runs/baseline/rankings.jsonl. - bench/systems.py: JointRunner + ARM_COMPONENTS signal attribution. - bench/run.py: joint pass, rankings cache with a run fingerprint, per-query resume, latency summary, resident-memory progress lines. - bench/tests/test_joint.py: joint rankings equal the per-arm rankings for all five arms; the sparse corpus is scored once even with three arms that need it; cache fingerprint mismatch and malformed rows fail fast. - bench/reports/EXTERNAL_APPENDIX.md: what a real Mem0/Zep run would cost, and which published numbers are comparable at all. Verification: pytest bench/tests -q -> 57 passed; ruff check bench/ -> all checks passed; mypy (fresh cache) -> no issues in 25 source files.
66 lines
4.3 KiB
Markdown
66 lines
4.3 KiB
Markdown
# Appendix: what a real Mem0 run would cost, and why its numbers are not ours
|
||
|
||
`bench/counterpart.py` rebuilds the store-and-retrieve half of Mem0 and Zep inside this repo so the
|
||
comparison runs on one corpus, one machine, one scorer. The products themselves are not run here.
|
||
This appendix records what running them would take, and which of their published numbers can be
|
||
placed next to ours at all.
|
||
|
||
## what the counterpart reproduces, and what it does not
|
||
|
||
| behaviour | Mem0 / Zep paper | `bench/counterpart.py` |
|
||
|---|---|---|
|
||
| store granularity | LLM-extracted facts, deduplicated against existing facts | deterministic 400-character sentence chunks, deduplicated by `(holder, sha256(text))` |
|
||
| scope | `user_id` / `agent_id` / `run_id` | `holder` from the frozen corpus |
|
||
| invalidation | Zep closes an edge when a newer fact contradicts an older one | `supersede` closes the older item when a newer one in the same `(holder, subject)` group overlaps it by less than 0.5 |
|
||
| retrieval | dense + sparse + graph signals, `top_200` budget in the 2026 hosted stack | dense cosine over the same 384-dimensional embedder the ely-eye dense arm uses, `top_k=10` |
|
||
| scoring | LLM-as-a-judge on a generated answer | source-level recall@10 against frozen gold |
|
||
|
||
The extraction step is the part that cannot be made comparable: Mem0's `add()` calls an LLM per
|
||
ingest, so its store contents depend on the model, the prompt, and the sampling temperature. Running
|
||
it against this corpus would add an LLM to the ingest path and make the memory store itself
|
||
non-reproducible, while leaving the retrieval question untouched. That is why the reference point is
|
||
in-repo.
|
||
|
||
## published numbers worth keeping
|
||
|
||
From the Mem0 paper (arXiv 2504.19413v1, LOCOMO, overall LLM-as-a-judge score, mean ± std over 10 runs):
|
||
|
||
| system | overall judge score | tokens retrieved per query | search p50 / p95 (s) | total p50 / p95 (s) |
|
||
|---|---|---|---|---|
|
||
| full-context | 72.90 ± 0.19 | 26,031 | — | 9.870 / 17.117 |
|
||
| Mem0 (graph variant) | 68.44 ± 0.17 | 3,616 | 0.476 / 0.657 | 1.091 / 2.590 |
|
||
| Mem0 | 66.88 ± 0.15 | 1,764 | 0.148 / 0.200 | 0.708 / 1.440 |
|
||
| Zep | 65.99 ± 0.16 | 3,911 | 0.513 / 0.778 | 1.292 / 2.926 |
|
||
| best RAG configuration | 60.97 ± 0.20 | k × chunk size | 0.267 / 0.624 | 0.766 / 1.829 |
|
||
| LangMem | 58.10 ± 0.21 | 127 | 17.99 / 59.82 | 18.53 / 60.40 |
|
||
| A-Mem | 48.38 ± 0.15 | 2,520 | 0.668 / 1.485 | 1.410 / 4.374 |
|
||
|
||
Two things in that table matter more than the scores:
|
||
|
||
- **full-context wins the answer-quality column and loses the cost column.** 72.90 at 26,031 tokens
|
||
per query, p95 17.1 s. Every memory system trades quality for cost. Any claim that a memory layer
|
||
"beats the context window" has to say which column it means.
|
||
- **the store-construction cost is not in the table at all.** The paper tabulates search and answer
|
||
latency, not `add()`. It reports separately that Zep's construction runs multiple asynchronous LLM
|
||
calls, so retrieval immediately after an add often fails and improves hours later. A benchmark that
|
||
scores retrieval only, on a store built in advance, hides that entirely.
|
||
|
||
Sources: arXiv 2504.19413v1 (table 2 and sections 4.1-4.5), the mem0 repository README's 2026
|
||
algorithm table (LoCoMo 92.5, LongMemEval 94.4, ~7.0k tokens, p50 0.88-1.09 s, single-pass retrieval
|
||
at a `top_200` budget, managed platform), and `docs.mem0.ai/core-concepts/memory-evaluation`.
|
||
|
||
## why these numbers cannot sit in our ablation table
|
||
|
||
1. Different metric. Their column is judge-scored answer correctness on 300 LOCOMO questions. Ours is
|
||
recall@10 of the gold **source** over 2,928 answerable queries drawn from six third-party sets.
|
||
A system can rank the right passage last and still answer correctly from a paraphrase.
|
||
2. Different budget. They retrieve `top_200` items; we score `top_k=10`.
|
||
3. Different hardware and model. Their run used a hosted production stack; ours is CPU-only with a
|
||
384-dimensional multilingual MiniLM, no CUDA in this WSL image.
|
||
4. Different corpus age. LOCOMO has 10 dialogues; our conversation set is LongMemEval-cleaned with
|
||
7,188 session files and 854 holders, and it also carries five non-conversational modalities that
|
||
no memory product is evaluated on here.
|
||
|
||
The honest use of this appendix is the ordering it implies — full-context > memory systems > naive
|
||
RAG on answer quality, and the reverse on cost — plus the two cost columns the paper leaves out.
|