ZacharyZhang-NY
29163632af
The per-arm runner recomputed BM25 and the dense matmul for every arm, which put the full ablation at roughly 16 hours and left no way to survive a kill. JointRunner computes each signal once per query, attributes latency to the components each arm uses, and checkpoints every query to runs/baseline/rankings.jsonl. - bench/systems.py: JointRunner + ARM_COMPONENTS signal attribution. - bench/run.py: joint pass, rankings cache with a run fingerprint, per-query resume, latency summary, resident-memory progress lines. - bench/tests/test_joint.py: joint rankings equal the per-arm rankings for all five arms; the sparse corpus is scored once even with three arms that need it; cache fingerprint mismatch and malformed rows fail fast. - bench/reports/EXTERNAL_APPENDIX.md: what a real Mem0/Zep run would cost, and which published numbers are comparable at all. Verification: pytest bench/tests -q -> 57 passed; ruff check bench/ -> all checks passed; mypy (fresh cache) -> no issues in 25 source files.
4.3 KiB
Appendix: what a real Mem0 run would cost, and why its numbers are not ours
bench/counterpart.py rebuilds the store-and-retrieve half of Mem0 and Zep inside this repo so the
comparison runs on one corpus, one machine, one scorer. The products themselves are not run here.
This appendix records what running them would take, and which of their published numbers can be
placed next to ours at all.
what the counterpart reproduces, and what it does not
| behaviour | Mem0 / Zep paper | bench/counterpart.py |
|---|---|---|
| store granularity | LLM-extracted facts, deduplicated against existing facts | deterministic 400-character sentence chunks, deduplicated by (holder, sha256(text)) |
| scope | user_id / agent_id / run_id |
holder from the frozen corpus |
| invalidation | Zep closes an edge when a newer fact contradicts an older one | supersede closes the older item when a newer one in the same (holder, subject) group overlaps it by less than 0.5 |
| retrieval | dense + sparse + graph signals, top_200 budget in the 2026 hosted stack |
dense cosine over the same 384-dimensional embedder the ely-eye dense arm uses, top_k=10 |
| scoring | LLM-as-a-judge on a generated answer | source-level recall@10 against frozen gold |
The extraction step is the part that cannot be made comparable: Mem0's add() calls an LLM per
ingest, so its store contents depend on the model, the prompt, and the sampling temperature. Running
it against this corpus would add an LLM to the ingest path and make the memory store itself
non-reproducible, while leaving the retrieval question untouched. That is why the reference point is
in-repo.
published numbers worth keeping
From the Mem0 paper (arXiv 2504.19413v1, LOCOMO, overall LLM-as-a-judge score, mean ± std over 10 runs):
| system | overall judge score | tokens retrieved per query | search p50 / p95 (s) | total p50 / p95 (s) |
|---|---|---|---|---|
| full-context | 72.90 ± 0.19 | 26,031 | — | 9.870 / 17.117 |
| Mem0 (graph variant) | 68.44 ± 0.17 | 3,616 | 0.476 / 0.657 | 1.091 / 2.590 |
| Mem0 | 66.88 ± 0.15 | 1,764 | 0.148 / 0.200 | 0.708 / 1.440 |
| Zep | 65.99 ± 0.16 | 3,911 | 0.513 / 0.778 | 1.292 / 2.926 |
| best RAG configuration | 60.97 ± 0.20 | k × chunk size | 0.267 / 0.624 | 0.766 / 1.829 |
| LangMem | 58.10 ± 0.21 | 127 | 17.99 / 59.82 | 18.53 / 60.40 |
| A-Mem | 48.38 ± 0.15 | 2,520 | 0.668 / 1.485 | 1.410 / 4.374 |
Two things in that table matter more than the scores:
- full-context wins the answer-quality column and loses the cost column. 72.90 at 26,031 tokens per query, p95 17.1 s. Every memory system trades quality for cost. Any claim that a memory layer "beats the context window" has to say which column it means.
- the store-construction cost is not in the table at all. The paper tabulates search and answer
latency, not
add(). It reports separately that Zep's construction runs multiple asynchronous LLM calls, so retrieval immediately after an add often fails and improves hours later. A benchmark that scores retrieval only, on a store built in advance, hides that entirely.
Sources: arXiv 2504.19413v1 (table 2 and sections 4.1-4.5), the mem0 repository README's 2026
algorithm table (LoCoMo 92.5, LongMemEval 94.4, ~7.0k tokens, p50 0.88-1.09 s, single-pass retrieval
at a top_200 budget, managed platform), and docs.mem0.ai/core-concepts/memory-evaluation.
why these numbers cannot sit in our ablation table
- Different metric. Their column is judge-scored answer correctness on 300 LOCOMO questions. Ours is recall@10 of the gold source over 2,928 answerable queries drawn from six third-party sets. A system can rank the right passage last and still answer correctly from a paraphrase.
- Different budget. They retrieve
top_200items; we scoretop_k=10. - Different hardware and model. Their run used a hosted production stack; ours is CPU-only with a 384-dimensional multilingual MiniLM, no CUDA in this WSL image.
- Different corpus age. LOCOMO has 10 dialogues; our conversation set is LongMemEval-cleaned with 7,188 session files and 854 holders, and it also carries five non-conversational modalities that no memory product is evaluated on here.
The honest use of this appendix is the ordering it implies — full-context > memory systems > naive RAG on answer quality, and the reverse on cost — plus the two cost columns the paper leaves out.