Three defects made the external baseline numbers and labels wrong.
- bench/systems.py: _sentence_model rebuilt the model on every query, so dense arm
latency was mostly model loading and the latency-vs-size curve was flat because a
constant dominated it. Models are now cached per process.
- bench/counterpart.py: visible() and search() walked 404k Python objects per query.
Per-item holder/source/time columns are built once and the query path is numpy;
12.5 queries/s instead of 0.7.
- bench/metrics.py: the aggregate key was named wrong_holder_confidence_unanswerable
while it counts top hits that ARE from the asking holder. BASELINE.md read that
backwards. Renamed to top_hit_from_holder_unanswerable and the sentence rewritten.
- bench/run_external.py: the by-modality table read row['answerable'] but aggregate
emits n_answerable, so the writer died after scoring finished. Each store is now
scored twice, [holder scope] and [whole corpus], so the gain from the holder
boundary is separable from the gain from ranking; progress lines every 500 queries.
Verification: pytest bench/tests -q -> 61 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
Latency measured while another job hogs the cores is not comparable: the scale curve
moved from 1.315s to 0.809s at 50,000 sources between two runs of the same code.
- bench/systems.py: machine_facts() reports cpus plus the 1/5/15 minute load average.
- bench/run.py, bench/run_external.py: both reports carry it, and BASELINE.md labels
the latency table a ceiling when the machine was busy.
Verification: pytest bench/tests -q -> 60 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
The frozen baseline showed the unmodified RetrievalService agreeing with an equivalent
BM25 index on 0% of its top ten results. Reproduced by hand: its top ten atoms came
from one document's pages, so a source-level ranking cannot overlap it.
- bench/systems.py: live_product_rankings also counts distinct sources behind the
product's top-k atoms.
- bench/run.py: scale probes no longer filtered by gold (a source prefix keeps almost
no gold, so the 500/2,000/10,000 rows were silently skipped); scale_arm_names reads
only latency dicts, which is what crashed write_markdown on the note field; the
unanswerable share is computed instead of hard-coded.
- bench/tests/test_joint.py: 60 tests total, including the writer regression.
Verification: pytest bench/tests -q -> 60 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
The per-arm runner recomputed BM25 and the dense matmul for every arm, which put the
full ablation at roughly 16 hours and left no way to survive a kill. JointRunner
computes each signal once per query, attributes latency to the components each arm
uses, and checkpoints every query to runs/baseline/rankings.jsonl.
- bench/systems.py: JointRunner + ARM_COMPONENTS signal attribution.
- bench/run.py: joint pass, rankings cache with a run fingerprint, per-query resume,
latency summary, resident-memory progress lines.
- bench/tests/test_joint.py: joint rankings equal the per-arm rankings for all five
arms; the sparse corpus is scored once even with three arms that need it; cache
fingerprint mismatch and malformed rows fail fast.
- bench/reports/EXTERNAL_APPENDIX.md: what a real Mem0/Zep run would cost, and which
published numbers are comparable at all.
Verification: pytest bench/tests -q -> 57 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
ZacharyZhang-NY