Commit Graph
2 Commits
Author SHA1 Message Date
ZacharyZhang-NY 29163632af Score all five arms in one resumable pass
The per-arm runner recomputed BM25 and the dense matmul for every arm, which put the
full ablation at roughly 16 hours and left no way to survive a kill. JointRunner
computes each signal once per query, attributes latency to the components each arm
uses, and checkpoints every query to runs/baseline/rankings.jsonl.

- bench/systems.py: JointRunner + ARM_COMPONENTS signal attribution.
- bench/run.py: joint pass, rankings cache with a run fingerprint, per-query resume,
  latency summary, resident-memory progress lines.
- bench/tests/test_joint.py: joint rankings equal the per-arm rankings for all five
  arms; the sparse corpus is scored once even with three arms that need it; cache
  fingerprint mismatch and malformed rows fail fast.
- bench/reports/EXTERNAL_APPENDIX.md: what a real Mem0/Zep run would cost, and which
  published numbers are comparable at all.

Verification: pytest bench/tests -q -> 57 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
2026-10-07 01:45:55 -04:00
ZacharyZhang-NY a4953335e7 Join the product's source ids to the benchmark's before scoring
The merged database names a source by content hash (`11089_4f2a...`) while the
frozen sets name the same file by its corpus id, so every arm scored 0.000 until
load_atoms joined atoms through sources.path. It now returns corpus ids and
refuses an atom it cannot trace to a corpus file.

Also fixes found by running the harness:
- merge rebuilt atom_fts from the deduped atoms table; two attempts on one file
  left 288,266 fts rows for 165,973 atoms.
- merge accepts the two honest reasons a corpus file has no product source
  (500 byte-identical LongMemEval sessions collapse under content hashing, and
  12 photos that ONNX Runtime cannot decode) and reports them.
- --arms used action=extend against a non-empty default, so naming two arms ran
  seven.
- the dense cache is keyed by the atom set, so a scale-curve subset cannot be
  scored with full-corpus vectors.
- write_markdown crashed on a run that covered only some modalities.
2026-10-06 16:49:11 -04:00