Commit Graph
2 Commits
Author SHA1 Message Date
ZacharyZhang-NY 61f6506104 Explain the product path's zero overlap and fix the report writer
The frozen baseline showed the unmodified RetrievalService agreeing with an equivalent
BM25 index on 0% of its top ten results. Reproduced by hand: its top ten atoms came
from one document's pages, so a source-level ranking cannot overlap it.

- bench/systems.py: live_product_rankings also counts distinct sources behind the
  product's top-k atoms.
- bench/run.py: scale probes no longer filtered by gold (a source prefix keeps almost
  no gold, so the 500/2,000/10,000 rows were silently skipped); scale_arm_names reads
  only latency dicts, which is what crashed write_markdown on the note field; the
  unanswerable share is computed instead of hard-coded.
- bench/tests/test_joint.py: 60 tests total, including the writer regression.

Verification: pytest bench/tests -q -> 60 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
2026-10-07 11:33:44 -04:00
ZacharyZhang-NY 29163632af Score all five arms in one resumable pass
The per-arm runner recomputed BM25 and the dense matmul for every arm, which put the
full ablation at roughly 16 hours and left no way to survive a kill. JointRunner
computes each signal once per query, attributes latency to the components each arm
uses, and checkpoints every query to runs/baseline/rankings.jsonl.

- bench/systems.py: JointRunner + ARM_COMPONENTS signal attribution.
- bench/run.py: joint pass, rankings cache with a run fingerprint, per-query resume,
  latency summary, resident-memory progress lines.
- bench/tests/test_joint.py: joint rankings equal the per-arm rankings for all five
  arms; the sparse corpus is scored once even with three arms that need it; cache
  fingerprint mismatch and malformed rows fail fast.
- bench/reports/EXTERNAL_APPENDIX.md: what a real Mem0/Zep run would cost, and which
  published numbers are comparable at all.

Verification: pytest bench/tests -q -> 57 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
2026-10-07 01:45:55 -04:00