ZacharyZhang-NY
71733d86d6
A review of the p2-1 record found seven places where the prose had drifted away from the reports it cites. None were product defects; all were record keeping, and record keeping is what a memory system is judged on. The quantization row mis-transcribed nine latencies, and the two worst made dimension truncation look twice as fast as the full scan when the report measures six percent. The extrapolation table scaled latency by atoms and bytes by vectors under one heading that named neither switch. The picture index stats existed only in prose, so the harness now reads the index shape off the artifact and times the encoder on a sample, which reproduces the hand-timed build within seven percent. The dense arms named a store that holds no vectors, so the report names the store the vectors came from and refuses to run where a dense leg would rank an empty index. The frozen baseline was migrated in place by the product CLI while the picture index was built, so its pinned mtime, digest and schema version are re-pinned and the step that moved it is named. The sqlite_vec import added two mypy errors; the fix is the missing stub declaration, not an inline ignore, and the count is back to the recorded baseline of twenty-four.
bench — the measurement harness
Everything here measures the product as it is. No arm may read a benchmark gold label.
run order
python -m bench.fetch_corpus # 19 GB of third-party material -> bench/corpus + ~/ely_eye_bench_data
python -m bench.build_sets # freeze the 6 query sets -> bench/sets/
python -m bench.validate_sets # schema + gold + scope + modality checks (b-2 gate)
python -m bench.fetch_models # dense + rerank models used by the ablation arms
python -m bench.ingest_baseline # the unmodified product ingesting the whole corpus (16 -> 8 workers)
python -m bench.run # five-arm ablation + latency/scale -> bench/reports/BASELINE.md
python -m pytest bench/tests -q # metric definitions, hand-computed
bench/runs/ and bench/corpus/ are gitignored; bench/corpus/manifest.jsonl is tracked so the
corpus is reproducible from the manifest alone.
the arms (b-3)
| arm | what it is |
|---|---|
bm25_only |
the product's sparse signal alone |
dense_only |
the product's dense signal alone (product flag is off today) |
graph_only |
the product's graph signal alone — atom.relations is never populated, so it is a constant |
no_rerank |
the product's exact 0.55 / 0.35 / 0.10 fusion |
fusion_rerank |
the same fusion plus a cross-encoder rerank, the upper bound of the current design |
python -m bench.run also drives the unmodified RetrievalService.search over the merged
database and reports its latency plus how far its top-10 differs from the fast BM25 path.
known environment artifact
pi-lens' pyright reports Import "bench.sets_common" could not be resolved (and the same for
bench.metrics, bench.systems, ely_eye.*, cv2, onnxruntime, pypdf.errors) even though
every one of those imports works. It is not a repo defect:
effective_configon any bench file returns0 config file(s)withresolvedCwdat the repo root, so neitherpyproject.toml [tool.pyright]norpyrightconfig.json(both of which carryextraPaths: [".", "backend/src"]) is being read by that runner.- Runtime proof, run from the repo root with
.venvactive:
.venv/bin/python -c "import bench.sets_common, bench.metrics, bench.systems, bench.run, bench.ingest_baseline, \
bench.build_mmlongbench, ely_eye.config, ely_eye.db, cv2, onnxruntime, pypdf.errors; print('runtime proof OK')"
Treat that command as the check of record for import health; treat pyright's type findings
(it has caught real defects here: an AtomRunner typo, missing strict=True on zip,
dict[str, object] over-narrowing) as separate and worth fixing.
The pi-lens mypy cascade also replays stale findings after a fix: it attributes another
file's lines to whichever file was edited last (e.g. bench/fetch_models.py:107: "object" has no attribute "encode", where line 107 of that file is a Path call and the quoted text is
from bench/systems.py). The check of record is a fresh-cache run, which has passed on every
current file:
rm -rf /tmp/mypycache && .venv/bin/python -m mypy --cache-dir /tmp/mypycache bench/ # Success: no issues found
.venv/bin/python -m ruff check bench/ # All checks passed
.venv/bin/python -m pytest bench/tests -q # all green
corpus facts that constrain the numbers
- 50,789 sources / 5,308 queries (2,928 answerable, 2,380 unanswerable), all third-party.
- Video-MME ships mp4s for 129 of 900 videos; the set uses those 129 (387 questions).
- Video-MME has no temporal gold, so its gold region is the whole video — recorded in the set manifest.
- ScreenSpot-v2 boxes are
[x, y, w, h], verified against 755 image headers (xywh in bounds 1271/1272). - CodeSearchNet docstrings appear verbatim inside the code, so the query text is stripped before ingest or BM25 gets a free exact match.
- The product keys a source by
{stem[:48]}_{sha256[:16]}, so identical bytes in two sets collapse to one product source;ingest_report.jsonreportsshared_source_ids.