Join the product's source ids to the benchmark's before scoring

The merged database names a source by content hash (`11089_4f2a...`) while the
frozen sets name the same file by its corpus id, so every arm scored 0.000 until
load_atoms joined atoms through sources.path. It now returns corpus ids and
refuses an atom it cannot trace to a corpus file.

Also fixes found by running the harness:
- merge rebuilt atom_fts from the deduped atoms table; two attempts on one file
  left 288,266 fts rows for 165,973 atoms.
- merge accepts the two honest reasons a corpus file has no product source
  (500 byte-identical LongMemEval sessions collapse under content hashing, and
  12 photos that ONNX Runtime cannot decode) and reports them.
- --arms used action=extend against a non-empty default, so naming two arms ran
  seven.
- the dense cache is keyed by the atom set, so a scale-curve subset cannot be
  scored with full-corpus vectors.
- write_markdown crashed on a run that covered only some modalities.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-06 16:49:11 -04:00
1 parent fc0698c242
commit a4953335e7
7 files changed
+238 -26

No files matched your search

+12
View File
@@ -50,6 +50,18 @@ Treat that command as the check of record for import health; treat pyright's *ty
(it has caught real defects here: an `AtomRunner` typo, missing `strict=True` on `zip`,
`dict[str, object]` over-narrowing) as separate and worth fixing.
The pi-lens mypy cascade also replays **stale** findings after a fix: it attributes another
file's lines to whichever file was edited last (e.g. `bench/fetch_models.py:107: "object" has
no attribute "encode"`, where line 107 of that file is a `Path` call and the quoted text is
from `bench/systems.py`). The check of record is a fresh-cache run, which has passed on every
current file:
```bash
rm -rf /tmp/mypycache && .venv/bin/python -m mypy --cache-dir /tmp/mypycache bench/ # Success: no issues found
.venv/bin/python -m ruff check bench/ # All checks passed
.venv/bin/python -m pytest bench/tests -q # all green
```
## corpus facts that constrain the numbers
- 50,789 sources / 5,308 queries (2,928 answerable, 2,380 unanswerable), all third-party.