Join the product's source ids to the benchmark's before scoring
The merged database names a source by content hash (`11089_4f2a...`) while the frozen sets name the same file by its corpus id, so every arm scored 0.000 until load_atoms joined atoms through sources.path. It now returns corpus ids and refuses an atom it cannot trace to a corpus file. Also fixes found by running the harness: - merge rebuilt atom_fts from the deduped atoms table; two attempts on one file left 288,266 fts rows for 165,973 atoms. - merge accepts the two honest reasons a corpus file has no product source (500 byte-identical LongMemEval sessions collapse under content hashing, and 12 photos that ONNX Runtime cannot decode) and reports them. - --arms used action=extend against a non-empty default, so naming two arms ran seven. - the dense cache is keyed by the atom set, so a scale-curve subset cannot be scored with full-corpus vectors. - write_markdown crashed on a run that covered only some modalities.
This commit is contained in:
1 parent
fc0698c242
commit
a4953335e7
7 files changed
+238
-26
No files matched your search
@@ -50,6 +50,18 @@ Treat that command as the check of record for import health; treat pyright's *ty
|
||||
(it has caught real defects here: an `AtomRunner` typo, missing `strict=True` on `zip`,
|
||||
`dict[str, object]` over-narrowing) as separate and worth fixing.
|
||||
|
||||
The pi-lens mypy cascade also replays **stale** findings after a fix: it attributes another
|
||||
file's lines to whichever file was edited last (e.g. `bench/fetch_models.py:107: "object" has
|
||||
no attribute "encode"`, where line 107 of that file is a `Path` call and the quoted text is
|
||||
from `bench/systems.py`). The check of record is a fresh-cache run, which has passed on every
|
||||
current file:
|
||||
|
||||
```bash
|
||||
rm -rf /tmp/mypycache && .venv/bin/python -m mypy --cache-dir /tmp/mypycache bench/ # Success: no issues found
|
||||
.venv/bin/python -m ruff check bench/ # All checks passed
|
||||
.venv/bin/python -m pytest bench/tests -q # all green
|
||||
```
|
||||
|
||||
## corpus facts that constrain the numbers
|
||||
|
||||
- 50,789 sources / 5,308 queries (2,928 answerable, 2,380 unanswerable), all third-party.
|
||||
|
||||
Reference in new issue
Block a user