ZacharyZhang-NY/ely-eye
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Re-stamp the scale reports against the harness that now carries the fix

Browse Source 
Both reports were produced by a harness that had just changed, so their
stamps named a commit without that code in it. Re-running them on an idle
machine puts the stamp and the source back in sync.

Recall did not move: all seven leg configurations reproduce to the third
decimal, which is the part of the number anyone should act on. Latency moved
within its usual load band, and the encoder probe came in slower this time,
so the picture index's build cost is now quoted as a band rather than a
single remembered figure.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-08 19:30:46 -04:00
1 parent 71733d86d6
commit e80883cfc9
5 files changed
+89 -73
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

bench/reports/LEGS.md
+12 -10
View File
@@ -1,22 +1,24 @@
# LEGS: what each retrieval leg is worth
- commit: `dff5977ab638896cc8685c78ee41e5195fc0fbf2`, schema v8
- store: bench/runs/leg_check/data/ely_eye.sqlite (a copy of bench/runs/dense_scale; the frozen baseline is never opened)
- commit: `71733d86d6d27d1f1ebb75205d5e0599f253f791`, schema v8
- store: bench/runs/leg_check/data/ely_eye.sqlite — a copy of bench/runs/dense_scale, which carries 153,857 dense vectors built by bench/dense_scale. The build sits outside the measured latency, and the frozen baseline itself holds no vectors, so a dense arm there would rank an empty index.
- 300 answerable queries, each asked as the holder the frozen set assigns it
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [25.51, 9.62, 5.7]
- picture index: 53,668 atoms at 512 dimensions, 109,912,192 bytes, read from bench/runs/leg_check_index
- encoder speed: 16.8 pictures/s measured on 200 pictures, so the 53,668 pictures in that index cost about 53.3 minutes to encode
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [22.19, 9.9, 6.39]
Recall is scored on the first ten **sources** after atoms are collapsed to their source
by first hit, which is how every other arm in this repo is scored.
| leg configuration | atoms asked | recall@10 | MRR | hit@10 | 95% lower | p50 ms | p95 ms | distinct sources |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| words only | 10 | 0.832 | 0.683 | 0.833 | 0.787 | 65.5 | 134.9 | 4.0 |
| words only (the configured 24) | 24 | 0.848 | 0.688 | 0.850 | 0.805 | 72.5 | 155.9 | 6.0 |
| words only, wider page | 30 | 0.855 | 0.690 | 0.857 | 0.812 | 64.7 | 129.7 | 6.0 |
| + dense leg | 10 | 0.845 | 0.663 | 0.847 | 0.802 | 472.9 | 577.5 | 5.0 |
| + dense leg, wider page | 30 | 0.858 | 0.669 | 0.860 | 0.816 | 413.7 | 525.8 | 7.0 |
| + picture leg | 10 | 0.815 | 0.553 | 0.817 | 0.769 | 132.5 | 271.3 | 8.0 |
| + picture leg, wider page | 30 | 0.825 | 0.551 | 0.827 | 0.780 | 162.8 | 283.7 | 20.0 |
| words only | 10 | 0.832 | 0.683 | 0.833 | 0.787 | 67.5 | 146.7 | 4.0 |
| words only (the configured 24) | 24 | 0.848 | 0.688 | 0.850 | 0.805 | 67.6 | 129.9 | 6.0 |
| words only, wider page | 30 | 0.855 | 0.690 | 0.857 | 0.812 | 72.0 | 156.0 | 6.0 |
| + dense leg | 10 | 0.845 | 0.663 | 0.847 | 0.802 | 474.8 | 579.1 | 5.0 |
| + dense leg, wider page | 30 | 0.858 | 0.669 | 0.860 | 0.816 | 416.0 | 543.4 | 7.0 |
| + picture leg | 10 | 0.815 | 0.553 | 0.817 | 0.769 | 138.1 | 303.6 | 8.0 |
| + picture leg, wider page | 30 | 0.825 | 0.551 | 0.827 | 0.780 | 166.7 | 314.2 | 20.0 |
## recall@10 by modality