ZacharyZhang-NY
c4a8c36416
Every number in CHECKLIST p2-1 and p0-3.7 came out of a scratch script: the
kind of artifact a reader cannot re-run, and the kind that quietly drifts
away from the code it describes. These reports are generated by committed
harnesses and each one stamps the commit that carries the code it measures.
DENSE.md 165,973 atoms / 153,857 vectors: build 1421.0s, search p50 75.6ms
p95 93.0ms, 0.998 agreement with a brute-force scan, one write into
a full index p50 518ms, 1,729 bytes per stored vector.
QUANT.md twelve encodings measured on the same vectors: int8 keeps 0.977 of
the ranking for a quarter of the bytes and no speed, bit is 14x
faster and keeps 0.728, so bit is a coarse filter and nothing more.
TERMS.md the stop-word bound swept, not guessed: 0.05 is the measured peak
(recall@10 0.803 at p50 69.5ms) and also the fastest of the rows
that keep recall.
SCALE.md the product's own search at 165,973 atoms: p50 71.7ms against the
frozen baseline's 20.16s.
LEGS.md what each retrieval leg is worth, per modality, with the page size
the product actually asks for in the table.
The acceptance bar the operator set is recall@10 >= 0.90 at n>=300. It is
not met. The best measured row is 0.858 (dense leg on, page widened to 30,
p50 414ms); the shipped configuration gives 0.848 at p50 72.5ms; the
in-repo counterpart at holder scope gives 0.866. The shortfall is
concentrated in picture golds (video 0.583-0.708, ui_screenshot 0.500-0.583,
image 0.750) while the word-shaped ones already pass (code 0.953, pdf 0.919).
Latency passes by three hundred fold; recall does not, and the gap is four
points of cross-modal retrieval, not index scale.
2.3 KiB
2.3 KiB
LEGS: what each retrieval leg is worth
- commit:
dff5977ab638896cc8685c78ee41e5195fc0fbf2, schema v8 - store: bench/runs/leg_check/data/ely_eye.sqlite (a copy of bench/runs/dense_scale; the frozen baseline is never opened)
- 300 answerable queries, each asked as the holder the frozen set assigns it
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [25.51, 9.62, 5.7]
Recall is scored on the first ten sources after atoms are collapsed to their source by first hit, which is how every other arm in this repo is scored.
| leg configuration | atoms asked | recall@10 | MRR | hit@10 | 95% lower | p50 ms | p95 ms | distinct sources |
|---|---|---|---|---|---|---|---|---|
| words only | 10 | 0.832 | 0.683 | 0.833 | 0.787 | 65.5 | 134.9 | 4.0 |
| words only (the configured 24) | 24 | 0.848 | 0.688 | 0.850 | 0.805 | 72.5 | 155.9 | 6.0 |
| words only, wider page | 30 | 0.855 | 0.690 | 0.857 | 0.812 | 64.7 | 129.7 | 6.0 |
| + dense leg | 10 | 0.845 | 0.663 | 0.847 | 0.802 | 472.9 | 577.5 | 5.0 |
| + dense leg, wider page | 30 | 0.858 | 0.669 | 0.860 | 0.816 | 413.7 | 525.8 | 7.0 |
| + picture leg | 10 | 0.815 | 0.553 | 0.817 | 0.769 | 132.5 | 271.3 | 8.0 |
| + picture leg, wider page | 30 | 0.825 | 0.551 | 0.827 | 0.780 | 162.8 | 283.7 | 20.0 |
recall@10 by modality
| leg configuration | code | image | ui_screenshot | video | |
|---|---|---|---|---|---|
| words only | 0.953 | 0.750 | 0.849 | 0.583 | 0.583 |
| words only (the configured 24) | 0.953 | 0.750 | 0.895 | 0.583 | 0.625 |
| words only, wider page | 0.953 | 0.750 | 0.919 | 0.583 | 0.625 |
| + dense leg | 0.938 | 0.750 | 0.884 | 0.639 | 0.625 |
| + dense leg, wider page | 0.938 | 0.750 | 0.919 | 0.639 | 0.667 |
| + picture leg | 0.930 | 0.750 | 0.849 | 0.528 | 0.583 |
| + picture leg, wider page | 0.898 | 0.750 | 0.907 | 0.500 | 0.708 |
Query counts per modality: code 128, image 26, pdf 86, ui_screenshot 36, video 24.
The picture leg is not a switch this harness flips per row: the product runs it only when the sparse page comes back thin, so these rows measure the product with the leg available, not a forced always-on fusion.
Latency is wall time per search on the machine named above with the thread budget the benchmark pins; a run on a loaded machine is not comparable.