ZacharyZhang-NY/ely-eye
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
deabbf76b89b0bb977fb24cc0513f35d4dfaad21
ely-eye/bench/reports/BASELINE.md
T
ZacharyZhang-NY c4e2982506 Record machine load with every latency number 
Latency measured while another job hogs the cores is not comparable: the scale curve
moved from 1.315s to 0.809s at 50,000 sources between two runs of the same code.

- bench/systems.py: machine_facts() reports cpus plus the 1/5/15 minute load average.
- bench/run.py, bench/run_external.py: both reports carry it, and BASELINE.md labels
  the latency table a ceiling when the machine was busy.

Verification: pytest bench/tests -q -> 60 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
2026-10-07 12:03:44 -04:00

3.0 KiB
Raw Blame History

BASELINE: what the product retrieves today

  • generated 2026-10-07T15:33:42+00:00 at commit 29163632af
  • corpus 50278 sources, 5308 queries (2928 answerable)
  • models: dense sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, rerank BAAI/bge-reranker-base
arm recall@1 recall@5 recall@10 nDCG@10 MRR hit@10 (95% lower) p50 s p95 s
chance (random ranking) 0.000 0.000 0.000 - - - - -
bm25_only 0.313 0.465 0.519 0.415 0.383 0.524 (0.505) 0.563 3.141
dense_only 0.227 0.363 0.424 0.321 0.290 0.429 (0.411) 1.728 2.883
graph_only 0.000 0.001 0.001 0.000 0.000 0.001 (0.000) 0.043 0.058
no_rerank 0.342 0.510 0.565 0.452 0.418 0.571 (0.553) 2.404 5.169
fusion_rerank 0.236 0.391 0.453 0.341 0.309 0.459 (0.441) 4.538 9.043

recall@10 by modality

arm pdf image ui_screenshot video code conversation
bm25_only 0.704 0.637 0.322 0.126 0.518 0.368
dense_only 0.698 0.481 0.075 0.228 0.382 0.304
graph_only 0.000 0.000 0.000 0.000 0.002 0.000
no_rerank 0.742 0.669 0.297 0.153 0.592 0.447
fusion_rerank 0.253 0.692 0.300 0.088 0.629 0.486

unanswerable queries (44.8% of the suite)

  • bm25_only: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 38% of them
  • dense_only: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 25% of them
  • graph_only: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 0% of them
  • no_rerank: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 40% of them
  • fusion_rerank: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 34% of them

the unmodified product path

  • RetrievalService.search: p50 20.16s, mean 20.67s, max 48.90s over 40 queries
  • top-10 overlap with the fast BM25 path: 0.0%, identical order 0.0%
  • its top-10 atoms cover only 7.6 distinct sources (min 1): the product ranks atoms and never diversifies, so ten slots fill with fragments of one document and the overlap with a source-level ranking is near zero

latency vs corpus size

sources atoms bm25_only p50/p95 s dense_only p50/p95 s
500 500 0.001/0.002 1.380/1.576
2000 2000 0.008/0.016 1.398/2.595
10000 10000 0.052/0.128 1.338/2.266
50000 165695 0.809/2.274 1.496/2.464

latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless

reading

  • recall counts a source as found only when it is the source a third party marked correct.
  • abstention is the only honest number for unanswerable queries: a retriever that always returns ten results never says it does not know.
  • chance is the floor any of these arms must beat; a number near chance is not a capability.
Reference in new issue View Git Blame Copy Permalink