ZacharyZhang-NY
c4e2982506
Latency measured while another job hogs the cores is not comparable: the scale curve moved from 1.315s to 0.809s at 50,000 sources between two runs of the same code. - bench/systems.py: machine_facts() reports cpus plus the 1/5/15 minute load average. - bench/run.py, bench/run_external.py: both reports carry it, and BASELINE.md labels the latency table a ceiling when the machine was busy. Verification: pytest bench/tests -q -> 60 passed; ruff check bench/ -> all checks passed; mypy (fresh cache) -> no issues in 25 source files.
57 lines
3.0 KiB
Markdown
57 lines
3.0 KiB
Markdown
# BASELINE: what the product retrieves today
|
|
|
|
- generated 2026-10-07T15:33:42+00:00 at commit 29163632af74
|
|
- corpus 50278 sources, 5308 queries (2928 answerable)
|
|
- models: dense `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, rerank `BAAI/bge-reranker-base`
|
|
|
|
| arm | recall@1 | recall@5 | recall@10 | nDCG@10 | MRR | hit@10 (95% lower) | p50 s | p95 s |
|
|
|---|---|---|---|---|---|---|---|---|
|
|
| chance (random ranking) | 0.000 | 0.000 | 0.000 | - | - | - | - | - |
|
|
| bm25_only | 0.313 | 0.465 | 0.519 | 0.415 | 0.383 | 0.524 (0.505) | 0.563 | 3.141 |
|
|
| dense_only | 0.227 | 0.363 | 0.424 | 0.321 | 0.290 | 0.429 (0.411) | 1.728 | 2.883 |
|
|
| graph_only | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.001 (0.000) | 0.043 | 0.058 |
|
|
| no_rerank | 0.342 | 0.510 | 0.565 | 0.452 | 0.418 | 0.571 (0.553) | 2.404 | 5.169 |
|
|
| fusion_rerank | 0.236 | 0.391 | 0.453 | 0.341 | 0.309 | 0.459 (0.441) | 4.538 | 9.043 |
|
|
|
|
## recall@10 by modality
|
|
|
|
| arm | pdf | image | ui_screenshot | video | code | conversation |
|
|
|---|---|---|---|---|---|---|
|
|
| bm25_only | 0.704 | 0.637 | 0.322 | 0.126 | 0.518 | 0.368 |
|
|
| dense_only | 0.698 | 0.481 | 0.075 | 0.228 | 0.382 | 0.304 |
|
|
| graph_only | 0.000 | 0.000 | 0.000 | 0.000 | 0.002 | 0.000 |
|
|
| no_rerank | 0.742 | 0.669 | 0.297 | 0.153 | 0.592 | 0.447 |
|
|
| fusion_rerank | 0.253 | 0.692 | 0.300 | 0.088 | 0.629 | 0.486 |
|
|
|
|
## unanswerable queries (44.8% of the suite)
|
|
|
|
- bm25_only: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 38% of them
|
|
- dense_only: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 25% of them
|
|
- graph_only: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 0% of them
|
|
- no_rerank: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 40% of them
|
|
- fusion_rerank: abstains on 0% of unanswerable queries, and puts a wrong-holder source on top for 34% of them
|
|
|
|
## the unmodified product path
|
|
|
|
- RetrievalService.search: p50 20.16s, mean 20.67s, max 48.90s over 40 queries
|
|
- top-10 overlap with the fast BM25 path: 0.0%, identical order 0.0%
|
|
- its top-10 atoms cover only 7.6 distinct sources (min 1): the product ranks atoms and never diversifies, so ten slots fill with fragments of one document and the overlap with a source-level ranking is near zero
|
|
|
|
## latency vs corpus size
|
|
|
|
| sources | atoms | bm25_only p50/p95 s | dense_only p50/p95 s |
|
|
|---|---|---|---|
|
|
| 500 | 500 | 0.001/0.002 | 1.380/1.576 |
|
|
| 2000 | 2000 | 0.008/0.016 | 1.398/2.595 |
|
|
| 10000 | 10000 | 0.052/0.128 | 1.338/2.266 |
|
|
| 50000 | 165695 | 0.809/2.274 | 1.496/2.464 |
|
|
|
|
_latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless_
|
|
|
|
## reading
|
|
|
|
- recall counts a source as found only when it is the source a third party marked correct.
|
|
- abstention is the only honest number for unanswerable queries: a retriever that always
|
|
returns ten results never says it does not know.
|
|
- chance is the floor any of these arms must beat; a number near chance is not a capability.
|