Record machine load with every latency number

Latency measured while another job hogs the cores is not comparable: the scale curve
moved from 1.315s to 0.809s at 50,000 sources between two runs of the same code.

- bench/systems.py: machine_facts() reports cpus plus the 1/5/15 minute load average.
- bench/run.py, bench/run_external.py: both reports carry it, and BASELINE.md labels
  the latency table a ceiling when the machine was busy.

Verification: pytest bench/tests -q -> 60 passed; ruff check bench/ -> all checks
passed; mypy (fresh cache) -> no issues in 25 source files.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 12:03:44 -04:00
1 parent 61f6506104
commit c4e2982506
6 files changed
+62 -27

No files matched your search

+11 -5
View File
@@ -1,6 +1,6 @@
# BASELINE: what the product retrieves today
- generated 2026-10-07T05:35:00+00:00 at commit a4953335e77f
- generated 2026-10-07T15:33:42+00:00 at commit 29163632af74
- corpus 50278 sources, 5308 queries (2928 answerable)
- models: dense `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, rerank `BAAI/bge-reranker-base`
@@ -33,14 +33,20 @@
## the unmodified product path
- RetrievalService.search: p50 15.54s, mean 16.68s, max 56.14s over 40 queries
- RetrievalService.search: p50 20.16s, mean 20.67s, max 48.90s over 40 queries
- top-10 overlap with the fast BM25 path: 0.0%, identical order 0.0%
- its top-10 atoms cover only 7.6 distinct sources (min 1): the product ranks atoms and never diversifies, so ten slots fill with fragments of one document and the overlap with a source-level ranking is near zero
## latency vs corpus size
| sources | atoms | bm25_only | dense_only |
|---|---|---|---|---|
| 50000 | 165695 | 0.804/1.576 | 1.446/2.538 |
| sources | atoms | bm25_only p50/p95 s | dense_only p50/p95 s |
|---|---|---|---|
| 500 | 500 | 0.001/0.002 | 1.380/1.576 |
| 2000 | 2000 | 0.008/0.016 | 1.398/2.595 |
| 10000 | 10000 | 0.052/0.128 | 1.338/2.266 |
| 50000 | 165695 | 0.809/2.274 | 1.496/2.464 |
_latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless_
## reading
+20 -20
View File
@@ -564,9 +564,9 @@
"min": 1
},
"identical_top10": 0.0,
"max": 63.2751505510023,
"mean": 21.701527132049886,
"p50": 21.831516211001144,
"max": 48.895113198996114,
"mean": 20.665571158625244,
"p50": 20.157494062004844,
"queries": 40,
"top10_overlap_with_bm25_only": 0.0,
"top_k": 10
@@ -575,7 +575,7 @@
"answerable": 2928,
"corpus_manifest_sha256": "8fc1ed861c891d778853d41107630869f9eb267a669c5d1038e389de80114347",
"corpus_sources": 50278,
"generated_at": "2026-10-07T14:38:12+00:00",
"generated_at": "2026-10-07T15:33:42+00:00",
"git_commit": "29163632af7418758e79a79a748f68050ca1e0b6",
"limit_per_modality": null,
"machine": {
@@ -602,12 +602,12 @@
{
"atoms": 500,
"bm25_only": {
"p50": 0.0007463944966730196,
"p95": 0.0026389723490865425
"p50": 0.0006466769991675392,
"p95": 0.00181096804953995
},
"dense_only": {
"p50": 2.053756675497425,
"p95": 3.7252893005985244
"p50": 1.3797472870028287,
"p95": 1.5762464397041183
},
"note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless",
"queries": 40,
@@ -616,12 +616,12 @@
{
"atoms": 2000,
"bm25_only": {
"p50": 0.00923213200076134,
"p95": 0.023986968196186332
"p50": 0.008334371003002161,
"p95": 0.015740483100307734
},
"dense_only": {
"p50": 2.2580291965023207,
"p95": 4.0939969898456186
"p50": 1.3979396294998878,
"p95": 2.595364080850413
},
"note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless",
"queries": 40,
@@ -630,12 +630,12 @@
{
"atoms": 10000,
"bm25_only": {
"p50": 0.11317921599766123,
"p95": 0.2961666731029253
"p50": 0.05245087000002968,
"p95": 0.12780161215196129
},
"dense_only": {
"p50": 2.269751224499487,
"p95": 3.6929099251963633
"p50": 1.3382894579990534,
"p95": 2.2659613330957646
},
"note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless",
"queries": 40,
@@ -644,12 +644,12 @@
{
"atoms": 165695,
"bm25_only": {
"p50": 1.3151575949996186,
"p95": 3.7234880074014645
"p50": 0.8085095155001909,
"p95": 2.273631770645442
},
"dense_only": {
"p50": 2.3852021460043034,
"p95": 4.008884304698221
"p50": 1.4964509784986149,
"p95": 2.4635968941045574
},
"note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless",
"queries": 40,