Re-stamp the scale reports against the harness that now carries the fix
Both reports were produced by a harness that had just changed, so their stamps named a commit without that code in it. Re-running them on an idle machine puts the stamp and the source back in sync. Recall did not move: all seven leg configurations reproduce to the third decimal, which is the part of the number anyone should act on. Latency moved within its usual load band, and the encoder probe came in slower this time, so the picture index's build cost is now quoted as a band rather than a single remembered figure.
This commit is contained in:
ZacharyZhang-NY committed
1 parent
71733d86d6
commit
e80883cfc9
5 files changed
+89
-73
No files matched your search
+11
-11
@@ -1,23 +1,23 @@
|
||||
# DENSE: the vector leg at corpus size (commit dff5977ab)
|
||||
# DENSE: the vector leg at corpus size (commit 71733d86d)
|
||||
|
||||
- database: `bench/runs/dense_scale/data/ely_eye.sqlite`
|
||||
- atoms: 165973, memories with words: 153857, vectors: 153857, pending: 0
|
||||
- model: `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, width 384
|
||||
- build: 1421.0s for 153857 memories (108 per second)
|
||||
- writing one memory into a full index: p50 518ms · p95 543ms over 50 writes
|
||||
- build: 1444.9s for 153857 memories (106 per second)
|
||||
- writing one memory into a full index: p50 508ms · p95 535ms over 50 writes
|
||||
- the repair scan (`ely-eye build-dense-index`) over a full index: 1.3s, and it is not on the search path
|
||||
- search p50 75.6ms · p95 93.0ms · max 123.6ms over 300 queries
|
||||
- of which encoding the question: p50 12.4ms · p95 25.8ms
|
||||
- search p50 73.0ms · p95 75.7ms · max 85.6ms over 300 queries
|
||||
- of which encoding the question: p50 11.3ms · p95 24.0ms
|
||||
- against a brute-force scan: 0.998 of the top 10 over 40 probes (a returned atom counts when its true cosine is at least the k-th best, so ties cannot fail the check)
|
||||
- bytes per stored vector: 1729 (266059776 bytes for 153857 vectors, dbstat page sizes over the vec0 shadow tables)
|
||||
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [3.18, 4.58, 5.42]
|
||||
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [3.53, 4.63, 4.85]
|
||||
|
||||
## One million memories on this machine
|
||||
## One million vectors on this machine
|
||||
|
||||
- p50 455ms · p95 561ms
|
||||
- p50 475ms · p95 492ms
|
||||
- vector table: 1.6 GiB
|
||||
- first build: 153.9 minutes
|
||||
- writing one memory: 3123ms — a vec0 delete by id scans the table, so this grows with the count too
|
||||
- assumption: sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the count; the model, the machine and the thread budget stay the same
|
||||
- first build: 156.5 minutes
|
||||
- writing one vector: 3303ms — a vec0 delete by id scans the table, so this grows with the count too
|
||||
- assumption: sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the vector count; every row here is scaled by 1000000 / 153857 vectors, and the model, the machine and the thread budget stay the same
|
||||
|
||||
_Latency is wall time per search on the machine named above with the thread budget the benchmark pins; a run on a loaded machine is not comparable._
|
||||
+12
-10
@@ -1,22 +1,24 @@
|
||||
# LEGS: what each retrieval leg is worth
|
||||
|
||||
- commit: `dff5977ab638896cc8685c78ee41e5195fc0fbf2`, schema v8
|
||||
- store: bench/runs/leg_check/data/ely_eye.sqlite (a copy of bench/runs/dense_scale; the frozen baseline is never opened)
|
||||
- commit: `71733d86d6d27d1f1ebb75205d5e0599f253f791`, schema v8
|
||||
- store: bench/runs/leg_check/data/ely_eye.sqlite — a copy of bench/runs/dense_scale, which carries 153,857 dense vectors built by bench/dense_scale. The build sits outside the measured latency, and the frozen baseline itself holds no vectors, so a dense arm there would rank an empty index.
|
||||
- 300 answerable queries, each asked as the holder the frozen set assigns it
|
||||
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [25.51, 9.62, 5.7]
|
||||
- picture index: 53,668 atoms at 512 dimensions, 109,912,192 bytes, read from bench/runs/leg_check_index
|
||||
- encoder speed: 16.8 pictures/s measured on 200 pictures, so the 53,668 pictures in that index cost about 53.3 minutes to encode
|
||||
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [22.19, 9.9, 6.39]
|
||||
|
||||
Recall is scored on the first ten **sources** after atoms are collapsed to their source
|
||||
by first hit, which is how every other arm in this repo is scored.
|
||||
|
||||
| leg configuration | atoms asked | recall@10 | MRR | hit@10 | 95% lower | p50 ms | p95 ms | distinct sources |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| words only | 10 | 0.832 | 0.683 | 0.833 | 0.787 | 65.5 | 134.9 | 4.0 |
|
||||
| words only (the configured 24) | 24 | 0.848 | 0.688 | 0.850 | 0.805 | 72.5 | 155.9 | 6.0 |
|
||||
| words only, wider page | 30 | 0.855 | 0.690 | 0.857 | 0.812 | 64.7 | 129.7 | 6.0 |
|
||||
| + dense leg | 10 | 0.845 | 0.663 | 0.847 | 0.802 | 472.9 | 577.5 | 5.0 |
|
||||
| + dense leg, wider page | 30 | 0.858 | 0.669 | 0.860 | 0.816 | 413.7 | 525.8 | 7.0 |
|
||||
| + picture leg | 10 | 0.815 | 0.553 | 0.817 | 0.769 | 132.5 | 271.3 | 8.0 |
|
||||
| + picture leg, wider page | 30 | 0.825 | 0.551 | 0.827 | 0.780 | 162.8 | 283.7 | 20.0 |
|
||||
| words only | 10 | 0.832 | 0.683 | 0.833 | 0.787 | 67.5 | 146.7 | 4.0 |
|
||||
| words only (the configured 24) | 24 | 0.848 | 0.688 | 0.850 | 0.805 | 67.6 | 129.9 | 6.0 |
|
||||
| words only, wider page | 30 | 0.855 | 0.690 | 0.857 | 0.812 | 72.0 | 156.0 | 6.0 |
|
||||
| + dense leg | 10 | 0.845 | 0.663 | 0.847 | 0.802 | 474.8 | 579.1 | 5.0 |
|
||||
| + dense leg, wider page | 30 | 0.858 | 0.669 | 0.860 | 0.816 | 416.0 | 543.4 | 7.0 |
|
||||
| + picture leg | 10 | 0.815 | 0.553 | 0.817 | 0.769 | 138.1 | 303.6 | 8.0 |
|
||||
| + picture leg, wider page | 30 | 0.825 | 0.551 | 0.827 | 0.780 | 166.7 | 314.2 | 20.0 |
|
||||
|
||||
## recall@10 by modality
|
||||
|
||||
|
||||
+26
-25
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"commit": "dff5977ab638896cc8685c78ee41e5195fc0fbf2",
|
||||
"commit": "71733d86d6d27d1f1ebb75205d5e0599f253f791",
|
||||
"database": "bench/runs/dense_scale/data/ely_eye.sqlite",
|
||||
"atoms": 165973,
|
||||
"memories_with_words": 153857,
|
||||
@@ -7,17 +7,17 @@
|
||||
"pending": 0,
|
||||
"dim": 384,
|
||||
"model": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
|
||||
"copy_seconds": 0.665644496002642,
|
||||
"build_seconds": 1421.0276969089973,
|
||||
"texts_per_second": 108.2716405420302,
|
||||
"repair_seconds": 1.2946931579936063,
|
||||
"copy_seconds": 4.653337935000309,
|
||||
"build_seconds": 1444.851663221998,
|
||||
"texts_per_second": 106.48636390596745,
|
||||
"repair_seconds": 1.2708722250099527,
|
||||
"writes": 50,
|
||||
"write_latency": {
|
||||
"queries": 50,
|
||||
"mean": 0.5204645371803781,
|
||||
"p50": 0.5183807220018934,
|
||||
"p95": 0.5429989652060613,
|
||||
"max": 0.5760761429992272
|
||||
"mean": 0.5132235406010295,
|
||||
"p50": 0.50813193799695,
|
||||
"p95": 0.5353954541016719,
|
||||
"max": 0.5569016869994812
|
||||
},
|
||||
"probes": 300,
|
||||
"probe_modalities": [
|
||||
@@ -29,17 +29,17 @@
|
||||
],
|
||||
"latency": {
|
||||
"queries": 300,
|
||||
"mean": 0.07861847096656371,
|
||||
"p50": 0.07557305500085931,
|
||||
"p95": 0.09303550984986941,
|
||||
"max": 0.123588470989489
|
||||
"mean": 0.07290171435301697,
|
||||
"p50": 0.07302105800044956,
|
||||
"p95": 0.07570205154406721,
|
||||
"max": 0.08561979999649338
|
||||
},
|
||||
"encode_latency": {
|
||||
"queries": 300,
|
||||
"mean": 0.014255492710411393,
|
||||
"p50": 0.012439899503078777,
|
||||
"p95": 0.025788354348333098,
|
||||
"max": 0.0513256230042316
|
||||
"mean": 0.012841759066456385,
|
||||
"p50": 0.011289245005173143,
|
||||
"p95": 0.024048260039126034,
|
||||
"max": 0.02614720400015358
|
||||
},
|
||||
"correctness": {
|
||||
"probes": 40,
|
||||
@@ -52,12 +52,13 @@
|
||||
"bytes_method": "dbstat page sizes over the vec0 shadow tables",
|
||||
"extrapolation": {
|
||||
"to": 1000000,
|
||||
"assumption": "sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the count; the model, the machine and the thread budget stay the same",
|
||||
"p50_seconds": 0.45533342773137386,
|
||||
"p95_seconds": 0.5605460517666694,
|
||||
"write_p50_seconds": 3.1232834376789804,
|
||||
"basis": 153857,
|
||||
"assumption": "sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the vector count; every row here is scaled by 1000000 / 153857 vectors, and the model, the machine and the thread budget stay the same",
|
||||
"p50_seconds": 0.47460341746199103,
|
||||
"p95_seconds": 0.49202864701682225,
|
||||
"write_p50_seconds": 3.3026247619344584,
|
||||
"gibibytes": 1.6105050507025191,
|
||||
"build_seconds": 9236.028889871746
|
||||
"build_seconds": 9390.873754343305
|
||||
},
|
||||
"machine": {
|
||||
"system": "Linux",
|
||||
@@ -65,9 +66,9 @@
|
||||
"cpus": 32,
|
||||
"threads_per_query": 4,
|
||||
"load_average_1_5_15": [
|
||||
3.18,
|
||||
4.58,
|
||||
5.42
|
||||
3.53,
|
||||
4.63,
|
||||
4.85
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,20 @@
|
||||
{
|
||||
"commit": "dff5977ab638896cc8685c78ee41e5195fc0fbf2",
|
||||
"commit": "71733d86d6d27d1f1ebb75205d5e0599f253f791",
|
||||
"schema_version": 8,
|
||||
"store": "bench/runs/leg_check/data/ely_eye.sqlite",
|
||||
"source_store": "bench/runs/dense_scale",
|
||||
"dense_vectors": 153857,
|
||||
"picture_index": {
|
||||
"path": "bench/runs/leg_check_index",
|
||||
"atoms": 53668,
|
||||
"dim": 512,
|
||||
"bytes": 109912192
|
||||
},
|
||||
"encode_rate": {
|
||||
"pictures": 200,
|
||||
"seconds": 11.912494190008147,
|
||||
"per_second": 16.7890952818054
|
||||
},
|
||||
"probes": 300,
|
||||
"arms": [
|
||||
{
|
||||
@@ -13,8 +26,8 @@
|
||||
"mrr": 0.6832447089947089,
|
||||
"hit_at_10": 0.8333333333333334,
|
||||
"wilson_lower": 0.7870033503873167,
|
||||
"p50_seconds": 0.0654734540003119,
|
||||
"p95_seconds": 0.13490134734311146,
|
||||
"p50_seconds": 0.0674771249978221,
|
||||
"p95_seconds": 0.14665940315098855,
|
||||
"median_distinct_sources": 4.0,
|
||||
"by_modality": {
|
||||
"code": 0.953125,
|
||||
@@ -40,8 +53,8 @@
|
||||
"mrr": 0.6875571326821327,
|
||||
"hit_at_10": 0.85,
|
||||
"wilson_lower": 0.8051823888546709,
|
||||
"p50_seconds": 0.07251374649786158,
|
||||
"p95_seconds": 0.15588207065375176,
|
||||
"p50_seconds": 0.0675995074925595,
|
||||
"p95_seconds": 0.12988004180151622,
|
||||
"median_distinct_sources": 6.0,
|
||||
"by_modality": {
|
||||
"code": 0.953125,
|
||||
@@ -67,8 +80,8 @@
|
||||
"mrr": 0.6895710215710216,
|
||||
"hit_at_10": 0.8566666666666667,
|
||||
"wilson_lower": 0.8124994228503402,
|
||||
"p50_seconds": 0.06468957299512113,
|
||||
"p95_seconds": 0.1297007056004077,
|
||||
"p50_seconds": 0.07200110949634109,
|
||||
"p95_seconds": 0.15601039270331973,
|
||||
"median_distinct_sources": 6.0,
|
||||
"by_modality": {
|
||||
"code": 0.953125,
|
||||
@@ -94,8 +107,8 @@
|
||||
"mrr": 0.6632195767195768,
|
||||
"hit_at_10": 0.8466666666666667,
|
||||
"wilson_lower": 0.8015339429418983,
|
||||
"p50_seconds": 0.4729458744986914,
|
||||
"p95_seconds": 0.5775450093562541,
|
||||
"p50_seconds": 0.4748052095019375,
|
||||
"p95_seconds": 0.5791264187995694,
|
||||
"median_distinct_sources": 5.0,
|
||||
"by_modality": {
|
||||
"code": 0.9375,
|
||||
@@ -121,8 +134,8 @@
|
||||
"mrr": 0.6686637344137344,
|
||||
"hit_at_10": 0.86,
|
||||
"wilson_lower": 0.8161683876490509,
|
||||
"p50_seconds": 0.41367531901050825,
|
||||
"p95_seconds": 0.5258364057008293,
|
||||
"p50_seconds": 0.41600067100080196,
|
||||
"p95_seconds": 0.5433694835548523,
|
||||
"median_distinct_sources": 7.0,
|
||||
"by_modality": {
|
||||
"code": 0.9375,
|
||||
@@ -148,8 +161,8 @@
|
||||
"mrr": 0.5525886243386243,
|
||||
"hit_at_10": 0.8166666666666667,
|
||||
"wilson_lower": 0.7689713823226987,
|
||||
"p50_seconds": 0.13248339149868116,
|
||||
"p95_seconds": 0.2713221473451995,
|
||||
"p50_seconds": 0.1381109430003562,
|
||||
"p95_seconds": 0.3035783412473393,
|
||||
"median_distinct_sources": 8.0,
|
||||
"by_modality": {
|
||||
"code": 0.9296875,
|
||||
@@ -175,8 +188,8 @@
|
||||
"mrr": 0.5507809648412589,
|
||||
"hit_at_10": 0.8266666666666667,
|
||||
"wilson_lower": 0.7797738592974309,
|
||||
"p50_seconds": 0.16283495400421089,
|
||||
"p95_seconds": 0.2837207488548302,
|
||||
"p50_seconds": 0.16669872149941511,
|
||||
"p95_seconds": 0.3142305556466454,
|
||||
"median_distinct_sources": 20.0,
|
||||
"by_modality": {
|
||||
"code": 0.8984375,
|
||||
@@ -200,9 +213,9 @@
|
||||
"cpus": 32,
|
||||
"threads_per_query": 4,
|
||||
"load_average_1_5_15": [
|
||||
25.51,
|
||||
9.62,
|
||||
5.7
|
||||
22.19,
|
||||
9.9,
|
||||
6.39
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user