diff --git a/CHECKLIST.md b/CHECKLIST.md index 17a7c33..181706d 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -75,9 +75,9 @@ - [x] p0-3.4 七个过滤器(holder/scenario/source/modality/since/until/min_trust)+ base64 游标;`since > until` 报 "search since must not be after until",游标坏了报 "search cursor is unreadable" - [x] p0-3.5 RRF(`RRF_K = 60`)取代 `0.55/0.35/0.10`;顺手删掉无人调用的 `tokenize()` / `lexical_score()` / `_graph_score()`(graph 项恒为 0,因为证据边从不参与排序) - [x] p0-3.6 Go 的 `search_atoms` / `compile_context` 改走 `POST /api/search`,删掉 Go 侧第二套分词器(`store.SearchAtoms` / `ftsQuery` / `tokenPattern`)——两面不可能再漂开,因为只剩一个排序实现 - - [x] p0-3.7 100k atom 下 p50 <100ms:`python -m bench.scale_check` → `bench/reports/SCALE.md`。实测 **165,973 atom(bar 的 1.66 倍)、300 探针、p50 57ms / p95 138ms**,机器 load 1.76 / 32 核 / 每查询 4 线程;基线 p50 20.16s ⇒ **快 343 倍**(SCALE.md 重跑:p50 58.7ms / p95 143.0ms)。同一探针集上的质量与延迟扫描见 `bench/reports/TERMS.md`(n=173 holder 作用域,0.05 档 recall@10 0.803、MRR 0.652、p50 64.8ms / p95 121.7ms)。 + - [x] p0-3.7 100k atom 下 p50 <100ms:`python -m bench.scale_check` → `bench/reports/SCALE.md`。报告那一次实测 **165,973 atom(bar 的 1.66 倍)、300 探针、p50 71.7ms / p95 217.7ms / max 780.5ms**,机器 load [2.31, 3.72, 7.9] / 32 核 / 每查询 4 线程;基线 p50 20.16s ⇒ **快 281 倍**。同一脚本在更空的机器上跑到过 p50 56.5ms / 58.7ms(load 1.5–1.8)——延迟随负载走,所以引用报告那一次,不挑最好看的。同一探针集上的质量与延迟扫描见 `bench/reports/TERMS.md`(n=173 holder 作用域,0.05 档 recall@10 0.803、MRR 0.652、p50 69.5ms / p95 131.5ms)。 - 根因不是 SQL 慢,是 OR 表达式里那些「每个文档都有」的虚词:`"parsing"* OR "error"* OR "in"* OR "the"* …` 命中 127,145 行、305ms,而中文二元组短语查询 32 命中只要 1.0ms。bm25 必须给每条命中打分,延迟就跟着命中数走。 - - 修法:用索引自己当 IDF 源,按文档频率丢掉虚词(`text_index.rare_terms`,`COMMON_TERM_FRACTION = 0.05`,无手写停用词表,换语料自动换答案)。0.05 是扫出来的,扫描器已入库成证据:`bench/term_sweep.py` → `bench/reports/TERMS.md`(n=173,冻结库的副本):0.20→0.792/385.9ms、0.10→0.792/372.9ms、**0.05→0.803/64.8ms**、0.02→0.792/30.3ms、0.01→0.775/16.8ms、0.005→0.740/9.3ms —— 召回峰值就在 0.05。早先 scratch 脚本扫出的 0.893/0.899/0.870 已被取代:那些数字没有入库的产物,且是在检索改写(候选放宽规则、向量腿入库)之前测的。为此 `rare_terms(connection, terms, *, fraction=None)` 改成调用时解析上界——def 期默认值扫不动。 + - 修法:用索引自己当 IDF 源,按文档频率丢掉虚词(`text_index.rare_terms`,`COMMON_TERM_FRACTION = 0.05`,无手写停用词表,换语料自动换答案)。0.05 是扫出来的,扫描器已入库成证据:`bench/term_sweep.py` → `bench/reports/TERMS.md`(n=173,冻结库的副本):0.20→0.792/420.3ms、0.10→0.792/467.5ms、**0.05→0.803/69.5ms**、0.02→0.792/33.4ms、0.01→0.775/19.2ms、0.005→0.740/10.8ms —— 召回峰值就在 0.05。早先 scratch 脚本扫出的 0.893/0.899/0.870 已被取代:那些数字没有入库的产物,且是在检索改写(候选放宽规则、向量腿入库)之前测的。为此 `rare_terms(connection, terms, *, fraction=None)` 改成调用时解析上界——def 期默认值扫不动。 - 计数缓存 `term_df`(schema v6):一个词只数一次,任何 atom 写入都清空它,所以缓存不会比它所描述的库更老。冷缓存 p50 73ms / 热 63ms,1,324 个词被缓存。 - 验收:`pytest backend/tests/test_text_index.py backend/tests/test_search_api.py` 绿(18 条,真 FTS5 表 + TestClient):「钥匙」命中「我们的钥匙放在玄关抽屉里」、`search atoms` 命中 `func SearchAtoms(ctx context.Context)`、`forg` 命中 forgotten、全角 `Search Atoms` 命中、游标翻页无重复、supersede 后旧记忆不再应答;MCP 一致性由 `ely-eye-mcp/internal/mcpserver/search_tools_test.go`(3 条)+ `bash scripts/mcp_write_smoke.sh` PASS 证明;偏差:checklist 写的 `tests/test_retrieval_parity.py` 不存在,实际在 `backend/tests/` 与 Go 测试里 - [x] **p0-4** 卡带变成可挂载作用域 @@ -349,12 +349,18 @@ - `python -m bench.quant_probe` → `bench/reports/QUANT.md`(165,973 向量、200 探针、top 10,tie-aware 一致度):sqlite-vec float 40.3ms/1.0、+holder 过滤 29.3ms/0.599、int8 35.3ms/**0.977**、int8+holder 23.0/0.595、bit 2.7ms/0.728、bit+holder 4.2/0.513、numpy float32 全扫(真值)8.0ms/1.0、numpy 符号位 hamming 19.1/0.011、float32 前 192/128/64 维 4.4/0.722、4.0/0.57、4.0/0.43、**bit 粗排@200 → numpy 里 float32 精排 23.7ms/0.983**。每向量字节 f32 1536 / int8 384 / bit 48。 - 结论:int8 省 4× 空间几乎不掉质量(0.977)但不省时间;bit 省 32× 空间和 14× 时间、只剩 0.728,只能当粗排;两级方案必须在 vec0 **外面**精排(在库里用 `rowid IN (…)` 回捞反而更慢)。MRL 截断是在**不是 MRL 训练**的 MiniLM 上测的,所以那是真实代价不是白拿;切片必须是存下来的连续维,不是查询时切(非连续视图点集会掉速度)。 - 偏差:精排腿(`bge-reranker-base`)早在基线消融里就有并测过(加了它 recall@10 从 0.565 掉到 0.453),所以这里没再为它编一套故事;粗排+精排这条链测出 0.983 一致度但**没接线**,因为 dense 腿默认关(见下)。 - - [x] p2-1.3 100k 实测 + 1M 外推:`python -m bench.dense_scale` → `bench/reports/DENSE.md`(stamp cd9851a) - - 实测 **165,973 atom / 153,857 向量**:建库 1599.1s(96 向量/秒)、**检索 p50 74.6ms / p95 77.5ms / max 89.0ms(300 查询)**、其中编码问题 p50 11.3ms、与暴力扫描 tie-aware 一致度 **0.998**(40 探针)、**往满索引里写一条记忆 p50 514ms / p95 536ms(50 次写)**、失配扫描 1.7s(不在检索路径上)、**每向量 1,729 字节**(dbstat 数 vec0 影子表:266,059,776 B)。 - - 1M 外推:p50 443ms / p95 463ms、向量表 1.6 GiB、首次建库 154.2 分钟、单条写 3132ms,假设随行印出(逐条扫 ⇒ 线性)。 + - [x] p2-1.3 100k 实测 + 1M 外推:`python -m bench.dense_scale` → `bench/reports/DENSE.md`(stamp dff5977) + - 实测 **165,973 atom / 153,857 向量**:建库 1421.0s(108 向量/秒)、**检索 p50 75.6ms / p95 93.0ms / max 123.6ms(300 查询)**、其中编码问题 p50 12.4ms / p95 25.8ms、与暴力扫描 tie-aware 一致度 **0.998**(40 探针)、**往满索引里写一条记忆 p50 518ms / p95 543ms(50 次写)**、失配扫描 1.3s(不在检索路径上)、**每向量 1,729 字节**(dbstat 数 vec0 影子表:266,059,776 B)、机器 load [3.18, 4.58, 5.42]。 + - 1M 外推:p50 455ms / p95 561ms、向量表 1.6 GiB、首次建库 153.9 分钟、单条写 3123ms(vec0 按 id 删除也要扫表,所以写同样随规模线性),假设随行印出(逐条扫 ⇒ 线性)。 - tie-aware 的必要性:语料里有重复 atom,分数并列 ⇒ top-K 的 id 集合本身有歧义,两个都对引擎按 id 比会读出 0.827 的假召回。判定改成「返回的 atom 真实 cosine ≥ 第 K 名 cosine」。 - - 验收:100k dense 腿 p50 <100ms → **165,973 atom(bar 的 1.66 倍)p50 74.6ms**(DENSE.md);新 atom 立即可查 → `pytest backend/tests/test_vector_index.py`(写进去当场能搜到、视图藏起来的 dense 腿也藏、备份带走向量恢复后照样应答)。 - - 数字逼出来的决定:**dense 腿默认关**(`config.py:32 dense_embeddings_enabled=False`)。同一 300 条 holder 作用域探针上,开腿 recall@10 0.832→0.845、MRR 0.683→0.663、p50 64ms→482ms。多 1.3 个点的召回、慢 7 倍、首位质量还降——默认值就该是关,腿留着(已增量、已入库)等更大的语料或更好的嵌入模型。 + - 验收:100k dense 腿 p50 <100ms → **165,973 atom(bar 的 1.66 倍)p50 75.6ms**(DENSE.md);新 atom 立即可查 → `pytest backend/tests/test_vector_index.py`(写进去当场能搜到、视图藏起来的 dense 腿也藏、备份带走向量恢复后照样应答)。 + - [x] p2-1.4 每条腿值多少钱:`python -m bench.leg_check` → `bench/reports/LEGS.md`(stamp dff5977,300 条 holder 作用域探针,7 个配置,每臂带 n / MRR / Wilson 下界 / p50 / p95 / 分模态召回) + - 只有词的 FTS 页:问 10 个 atom recall@10 **0.832**、MRR 0.683、p50 65.5ms;问产品配置的 24 个 **0.848** / 0.688 / 72.5ms;页放宽到 30 **0.855** / 0.690 / 64.7ms(放宽不要钱,10→24 就白捡 1.6 个点)。 + - 开 dense 腿:10 个 atom **0.845** / MRR 0.663 / p50 472.9ms;页 30 **0.858** / 0.669 / 413.7ms。多 1.3–2.6 个点召回、MRR 降、慢 6–7 倍。 + - 开图片腿(CLIP 索引 53,668 张图、建库 2901.7s、512 维):10 个 atom **0.815** / MRR 0.553 / 132.5ms;页 30 **0.825** / 0.551 / 162.8ms。分模态才看得见它在干什么:**video 0.583→0.708、pdf 0.849→0.907 涨,code 0.953→0.898、ui_screenshot 0.583→0.500 跌**——它救的正是金标是图的题,代价是稀释别的。 + - 产品的图片腿不是开关:`retrieval.py:136` `if image is None and not thin: return {}`,稀疏页薄到 2 条以下才问图。所以这几行测的是「腿可用」,不是强行常开融合。 + - **没达到的验收线,写在这里**:用户定的验收是 recall@10 ≥0.90(n≥300)。测出来最好的一行是 **0.858**(dense 腿开、页 30、p50 414ms),默认配置 **0.848**(p50 72.5ms),CURRENT.md 里按 10 个 atom 问是 0.832;仓内 Mem0/Zep 式对照在 holder 作用域是 **0.866**。差距集中在图(image 0.750、ui_screenshot 0.500–0.583、video 0.583–0.708),文字的题(code 0.953、pdf 0.919)已经过了线。诚实结论:**延迟达标(281 倍于基线、p50 71.7ms),召回没达标**,剩下 4 个点是跨模态召回,不是索引规模问题。 + - 数字逼出来的决定:**dense 腿默认关**(`config.py:32 dense_embeddings_enabled=False`)。同一 300 条 holder 作用域探针上(LEGS.md),开腿 recall@10 0.832→0.845、MRR 0.683→0.663、p50 65.5ms→472.9ms。多 1.3 个点的召回、慢 7 倍、首位质量还降——默认值就该是关,腿留着(已增量、已入库)等更大的语料或更好的嵌入模型。 - 顺带修的测量缺陷:`copy_store()` 只拷 SQLite,所以任何要跑视觉腿的 harness 都得自己带上 `objects/`(25 GB,探针里用符号链接)与 `cache/clip_image_index.npy`;`bench/scale_check.py` 原本直接打开冻结基线库、把它**原地迁移**了,现改成在副本上测(`--scratch`)。 - [ ] **p2-2** 安全与隐私 - [ ] p2-2.1 FastAPI 鉴权 + Origin 校验 diff --git a/bench/reports/CURRENT.md b/bench/reports/CURRENT.md index 5307ffd..f58ba6b 100644 --- a/bench/reports/CURRENT.md +++ b/bench/reports/CURRENT.md @@ -1,21 +1,19 @@ # CURRENT: what the product retrieves at this build -- generated 2026-10-07T21:24:15+00:00 at commit dfc6cd4c2889 -- measured store schema: v5 +- generated 2026-10-08T20:17:12+00:00 at commit dff5977ab638 +- measured store schema: v8 - corpus 50278 sources, 5308 queries (2928 answerable) - models: dense `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, rerank `BAAI/bge-reranker-base` -- machine: 32 cores, 4 threads per query, load average 1.2 when the report was written +- machine: 32 cores, 4 threads per query, load average 5.1 when the report was written | arm | recall@1 | recall@5 | recall@10 | nDCG@10 | MRR | hit@10 (95% lower) | p50 s | p95 s | |---|---|---|---|---|---|---|---|---| | chance (random ranking) | 0.000 | 0.000 | 0.000 | - | - | - | - | - | - -_re-timed on a stride sample with warm models; recall comes from the full pass (n=600 queries)_ -| bm25_only | 0.313 | 0.465 | 0.519 | 0.415 | 0.383 | 0.524 (0.505) | 0.631 | 2.768 | -| dense_only | 0.227 | 0.363 | 0.424 | 0.321 | 0.290 | 0.429 (0.411) | 0.081 | 0.134 | -| graph_only | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.001 (0.000) | 0.055 | 0.095 | -| no_rerank | 0.342 | 0.510 | 0.565 | 0.452 | 0.418 | 0.571 (0.553) | 0.643 | 2.786 | -| fusion_rerank | 0.236 | 0.391 | 0.453 | 0.341 | 0.309 | 0.459 (0.441) | 7.000 | 13.247 | +| bm25_only | 0.313 | 0.465 | 0.519 | 0.415 | 0.383 | 0.524 (0.505) | 0.563 | 3.141 | +| dense_only | 0.227 | 0.363 | 0.424 | 0.321 | 0.290 | 0.429 (0.411) | 1.728 | 2.883 | +| graph_only | 0.000 | 0.001 | 0.001 | 0.000 | 0.000 | 0.001 (0.000) | 0.043 | 0.058 | +| no_rerank | 0.342 | 0.510 | 0.565 | 0.452 | 0.418 | 0.571 (0.553) | 2.404 | 5.169 | +| fusion_rerank | 0.236 | 0.391 | 0.453 | 0.341 | 0.309 | 0.459 (0.441) | 4.538 | 9.043 | ## recall@10 by modality @@ -35,24 +33,12 @@ _re-timed on a stride sample with warm models; recall comes from the full pass ( - no_rerank: abstains on 0% of unanswerable queries, and its top hit comes from the query's own holder for 40% of them; the rest are confident answers pulled out of somebody else's memory - fusion_rerank: abstains on 0% of unanswerable queries, and its top hit comes from the query's own holder for 34% of them; the rest are confident answers pulled out of somebody else's memory -## the unmodified product path +## the product path, asked as its holder -- RetrievalService.search: p50 0.68s, mean 1.08s, max 9.36s over 40 queries -- top-10 overlap with the fast BM25 path: 0.0%, identical order 0.0% -- its top-10 atoms cover only 7.7 distinct sources (min 1): the product ranks atoms and never diversifies, so ten slots fill with fragments of one document and the overlap with a source-level ranking is near zero - -## latency vs corpus size - -| sources | atoms | bm25_only p50/p95 s | dense_only p50/p95 s | -|---|---|---|---| -| 500 | 500 | 0.001/0.003 | 0.018/0.031 | -| 2000 | 2000 | 0.005/0.010 | 0.045/0.078 | -| 10000 | 10000 | 0.035/0.110 | 0.079/0.100 | -| 50000 | 165695 | 0.944/2.376 | 0.090/0.108 | - -_latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless_ - -_latency was measured at load average 1.2 on 32 cores; a busy machine inflates these figures, so read them as a ceiling_ +- RetrievalService.search as the query's own holder: recall@10 0.832, MRR 0.683, hit@10 0.833 (95% lower 0.787) over n=300 answerable queries +- latency p50 0.074s, p95 0.146s, mean 0.079s, max 0.314s +- top-10 overlap with the fast BM25 path: 13.4%, identical order 0.0% +- its top-10 atoms cover 5.2 distinct sources on average (min 0): the product ranks atoms and never diversifies, so ten slots can fill with fragments of one document ## reading diff --git a/bench/reports/DENSE.md b/bench/reports/DENSE.md new file mode 100644 index 0000000..bb8bec9 --- /dev/null +++ b/bench/reports/DENSE.md @@ -0,0 +1,23 @@ +# DENSE: the vector leg at corpus size (commit dff5977ab) + +- database: `bench/runs/dense_scale/data/ely_eye.sqlite` +- atoms: 165973, memories with words: 153857, vectors: 153857, pending: 0 +- model: `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, width 384 +- build: 1421.0s for 153857 memories (108 per second) +- writing one memory into a full index: p50 518ms · p95 543ms over 50 writes +- the repair scan (`ely-eye build-dense-index`) over a full index: 1.3s, and it is not on the search path +- search p50 75.6ms · p95 93.0ms · max 123.6ms over 300 queries +- of which encoding the question: p50 12.4ms · p95 25.8ms +- against a brute-force scan: 0.998 of the top 10 over 40 probes (a returned atom counts when its true cosine is at least the k-th best, so ties cannot fail the check) +- bytes per stored vector: 1729 (266059776 bytes for 153857 vectors, dbstat page sizes over the vec0 shadow tables) +- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [3.18, 4.58, 5.42] + +## One million memories on this machine + +- p50 455ms · p95 561ms +- vector table: 1.6 GiB +- first build: 153.9 minutes +- writing one memory: 3123ms — a vec0 delete by id scans the table, so this grows with the count too +- assumption: sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the count; the model, the machine and the thread budget stay the same + +_Latency is wall time per search on the machine named above with the thread budget the benchmark pins; a run on a loaded machine is not comparable._ diff --git a/bench/reports/LEGS.md b/bench/reports/LEGS.md new file mode 100644 index 0000000..1a1ed95 --- /dev/null +++ b/bench/reports/LEGS.md @@ -0,0 +1,37 @@ +# LEGS: what each retrieval leg is worth + +- commit: `dff5977ab638896cc8685c78ee41e5195fc0fbf2`, schema v8 +- store: bench/runs/leg_check/data/ely_eye.sqlite (a copy of bench/runs/dense_scale; the frozen baseline is never opened) +- 300 answerable queries, each asked as the holder the frozen set assigns it +- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [25.51, 9.62, 5.7] + +Recall is scored on the first ten **sources** after atoms are collapsed to their source +by first hit, which is how every other arm in this repo is scored. + +| leg configuration | atoms asked | recall@10 | MRR | hit@10 | 95% lower | p50 ms | p95 ms | distinct sources | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| words only | 10 | 0.832 | 0.683 | 0.833 | 0.787 | 65.5 | 134.9 | 4.0 | +| words only (the configured 24) | 24 | 0.848 | 0.688 | 0.850 | 0.805 | 72.5 | 155.9 | 6.0 | +| words only, wider page | 30 | 0.855 | 0.690 | 0.857 | 0.812 | 64.7 | 129.7 | 6.0 | +| + dense leg | 10 | 0.845 | 0.663 | 0.847 | 0.802 | 472.9 | 577.5 | 5.0 | +| + dense leg, wider page | 30 | 0.858 | 0.669 | 0.860 | 0.816 | 413.7 | 525.8 | 7.0 | +| + picture leg | 10 | 0.815 | 0.553 | 0.817 | 0.769 | 132.5 | 271.3 | 8.0 | +| + picture leg, wider page | 30 | 0.825 | 0.551 | 0.827 | 0.780 | 162.8 | 283.7 | 20.0 | + +## recall@10 by modality + +| leg configuration | code | image | pdf | ui_screenshot | video | +| --- | --- | --- | --- | --- | --- | +| words only | 0.953 | 0.750 | 0.849 | 0.583 | 0.583 | +| words only (the configured 24) | 0.953 | 0.750 | 0.895 | 0.583 | 0.625 | +| words only, wider page | 0.953 | 0.750 | 0.919 | 0.583 | 0.625 | +| + dense leg | 0.938 | 0.750 | 0.884 | 0.639 | 0.625 | +| + dense leg, wider page | 0.938 | 0.750 | 0.919 | 0.639 | 0.667 | +| + picture leg | 0.930 | 0.750 | 0.849 | 0.528 | 0.583 | +| + picture leg, wider page | 0.898 | 0.750 | 0.907 | 0.500 | 0.708 | + +_Query counts per modality: code 128, image 26, pdf 86, ui_screenshot 36, video 24._ + +The picture leg is not a switch this harness flips per row: the product runs it only when the sparse page comes back thin, so these rows measure the product with the leg available, not a forced always-on fusion. + +_Latency is wall time per search on the machine named above with the thread budget the benchmark pins; a run on a loaded machine is not comparable._ diff --git a/bench/reports/QUANT.md b/bench/reports/QUANT.md new file mode 100644 index 0000000..023493c --- /dev/null +++ b/bench/reports/QUANT.md @@ -0,0 +1,28 @@ +# QUANT: what each vector encoding buys (commit cd9851a12) + +- corpus: 165973 vectors, 384 dimensions, 200 probes, top 10 +- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [2.46, 4.17, 4.58] +- probes are drawn from the corpus itself, so these arms are compared with each other, not quoted as retrieval quality; a hit is any returned atom whose true cosine is at least the 10-th best, because byte-identical atoms make the id set ambiguous + +| arm | p50 ms | p95 ms | recall@10 | +| --- | --- | --- | --- | +| sqlite-vec float | 41.2 | 55.4 | 1.0 | +| sqlite-vec float + holder filter | 29.8 | 30.7 | 0.599 | +| sqlite-vec int8 | 35.5 | 42.2 | 0.977 | +| sqlite-vec int8 + holder filter | 22.3 | 28.1 | 0.595 | +| sqlite-vec bit | 2.6 | 2.8 | 0.728 | +| sqlite-vec bit + holder filter | 4.1 | 4.3 | 0.513 | +| numpy float32 scan (the truth) | 8.0 | 9.6 | 1.0 | +| numpy hamming (sign bit) | 18.9 | 19.8 | 0.011 | +| numpy float32 first 192 dims | 7.5 | 8.2 | 0.722 | +| numpy float32 first 128 dims | 7.9 | 8.9 | 0.57 | +| numpy float32 first 64 dims | 4.0 | 8.3 | 0.43 | +| bit coarse@200 -> float32 rerank in numpy | 23.7 | 24.7 | 0.983 | + +| encoding | bytes per vector | +| --- | --- | +| float32 | 1536 | +| int8 | 384 | +| bit | 48 | + +_Latency is wall time per query on the machine named above with the thread budget the benchmark pins; a run on a loaded machine is not comparable._ diff --git a/bench/reports/SCALE.md b/bench/reports/SCALE.md index 980201a..6746d70 100644 --- a/bench/reports/SCALE.md +++ b/bench/reports/SCALE.md @@ -1,10 +1,14 @@ # SCALE: the product's own search at corpus size -- database: bench/runs/baseline/data/ely_eye.sqlite +- commit: `dff5977ab638896cc8685c78ee41e5195fc0fbf2` +- database: bench/runs/scale/data/ely_eye.sqlite +- copied from the frozen baseline in 5.0s and migrated in 0.0s to schema 8; the baseline store itself is left untouched - atoms: 165973, probes: 300 (code, image, pdf, ui_screenshot, video) - dense embeddings: False -- opening the store (migrations included): 0.000s -- p50 0.0570s · p95 0.1383s · max 0.5395s over 300 queries -- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [1.77, 1.62, 1.64] +- p50 0.0717s · p95 0.2177s · max 0.7805s over 300 queries +- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [2.31, 3.72, 7.9] + +The baseline recorded p50 20.16s on this corpus because the old search pulled every +atom into Python and rebuilt a BM25 index per query. _Latency is wall time per search call on an idle machine with the thread budget the benchmark pins; a run on a loaded machine is not comparable._ diff --git a/bench/reports/TERMS.md b/bench/reports/TERMS.md new file mode 100644 index 0000000..2b4dd43 --- /dev/null +++ b/bench/reports/TERMS.md @@ -0,0 +1,20 @@ +# TERMS: which query words are worth paying for + +- commit: `dff5977ab638896cc8685c78ee41e5195fc0fbf2` +- store: bench/runs/term_sweep/data/ely_eye.sqlite (copied from the frozen baseline in 5.1s) +- 173 answerable probes, each asked as the holder the frozen set assigns it, ranked by the unmodified `RetrievalService` +- the bound the product ships is 0.05; the measured recall peak sits at 0.05 +- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [2.32, 3.87, 8.09] + +| bound | recall@10 | MRR | hit@10 | p50 ms | p95 ms | median distinct sources | +| --- | --- | --- | --- | --- | --- | --- | +| 0.200 | 0.792 | 0.629 | 0.792 | 420.3 | 1704.1 | 6 | +| 0.100 | 0.792 | 0.632 | 0.792 | 467.5 | 1747.5 | 6 | +| 0.050 **(shipped)** | 0.803 | 0.652 | 0.803 | 69.5 | 131.5 | 5 | +| 0.020 | 0.792 | 0.631 | 0.792 | 33.4 | 88.4 | 4 | +| 0.010 | 0.775 | 0.617 | 0.775 | 19.2 | 63.0 | 4 | +| 0.005 | 0.740 | 0.563 | 0.740 | 10.8 | 69.9 | 3 | + +The bound is a share of the library: a word carried by more atoms than that is dropped from the OR query, because bm25 scores every row the query touches. It is relative to the corpus on purpose — a corpus of PDFs and a corpus of shell history disagree about which of their own words are noise. + +_Latency is wall time per search on the machine named above with the thread budget the benchmark pins; a run on a loaded machine is not comparable._ diff --git a/bench/reports/current.json b/bench/reports/current.json index d6f5f6d..5b3ac62 100644 --- a/bench/reports/current.json +++ b/bench/reports/current.json @@ -89,12 +89,11 @@ } }, "latency_seconds": { - "max": 10.484337914007483, - "mean": 0.9426860555750318, - "note": "re-timed on a stride sample with warm models; recall comes from the full pass", - "p50": 0.63090720949549, - "p95": 2.7677196743439674, - "queries": 600 + "max": 70.10108924399901, + "mean": 1.033756499582343, + "p50": 0.5625866459999997, + "p95": 3.1407018873005645, + "queries": 5308 }, "overall": { "abstention_rate_answerable": 0.0, @@ -200,12 +199,11 @@ } }, "latency_seconds": { - "max": 1.5199895569967339, - "mean": 0.09428343026663545, - "note": "re-timed on a stride sample with warm models; recall comes from the full pass", - "p50": 0.08063396950092283, - "p95": 0.13402395970297218, - "queries": 600 + "max": 5.211815150996699, + "mean": 1.90850240464544, + "p50": 1.7281245969993506, + "p95": 2.8826986088994087, + "queries": 5308 }, "overall": { "abstention_rate_answerable": 0.0, @@ -311,12 +309,11 @@ } }, "latency_seconds": { - "max": 0.11273866199917393, - "mean": 0.06529701035842057, - "note": "re-timed on a stride sample with warm models; recall comes from the full pass", - "p50": 0.05544837899287813, - "p95": 0.09468995315801294, - "queries": 600 + "max": 0.09835406600177521, + "mean": 0.044743228177486605, + "p50": 0.04287036849927972, + "p95": 0.05790443480054818, + "queries": 5308 }, "overall": { "abstention_rate_answerable": 0.0, @@ -422,12 +419,11 @@ } }, "latency_seconds": { - "max": 10.533955075996346, - "mean": 0.9619427272766673, - "note": "re-timed on a stride sample with warm models; recall comes from the full pass", - "p50": 0.6425201155034301, - "p95": 2.786401089788338, - "queries": 600 + "max": 72.48293514099714, + "mean": 2.882459499187649, + "p50": 2.40399285599824, + "p95": 5.168583710499833, + "queries": 5308 }, "overall": { "abstention_rate_answerable": 0.0, @@ -533,12 +529,11 @@ } }, "latency_seconds": { - "max": 28.03755501300475, - "mean": 7.456139394386567, - "note": "re-timed on a stride sample with warm models; recall comes from the full pass", - "p50": 7.000474203505291, - "p95": 13.246865720593268, - "queries": 600 + "max": 77.63319287999548, + "mean": 5.166508504176891, + "p50": 4.5375522020012795, + "p95": 9.042528758149817, + "queries": 5308 }, "overall": { "abstention_rate_answerable": 0.0, @@ -563,32 +558,45 @@ "recall_at_5": 9.944707426707506e-05 }, "live_product": { + "abstention_rate_answerable": 0.016666666666666666, + "abstention_rate_unanswerable": 0.0, "distinct_sources_in_top10": { "max": 10, - "mean": 7.675, - "min": 1 + "mean": 5.193333333333333, + "min": 0 }, + "hit_at_10": 0.8333333333333334, + "hit_at_10_wilson_lower": 0.7870033503873167, "identical_top10": 0.0, - "max": 9.363575160998153, - "mean": 1.0766488450000906, - "p50": 0.6765933140050038, - "queries": 40, - "top10_overlap_with_bm25_only": 0.0, + "max": 0.3143954080005642, + "mean": 0.07851241317994816, + "mrr": 0.6832447089947089, + "n_answerable": 300, + "n_unanswerable": 0, + "ndcg_at_10": 0.7186678703708246, + "p50": 0.07425041199894622, + "p95": 0.14573381999798585, + "queries": 300, + "recall_at_1": 0.5966666666666667, + "recall_at_10": 0.8316666666666667, + "recall_at_5": 0.7816666666666666, + "top10_overlap_with_bm25_only": 0.13433333333333336, + "top_hit_from_holder_unanswerable": 0.0, "top_k": 10 }, "meta": { "answerable": 2928, "corpus_manifest_sha256": "8fc1ed861c891d778853d41107630869f9eb267a669c5d1038e389de80114347", "corpus_sources": 50278, - "generated_at": "2026-10-07T21:24:15+00:00", - "git_commit": "dfc6cd4c2889e872d3f7d983b18406e6ff7a622d", + "generated_at": "2026-10-08T20:17:12+00:00", + "git_commit": "dff5977ab638896cc8685c78ee41e5195fc0fbf2", "limit_per_modality": null, "machine": { "cpus": 32, "load_average_1_5_15": [ - 1.21, - 1.55, - 2.28 + 5.09, + 4.28, + 8.06 ], "release": "6.18.40.1-microsoft-standard-WSL2", "system": "Linux", @@ -608,64 +616,6 @@ "ui_screenshot": "2269dd623e67cd851fb3f6b24a4556a8462a37db98e12ad79577c5830167c7c5", "video": "4973231d9f16d214b14923d79a34b85c3d2b423624dea752b1ee91ba09ff778c" }, - "store_schema_version_at_start": 5 - }, - "scale": [ - { - "atoms": 500, - "bm25_only": { - "p50": 0.001200532500661211, - "p95": 0.0033308441004919584 - }, - "dense_only": { - "p50": 0.018240716999571305, - "p95": 0.03141984814501483 - }, - "note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless", - "queries": 40, - "sources": 500 - }, - { - "atoms": 2000, - "bm25_only": { - "p50": 0.0047556615027133375, - "p95": 0.009782735643602785 - }, - "dense_only": { - "p50": 0.044618372499826364, - "p95": 0.07758056885431869 - }, - "note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless", - "queries": 40, - "sources": 2000 - }, - { - "atoms": 10000, - "bm25_only": { - "p50": 0.03480406300150207, - "p95": 0.11021578034851681 - }, - "dense_only": { - "p50": 0.07940824350225739, - "p95": 0.09990645345169469 - }, - "note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless", - "queries": 40, - "sources": 10000 - }, - { - "atoms": 165695, - "bm25_only": { - "p50": 0.9441751839985955, - "p95": 2.3757928041479315 - }, - "dense_only": { - "p50": 0.08988298100302927, - "p95": 0.10847034014877863 - }, - "note": "latency only: a source prefix drops most gold sources, so recall at these sizes is meaningless", - "queries": 40, - "sources": 50000 - } - ] + "store_schema_version_at_start": 8 + } } \ No newline at end of file diff --git a/bench/reports/dense.json b/bench/reports/dense.json new file mode 100644 index 0000000..02fb131 --- /dev/null +++ b/bench/reports/dense.json @@ -0,0 +1,73 @@ +{ + "commit": "dff5977ab638896cc8685c78ee41e5195fc0fbf2", + "database": "bench/runs/dense_scale/data/ely_eye.sqlite", + "atoms": 165973, + "memories_with_words": 153857, + "vectors": 153857, + "pending": 0, + "dim": 384, + "model": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2", + "copy_seconds": 0.665644496002642, + "build_seconds": 1421.0276969089973, + "texts_per_second": 108.2716405420302, + "repair_seconds": 1.2946931579936063, + "writes": 50, + "write_latency": { + "queries": 50, + "mean": 0.5204645371803781, + "p50": 0.5183807220018934, + "p95": 0.5429989652060613, + "max": 0.5760761429992272 + }, + "probes": 300, + "probe_modalities": [ + "code", + "image", + "pdf", + "ui_screenshot", + "video" + ], + "latency": { + "queries": 300, + "mean": 0.07861847096656371, + "p50": 0.07557305500085931, + "p95": 0.09303550984986941, + "max": 0.123588470989489 + }, + "encode_latency": { + "queries": 300, + "mean": 0.014255492710411393, + "p50": 0.012439899503078777, + "p95": 0.025788354348333098, + "max": 0.0513256230042316 + }, + "correctness": { + "probes": 40, + "top_k": 10, + "tie_aware_agreement": 0.9975, + "note": "a returned atom counts when its true cosine is at least the k-th best, so ties cannot fail the check" + }, + "bytes_per_vector": 1729.2666307025354, + "vector_bytes": 266059776, + "bytes_method": "dbstat page sizes over the vec0 shadow tables", + "extrapolation": { + "to": 1000000, + "assumption": "sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the count; the model, the machine and the thread budget stay the same", + "p50_seconds": 0.45533342773137386, + "p95_seconds": 0.5605460517666694, + "write_p50_seconds": 3.1232834376789804, + "gibibytes": 1.6105050507025191, + "build_seconds": 9236.028889871746 + }, + "machine": { + "system": "Linux", + "release": "6.18.40.1-microsoft-standard-WSL2", + "cpus": 32, + "threads_per_query": 4, + "load_average_1_5_15": [ + 3.18, + 4.58, + 5.42 + ] + } +} \ No newline at end of file diff --git a/bench/reports/leg_check.json b/bench/reports/leg_check.json new file mode 100644 index 0000000..6f00ead --- /dev/null +++ b/bench/reports/leg_check.json @@ -0,0 +1,208 @@ +{ + "commit": "dff5977ab638896cc8685c78ee41e5195fc0fbf2", + "schema_version": 8, + "store": "bench/runs/leg_check/data/ely_eye.sqlite", + "probes": 300, + "arms": [ + { + "label": "words only", + "top_k": 10, + "dense": false, + "visual": false, + "recall_at_10": 0.8316666666666667, + "mrr": 0.6832447089947089, + "hit_at_10": 0.8333333333333334, + "wilson_lower": 0.7870033503873167, + "p50_seconds": 0.0654734540003119, + "p95_seconds": 0.13490134734311146, + "median_distinct_sources": 4.0, + "by_modality": { + "code": 0.953125, + "image": 0.75, + "pdf": 0.8488372093023255, + "ui_screenshot": 0.5833333333333334, + "video": 0.5833333333333334 + }, + "modality_counts": { + "code": 128, + "image": 26, + "pdf": 86, + "ui_screenshot": 36, + "video": 24 + } + }, + { + "label": "words only (the configured 24)", + "top_k": 24, + "dense": false, + "visual": false, + "recall_at_10": 0.8483333333333334, + "mrr": 0.6875571326821327, + "hit_at_10": 0.85, + "wilson_lower": 0.8051823888546709, + "p50_seconds": 0.07251374649786158, + "p95_seconds": 0.15588207065375176, + "median_distinct_sources": 6.0, + "by_modality": { + "code": 0.953125, + "image": 0.75, + "pdf": 0.8953488372093024, + "ui_screenshot": 0.5833333333333334, + "video": 0.625 + }, + "modality_counts": { + "code": 128, + "image": 26, + "pdf": 86, + "ui_screenshot": 36, + "video": 24 + } + }, + { + "label": "words only, wider page", + "top_k": 30, + "dense": false, + "visual": false, + "recall_at_10": 0.855, + "mrr": 0.6895710215710216, + "hit_at_10": 0.8566666666666667, + "wilson_lower": 0.8124994228503402, + "p50_seconds": 0.06468957299512113, + "p95_seconds": 0.1297007056004077, + "median_distinct_sources": 6.0, + "by_modality": { + "code": 0.953125, + "image": 0.75, + "pdf": 0.9186046511627907, + "ui_screenshot": 0.5833333333333334, + "video": 0.625 + }, + "modality_counts": { + "code": 128, + "image": 26, + "pdf": 86, + "ui_screenshot": 36, + "video": 24 + } + }, + { + "label": "+ dense leg", + "top_k": 10, + "dense": true, + "visual": false, + "recall_at_10": 0.845, + "mrr": 0.6632195767195768, + "hit_at_10": 0.8466666666666667, + "wilson_lower": 0.8015339429418983, + "p50_seconds": 0.4729458744986914, + "p95_seconds": 0.5775450093562541, + "median_distinct_sources": 5.0, + "by_modality": { + "code": 0.9375, + "image": 0.75, + "pdf": 0.8837209302325582, + "ui_screenshot": 0.6388888888888888, + "video": 0.625 + }, + "modality_counts": { + "code": 128, + "image": 26, + "pdf": 86, + "ui_screenshot": 36, + "video": 24 + } + }, + { + "label": "+ dense leg, wider page", + "top_k": 30, + "dense": true, + "visual": false, + "recall_at_10": 0.8583333333333333, + "mrr": 0.6686637344137344, + "hit_at_10": 0.86, + "wilson_lower": 0.8161683876490509, + "p50_seconds": 0.41367531901050825, + "p95_seconds": 0.5258364057008293, + "median_distinct_sources": 7.0, + "by_modality": { + "code": 0.9375, + "image": 0.75, + "pdf": 0.9186046511627907, + "ui_screenshot": 0.6388888888888888, + "video": 0.6666666666666666 + }, + "modality_counts": { + "code": 128, + "image": 26, + "pdf": 86, + "ui_screenshot": 36, + "video": 24 + } + }, + { + "label": "+ picture leg", + "top_k": 10, + "dense": false, + "visual": true, + "recall_at_10": 0.815, + "mrr": 0.5525886243386243, + "hit_at_10": 0.8166666666666667, + "wilson_lower": 0.7689713823226987, + "p50_seconds": 0.13248339149868116, + "p95_seconds": 0.2713221473451995, + "median_distinct_sources": 8.0, + "by_modality": { + "code": 0.9296875, + "image": 0.75, + "pdf": 0.8488372093023255, + "ui_screenshot": 0.5277777777777778, + "video": 0.5833333333333334 + }, + "modality_counts": { + "code": 128, + "image": 26, + "pdf": 86, + "ui_screenshot": 36, + "video": 24 + } + }, + { + "label": "+ picture leg, wider page", + "top_k": 30, + "dense": false, + "visual": true, + "recall_at_10": 0.825, + "mrr": 0.5507809648412589, + "hit_at_10": 0.8266666666666667, + "wilson_lower": 0.7797738592974309, + "p50_seconds": 0.16283495400421089, + "p95_seconds": 0.2837207488548302, + "median_distinct_sources": 20.0, + "by_modality": { + "code": 0.8984375, + "image": 0.75, + "pdf": 0.9069767441860465, + "ui_screenshot": 0.5, + "video": 0.7083333333333334 + }, + "modality_counts": { + "code": 128, + "image": 26, + "pdf": 86, + "ui_screenshot": 36, + "video": 24 + } + } + ], + "machine": { + "system": "Linux", + "release": "6.18.40.1-microsoft-standard-WSL2", + "cpus": 32, + "threads_per_query": 4, + "load_average_1_5_15": [ + 25.51, + 9.62, + 5.7 + ] + } +} \ No newline at end of file diff --git a/bench/reports/quant.json b/bench/reports/quant.json new file mode 100644 index 0000000..3abd721 --- /dev/null +++ b/bench/reports/quant.json @@ -0,0 +1,110 @@ +{ + "commit": "cd9851a12c000199ce5c211f937ef6c1e0138efa", + "vectors": 165973, + "dimensions": 384, + "probes": 200, + "top_k": 10, + "recall_note": "probes are drawn from the corpus itself, so these arms are compared with each other, not quoted as retrieval quality; a hit is any returned atom whose true cosine is at least the 10-th best, because byte-identical atoms make the id set ambiguous", + "bytes_per_vector": { + "float32": 1536, + "int8": 384, + "bit": 48 + }, + "arms": [ + { + "arm": "sqlite-vec float", + "queries": 200, + "p50_ms": 41.2, + "p95_ms": 55.4, + "recall_at_10": 1.0 + }, + { + "arm": "sqlite-vec float + holder filter", + "queries": 200, + "p50_ms": 29.8, + "p95_ms": 30.7, + "recall_at_10": 0.599 + }, + { + "arm": "sqlite-vec int8", + "queries": 200, + "p50_ms": 35.5, + "p95_ms": 42.2, + "recall_at_10": 0.977 + }, + { + "arm": "sqlite-vec int8 + holder filter", + "queries": 200, + "p50_ms": 22.3, + "p95_ms": 28.1, + "recall_at_10": 0.595 + }, + { + "arm": "sqlite-vec bit", + "queries": 200, + "p50_ms": 2.6, + "p95_ms": 2.8, + "recall_at_10": 0.728 + }, + { + "arm": "sqlite-vec bit + holder filter", + "queries": 200, + "p50_ms": 4.1, + "p95_ms": 4.3, + "recall_at_10": 0.513 + }, + { + "arm": "numpy float32 scan (the truth)", + "queries": 200, + "p50_ms": 8.0, + "p95_ms": 9.6, + "recall_at_10": 1.0 + }, + { + "arm": "numpy hamming (sign bit)", + "queries": 200, + "p50_ms": 18.9, + "p95_ms": 19.8, + "recall_at_10": 0.011 + }, + { + "arm": "numpy float32 first 192 dims", + "queries": 50, + "p50_ms": 7.5, + "p95_ms": 8.2, + "recall_at_10": 0.722 + }, + { + "arm": "numpy float32 first 128 dims", + "queries": 50, + "p50_ms": 7.9, + "p95_ms": 8.9, + "recall_at_10": 0.57 + }, + { + "arm": "numpy float32 first 64 dims", + "queries": 50, + "p50_ms": 4.0, + "p95_ms": 8.3, + "recall_at_10": 0.43 + }, + { + "arm": "bit coarse@200 -> float32 rerank in numpy", + "queries": 200, + "p50_ms": 23.7, + "p95_ms": 24.7, + "recall_at_10": 0.983 + } + ], + "machine": { + "system": "Linux", + "release": "6.18.40.1-microsoft-standard-WSL2", + "cpus": 32, + "threads_per_query": 4, + "load_average_1_5_15": [ + 2.46, + 4.17, + 4.58 + ] + } +} \ No newline at end of file diff --git a/bench/reports/scale.json b/bench/reports/scale.json index 16a7c0b..49f29c9 100644 --- a/bench/reports/scale.json +++ b/bench/reports/scale.json @@ -1,7 +1,10 @@ { - "database": "bench/runs/baseline/data/ely_eye.sqlite", + "commit": "dff5977ab638896cc8685c78ee41e5195fc0fbf2", + "database": "bench/runs/scale/data/ely_eye.sqlite", "atoms": 165973, - "migration_seconds": 0.00044073900062358007, + "copy_seconds": 4.996028091998596, + "schema_version": 8, + "migration_seconds": 0.03624351899634348, "dense_embeddings_enabled": false, "probes": 300, "probe_modalities": [ @@ -13,10 +16,10 @@ ], "latency": { "queries": 300, - "mean": 0.06524801251696771, - "p50": 0.057020749500225065, - "p95": 0.13832052519683205, - "max": 0.5395459059946006 + "mean": 0.0888331904130852, + "p50": 0.07169728449662216, + "p95": 0.21773518459958724, + "max": 0.7804536300027394 }, "machine": { "system": "Linux", @@ -24,9 +27,9 @@ "cpus": 32, "threads_per_query": 4, "load_average_1_5_15": [ - 1.77, - 1.62, - 1.64 + 2.31, + 3.72, + 7.9 ] } } \ No newline at end of file diff --git a/bench/reports/term_sweep.json b/bench/reports/term_sweep.json new file mode 100644 index 0000000..e9f07dd --- /dev/null +++ b/bench/reports/term_sweep.json @@ -0,0 +1,81 @@ +{ + "commit": "dff5977ab638896cc8685c78ee41e5195fc0fbf2", + "store": "bench/runs/term_sweep/data/ely_eye.sqlite", + "copy_seconds": 5.062254256001324, + "probes": 173, + "shipped_fraction": 0.05, + "peak_fraction": 0.05, + "arms": [ + { + "fraction": 0.2, + "shipped": false, + "recall_at_10": 0.791907514450867, + "mrr": 0.6286195981282686, + "hit_at_10": 0.791907514450867, + "p50_seconds": 0.4202946660007001, + "p95_seconds": 1.7041233958021622, + "median_distinct_sources": 6 + }, + { + "fraction": 0.1, + "shipped": false, + "recall_at_10": 0.791907514450867, + "mrr": 0.6315097715386733, + "hit_at_10": 0.791907514450867, + "p50_seconds": 0.46752902700245613, + "p95_seconds": 1.7474705398024528, + "median_distinct_sources": 6 + }, + { + "fraction": 0.05, + "shipped": true, + "recall_at_10": 0.8034682080924855, + "mrr": 0.6520300027525461, + "hit_at_10": 0.8034682080924855, + "p50_seconds": 0.069478709003306, + "p95_seconds": 0.13147423019836424, + "median_distinct_sources": 5 + }, + { + "fraction": 0.02, + "shipped": false, + "recall_at_10": 0.791907514450867, + "mrr": 0.6311244150839527, + "hit_at_10": 0.791907514450867, + "p50_seconds": 0.03335734199936269, + "p95_seconds": 0.08840757200086956, + "median_distinct_sources": 4 + }, + { + "fraction": 0.01, + "shipped": false, + "recall_at_10": 0.7745664739884393, + "mrr": 0.61696256537297, + "hit_at_10": 0.7745664739884393, + "p50_seconds": 0.019158394003170542, + "p95_seconds": 0.06302069279918217, + "median_distinct_sources": 4 + }, + { + "fraction": 0.005, + "shipped": false, + "recall_at_10": 0.7398843930635838, + "mrr": 0.5633659051289108, + "hit_at_10": 0.7398843930635838, + "p50_seconds": 0.01079572799790185, + "p95_seconds": 0.06986973820021376, + "median_distinct_sources": 3 + } + ], + "machine": { + "system": "Linux", + "release": "6.18.40.1-microsoft-standard-WSL2", + "cpus": 32, + "threads_per_query": 4, + "load_average_1_5_15": [ + 2.32, + 3.87, + 8.09 + ] + } +} \ No newline at end of file