Re-stamp the scale reports against the harness that now carries the fix
Both reports were produced by a harness that had just changed, so their stamps named a commit without that code in it. Re-running them on an idle machine puts the stamp and the source back in sync. Recall did not move: all seven leg configurations reproduce to the third decimal, which is the part of the number anyone should act on. Latency moved within its usual load band, and the encoder probe came in slower this time, so the picture index's build cost is now quoted as a band rather than a single remembered figure.
This commit is contained in:
ZacharyZhang-NY committed
1 parent
71733d86d6
commit
e80883cfc9
5 files changed
+89
-73
No files matched your search
+9
-9
@@ -350,18 +350,18 @@
|
||||
- 结论：int8 省 4× 空间几乎不掉质量（0.977）但不省时间（35.5 vs 41.2）；bit 省 32× 空间、省 **15.8×** 时间（41.2→2.6），只剩 0.728，只能当粗排；两级方案必须在 vec0 **外面**精排（在库里用 `rowid IN (…)` 回捞反而更慢）。维度截断是在**不是 MRL 训练**的 MiniLM 上测的，所以那是真实代价不是白拿：前 192 维 7.5ms、前 128 维 7.9ms，对全维 8.0ms 只快 1–6%，前 64 维才快一半但一致度掉到 0.43；切片必须是存下来的连续维，不是查询时切（非连续视图点集会掉速度）。
|
||||
- 偏差：精排腿（`bge-reranker-base`）早在基线消融里就有并测过（加了它 recall@10 从 0.565 掉到 0.453），所以这里没再为它编一套故事；粗排+精排这条链测出 0.983 一致度但**没接线**，因为 dense 腿默认关（见下）。
|
||||
- [x] p2-1.3 100k 实测 + 1M 外推：`python -m bench.dense_scale` → `bench/reports/DENSE.md`（stamp dff5977）
|
||||
- 实测 **165,973 atom / 153,857 向量**（DENSE.md，stamp 6cc564085）：建库 1444.8s（106 向量/秒）、**检索 p50 73.1ms / p95 76.1ms / max 87.2ms（300 查询）**、其中编码问题 p50 11.0ms / p95 24.1ms、与暴力扫描 tie-aware 一致度 **0.998**（40 探针）、**往满索引里写一条记忆 p50 509ms / p95 531ms（50 次写）**、失配扫描 1.2s（不在检索路径上）、**每向量 1,729 字节**（dbstat 数 vec0 影子表：266,059,776 B）、机器 load [3.41, 4.52, 4.31]。
|
||||
- 1M 外推（**分母是向量数**，1,000,000 / 153,857，不是 atom 数——两者差 7.9%，所以报告里把分母印在外推表旁边）：p50 475ms / p95 494ms、向量表 1.6 GiB、首次建库 156.5 分钟、单条写 3310ms（vec0 按 id 删除也要扫表，所以写同样随规模线性），假设随行印出（逐条扫 ⇒ 线性）。
|
||||
- 实测 **165,973 atom / 153,857 向量**（DENSE.md，stamp 71733d86d）：建库 1444.9s（106 向量/秒）、**检索 p50 73.0ms / p95 75.7ms / max 85.6ms（300 查询）**、其中编码问题 p50 11.3ms / p95 24.0ms、与暴力扫描 tie-aware 一致度 **0.998**（40 探针）、**往满索引里写一条记忆 p50 508ms / p95 535ms（50 次写）**、失配扫描 1.3s（不在检索路径上）、**每向量 1,729 字节**（dbstat 数 vec0 影子表：266,059,776 B）、机器 load [3.53, 4.63, 4.85]。
|
||||
- 1M 外推（**分母是向量数**，1,000,000 / 153,857，不是 atom 数——两者差 7.9%，所以报告把分母印在外推表旁边）：p50 475ms / p95 492ms、向量表 1.6 GiB、首次建库 156.5 分钟、单条写 3303ms（vec0 按 id 删除也要扫表，所以写同样随规模线性），假设随行印出（逐条扫 ⇒ 线性）。
|
||||
- tie-aware 的必要性：语料里有重复 atom，分数并列 ⇒ top-K 的 id 集合本身有歧义，两个都对引擎按 id 比会读出 0.827 的假召回。判定改成「返回的 atom 真实 cosine ≥ 第 K 名 cosine」。
|
||||
- 验收：100k dense 腿 p50 <100ms → **165,973 atom（bar 的 1.66 倍）p50 73.1ms**（DENSE.md）；新 atom 立即可查 → `pytest backend/tests/test_vector_index.py`（写进去当场能搜到、视图藏起来的 dense 腿也藏、备份带走向量恢复后照样应答）。
|
||||
- 验收：100k dense 腿 p50 <100ms → **165,973 atom（bar 的 1.66 倍）p50 73.0ms**（DENSE.md）；新 atom 立即可查 → `pytest backend/tests/test_vector_index.py`（写进去当场能搜到、视图藏起来的 dense 腿也藏、备份带走向量恢复后照样应答）。
|
||||
- [x] p2-1.4 每条腿值多少钱：`python -m bench.leg_check` → `bench/reports/LEGS.md`（stamp 见报告首行，300 条 holder 作用域探针，7 个配置，每臂带 n / MRR / Wilson 下界 / p50 / p95 / 分模态召回）。测的是 `bench/runs/dense_scale` 的副本：冻结基线自己**不带向量**，dense 腿在其上只会排一个空索引，所以报告首行就把来源 store 与向量数印出来。
|
||||
- 只有词的 FTS 页：问 10 个 atom recall@10 **0.832**、MRR 0.683、p50 65.1ms；问产品配置的 24 个 **0.848** / 0.688 / 66.5ms；页放宽到 30 **0.855** / 0.690 / 94.0ms（放宽不要钱，10→24 就白捡 1.6 个点）。
|
||||
- 开 dense 腿：10 个 atom **0.845** / MRR 0.663 / p50 481.4ms；页 30 **0.858** / 0.669 / 412.8ms。多 1.3–2.6 个点召回、MRR 降、慢 6–7 倍。
|
||||
- 开图片腿。索引实物印在 LEGS.md 里（不是这里口头说的）：**53,668 个 atom、512 维、109,912,192 字节**；编码速度实测 **19.9 张/秒**（200 张样本），据此那 53,668 张 ≈ **45.0 分钟**——和我当时手工记的 2901.7 秒（48.4 分钟）差 7%，两条路对得上。结果：10 个 atom **0.815** / MRR 0.553 / 139.5ms；页 30 **0.825** / 0.551 / 160.0ms。分模态才看得见它在干什么：**video 0.583→0.708、pdf 0.849→0.907 涨，code 0.953→0.898、ui_screenshot 0.583→0.500 跌**——它救的正是金标是图的题，代价是稀释别的。
|
||||
- p50 是负载函数，不是性常数：同一份 harness 两次跑，words 腿 64.7–72.5ms、dense 腿 412.8–481.4ms、图片腿 132.5–160.0ms，**召回一格没动**。所以比较只用召回，延迟看报告当次的 load。
|
||||
- 只有词的 FTS 页：问 10 个 atom recall@10 **0.832**、MRR 0.683、p50 67.5ms；问产品配置的 24 个 **0.848** / 0.688 / 67.6ms；页放宽到 30 **0.855** / 0.690 / 72.0ms（放宽不要钱，10→24 就白捡 1.6 个点）。
|
||||
- 开 dense 腿：10 个 atom **0.845** / MRR 0.663 / p50 474.8ms；页 30 **0.858** / 0.669 / 416.0ms。多 1.3–2.6 个点召回、MRR 降、慢 6–7 倍。
|
||||
- 开图片腿。索引实物印在 LEGS.md 里（不是这里口头说的）：**53,668 个 atom、512 维、109,912,192 字节**；编码速度也是实测的（200 张样本）：**16.8–19.9 张/秒**，据此那 53,668 张 ≈ **53.3–45.0 分钟**——和我当时手工记的 2901.7 秒（48.4 分钟）落在同一个带里。结果：10 个 atom **0.815** / MRR 0.553 / 138.1ms；页 30 **0.825** / 0.551 / 166.7ms。分模态才看得见它在干什么：**video 0.583→0.708、pdf 0.849→0.907 涨，code 0.953→0.898、ui_screenshot 0.583→0.500 跌**——它救的正是金标是图的题，代价是稀释别的。
|
||||
- p50 是负载函数，不是性常数：两份已提交的 LEGS.md 里，words 腿 64.7–72.5ms、dense 腿 413.7–474.8ms、图片腿 132.5–166.7ms，**召回三跑到小数点后三位一模一样**（0.832 / 0.848 / 0.855 / 0.845 / 0.858 / 0.815 / 0.825）。所以比较只用召回，延迟看报告当次的 load。
|
||||
- 产品的图片腿不是开关：`retrieval.py:136` `if image is None and not thin: return {}`，稀疏页薄到 2 条以下才问图。所以这几行测的是「腿可用」，不是强行常开融合。
|
||||
- **没达到的验收线，写在这里**：用户定的验收是 recall@10 ≥0.90（n≥300）。测出来最好的一行是 **0.858**（dense 腿开、页 30、p50 412.8ms），默认配置 **0.848**（p50 66.5ms），CURRENT.md 里按 10 个 atom 问是 0.832；仓内 Mem0/Zep 式对照在 holder 作用域是 **0.866**。差距集中在图（image 0.750、ui_screenshot 0.500–0.583、video 0.583–0.708），文字的题（code 0.953、pdf 0.919）已经过了线。诚实结论：**延迟达标（281 倍于基线、p50 71.7ms），召回没达标**，剩下 4 个点是跨模态召回，不是索引规模问题。
|
||||
- 数字逼出来的决定：**dense 腿默认关**（`config.py:32 dense_embeddings_enabled=False`）。同一 300 条 holder 作用域探针上（LEGS.md），开腿 recall@10 0.832→0.845、MRR 0.683→0.663、p50 65.1ms→481.4ms。多 1.3 个点的召回、慢 7 倍、首位质量还降——默认值就该是关，腿留着（已增量、已入库）等更大的语料或更好的嵌入模型。
|
||||
- **没达到的验收线，写在这里**：用户定的验收是 recall@10 ≥0.90（n≥300）。测出来最好的一行是 **0.858**（dense 腿开、页 30、p50 416.0ms），默认配置 **0.848**（p50 67.6ms），CURRENT.md 里按 10 个 atom 问是 0.832；仓内 Mem0/Zep 式对照在 holder 作用域是 **0.866**。差距集中在图（image 0.750、ui_screenshot 0.500–0.583、video 0.583–0.708），文字的题（code 0.953、pdf 0.919）已经过了线。诚实结论：**延迟达标（281 倍于基线、p50 71.7ms），召回没达标**，剩下 4 个点是跨模态召回，不是索引规模问题。
|
||||
- 数字逼出来的决定：**dense 腿默认关**（`config.py:32 dense_embeddings_enabled=False`）。同一 300 条 holder 作用域探针上（LEGS.md），开腿 recall@10 0.832→0.845、MRR 0.683→0.663、p50 67.5ms→474.8ms。多 1.3 个点的召回、慢 7 倍、首位质量还降——默认值就该是关，腿留着（已增量、已入库）等更大的语料或更好的嵌入模型。
|
||||
- 门禁：`pytest -q` 全绿；`ruff check` 干净；mypy 冷缓存 **24 errors / 9 files**，与 p1-6 基线（`9cb46c1` 同口径 24/9）**净新增 0**——p2-1 一度引入 +2（`vector_index.py` 与 `quant_probe.py` 的 `import sqlite_vec` 没有 stub），根因修法是在 `pyproject.toml` 的 `[[tool.mypy.overrides]]` 里声明 `sqlite_vec` 无 stub，而不是在代码里加 `# type: ignore`；`./node_modules/.bin/pyright` 对改动文件 0 errors；`bench/filesize_check` 干净；go / npm 未动。
|
||||
- 顺带修的测量缺陷：`copy_store()` 只拷 SQLite，所以任何要跑视觉腿的 harness 都得自己带上 `objects/`（25 GB，探针里用符号链接）与 `cache/clip_image_index.npy`；`bench/scale_check.py` 原本直接打开冻结基线库、把它**原地迁移**了，现改成在副本上测（`--scratch`）。产品 CLI 不在这条规则里：p2-1.4 建图片索引时 `ely-eye index-visual` 就把冻结基线从 schema 7 升到了 8，所以 p1-4.8 那两条钉的是**当前**的 mtime / sha256 / schema，并指名是哪一步改的。
|
||||
- [ ] **p2-2** 安全与隐私
|
||||
|
||||
+11
-11
@@ -1,23 +1,23 @@
|
||||
# DENSE: the vector leg at corpus size (commit dff5977ab)
|
||||
# DENSE: the vector leg at corpus size (commit 71733d86d)
|
||||
|
||||
- database: `bench/runs/dense_scale/data/ely_eye.sqlite`
|
||||
- atoms: 165973, memories with words: 153857, vectors: 153857, pending: 0
|
||||
- model: `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2`, width 384
|
||||
- build: 1421.0s for 153857 memories (108 per second)
|
||||
- writing one memory into a full index: p50 518ms · p95 543ms over 50 writes
|
||||
- build: 1444.9s for 153857 memories (106 per second)
|
||||
- writing one memory into a full index: p50 508ms · p95 535ms over 50 writes
|
||||
- the repair scan (`ely-eye build-dense-index`) over a full index: 1.3s, and it is not on the search path
|
||||
- search p50 75.6ms · p95 93.0ms · max 123.6ms over 300 queries
|
||||
- of which encoding the question: p50 12.4ms · p95 25.8ms
|
||||
- search p50 73.0ms · p95 75.7ms · max 85.6ms over 300 queries
|
||||
- of which encoding the question: p50 11.3ms · p95 24.0ms
|
||||
- against a brute-force scan: 0.998 of the top 10 over 40 probes (a returned atom counts when its true cosine is at least the k-th best, so ties cannot fail the check)
|
||||
- bytes per stored vector: 1729 (266059776 bytes for 153857 vectors, dbstat page sizes over the vec0 shadow tables)
|
||||
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [3.18, 4.58, 5.42]
|
||||
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [3.53, 4.63, 4.85]
|
||||
|
||||
## One million memories on this machine
|
||||
## One million vectors on this machine
|
||||
|
||||
- p50 455ms · p95 561ms
|
||||
- p50 475ms · p95 492ms
|
||||
- vector table: 1.6 GiB
|
||||
- first build: 153.9 minutes
|
||||
- writing one memory: 3123ms — a vec0 delete by id scans the table, so this grows with the count too
|
||||
- assumption: sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the count; the model, the machine and the thread budget stay the same
|
||||
- first build: 156.5 minutes
|
||||
- writing one vector: 3303ms — a vec0 delete by id scans the table, so this grows with the count too
|
||||
- assumption: sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the vector count; every row here is scaled by 1000000 / 153857 vectors, and the model, the machine and the thread budget stay the same
|
||||
|
||||
_Latency is wall time per search on the machine named above with the thread budget the benchmark pins; a run on a loaded machine is not comparable._
|
||||
+12
-10
@@ -1,22 +1,24 @@
|
||||
# LEGS: what each retrieval leg is worth
|
||||
|
||||
- commit: `dff5977ab638896cc8685c78ee41e5195fc0fbf2`, schema v8
|
||||
- store: bench/runs/leg_check/data/ely_eye.sqlite (a copy of bench/runs/dense_scale; the frozen baseline is never opened)
|
||||
- commit: `71733d86d6d27d1f1ebb75205d5e0599f253f791`, schema v8
|
||||
- store: bench/runs/leg_check/data/ely_eye.sqlite — a copy of bench/runs/dense_scale, which carries 153,857 dense vectors built by bench/dense_scale. The build sits outside the measured latency, and the frozen baseline itself holds no vectors, so a dense arm there would rank an empty index.
|
||||
- 300 answerable queries, each asked as the holder the frozen set assigns it
|
||||
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [25.51, 9.62, 5.7]
|
||||
- picture index: 53,668 atoms at 512 dimensions, 109,912,192 bytes, read from bench/runs/leg_check_index
|
||||
- encoder speed: 16.8 pictures/s measured on 200 pictures, so the 53,668 pictures in that index cost about 53.3 minutes to encode
|
||||
- machine: Linux 6.18.40.1-microsoft-standard-WSL2, 32 cores, 4 threads per query, load [22.19, 9.9, 6.39]
|
||||
|
||||
Recall is scored on the first ten **sources** after atoms are collapsed to their source
|
||||
by first hit, which is how every other arm in this repo is scored.
|
||||
|
||||
| leg configuration | atoms asked | recall@10 | MRR | hit@10 | 95% lower | p50 ms | p95 ms | distinct sources |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| words only | 10 | 0.832 | 0.683 | 0.833 | 0.787 | 65.5 | 134.9 | 4.0 |
|
||||
| words only (the configured 24) | 24 | 0.848 | 0.688 | 0.850 | 0.805 | 72.5 | 155.9 | 6.0 |
|
||||
| words only, wider page | 30 | 0.855 | 0.690 | 0.857 | 0.812 | 64.7 | 129.7 | 6.0 |
|
||||
| + dense leg | 10 | 0.845 | 0.663 | 0.847 | 0.802 | 472.9 | 577.5 | 5.0 |
|
||||
| + dense leg, wider page | 30 | 0.858 | 0.669 | 0.860 | 0.816 | 413.7 | 525.8 | 7.0 |
|
||||
| + picture leg | 10 | 0.815 | 0.553 | 0.817 | 0.769 | 132.5 | 271.3 | 8.0 |
|
||||
| + picture leg, wider page | 30 | 0.825 | 0.551 | 0.827 | 0.780 | 162.8 | 283.7 | 20.0 |
|
||||
| words only | 10 | 0.832 | 0.683 | 0.833 | 0.787 | 67.5 | 146.7 | 4.0 |
|
||||
| words only (the configured 24) | 24 | 0.848 | 0.688 | 0.850 | 0.805 | 67.6 | 129.9 | 6.0 |
|
||||
| words only, wider page | 30 | 0.855 | 0.690 | 0.857 | 0.812 | 72.0 | 156.0 | 6.0 |
|
||||
| + dense leg | 10 | 0.845 | 0.663 | 0.847 | 0.802 | 474.8 | 579.1 | 5.0 |
|
||||
| + dense leg, wider page | 30 | 0.858 | 0.669 | 0.860 | 0.816 | 416.0 | 543.4 | 7.0 |
|
||||
| + picture leg | 10 | 0.815 | 0.553 | 0.817 | 0.769 | 138.1 | 303.6 | 8.0 |
|
||||
| + picture leg, wider page | 30 | 0.825 | 0.551 | 0.827 | 0.780 | 166.7 | 314.2 | 20.0 |
|
||||
|
||||
## recall@10 by modality
|
||||
|
||||
|
||||
+26
-25
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"commit": "dff5977ab638896cc8685c78ee41e5195fc0fbf2",
|
||||
"commit": "71733d86d6d27d1f1ebb75205d5e0599f253f791",
|
||||
"database": "bench/runs/dense_scale/data/ely_eye.sqlite",
|
||||
"atoms": 165973,
|
||||
"memories_with_words": 153857,
|
||||
@@ -7,17 +7,17 @@
|
||||
"pending": 0,
|
||||
"dim": 384,
|
||||
"model": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
|
||||
"copy_seconds": 0.665644496002642,
|
||||
"build_seconds": 1421.0276969089973,
|
||||
"texts_per_second": 108.2716405420302,
|
||||
"repair_seconds": 1.2946931579936063,
|
||||
"copy_seconds": 4.653337935000309,
|
||||
"build_seconds": 1444.851663221998,
|
||||
"texts_per_second": 106.48636390596745,
|
||||
"repair_seconds": 1.2708722250099527,
|
||||
"writes": 50,
|
||||
"write_latency": {
|
||||
"queries": 50,
|
||||
"mean": 0.5204645371803781,
|
||||
"p50": 0.5183807220018934,
|
||||
"p95": 0.5429989652060613,
|
||||
"max": 0.5760761429992272
|
||||
"mean": 0.5132235406010295,
|
||||
"p50": 0.50813193799695,
|
||||
"p95": 0.5353954541016719,
|
||||
"max": 0.5569016869994812
|
||||
},
|
||||
"probes": 300,
|
||||
"probe_modalities": [
|
||||
@@ -29,17 +29,17 @@
|
||||
],
|
||||
"latency": {
|
||||
"queries": 300,
|
||||
"mean": 0.07861847096656371,
|
||||
"p50": 0.07557305500085931,
|
||||
"p95": 0.09303550984986941,
|
||||
"max": 0.123588470989489
|
||||
"mean": 0.07290171435301697,
|
||||
"p50": 0.07302105800044956,
|
||||
"p95": 0.07570205154406721,
|
||||
"max": 0.08561979999649338
|
||||
},
|
||||
"encode_latency": {
|
||||
"queries": 300,
|
||||
"mean": 0.014255492710411393,
|
||||
"p50": 0.012439899503078777,
|
||||
"p95": 0.025788354348333098,
|
||||
"max": 0.0513256230042316
|
||||
"mean": 0.012841759066456385,
|
||||
"p50": 0.011289245005173143,
|
||||
"p95": 0.024048260039126034,
|
||||
"max": 0.02614720400015358
|
||||
},
|
||||
"correctness": {
|
||||
"probes": 40,
|
||||
@@ -52,12 +52,13 @@
|
||||
"bytes_method": "dbstat page sizes over the vec0 shadow tables",
|
||||
"extrapolation": {
|
||||
"to": 1000000,
|
||||
"assumption": "sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the count; the model, the machine and the thread budget stay the same",
|
||||
"p50_seconds": 0.45533342773137386,
|
||||
"p95_seconds": 0.5605460517666694,
|
||||
"write_p50_seconds": 3.1232834376789804,
|
||||
"basis": 153857,
|
||||
"assumption": "sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the vector count; every row here is scaled by 1000000 / 153857 vectors, and the model, the machine and the thread budget stay the same",
|
||||
"p50_seconds": 0.47460341746199103,
|
||||
"p95_seconds": 0.49202864701682225,
|
||||
"write_p50_seconds": 3.3026247619344584,
|
||||
"gibibytes": 1.6105050507025191,
|
||||
"build_seconds": 9236.028889871746
|
||||
"build_seconds": 9390.873754343305
|
||||
},
|
||||
"machine": {
|
||||
"system": "Linux",
|
||||
@@ -65,9 +66,9 @@
|
||||
"cpus": 32,
|
||||
"threads_per_query": 4,
|
||||
"load_average_1_5_15": [
|
||||
3.18,
|
||||
4.58,
|
||||
5.42
|
||||
3.53,
|
||||
4.63,
|
||||
4.85
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,20 @@
|
||||
{
|
||||
"commit": "dff5977ab638896cc8685c78ee41e5195fc0fbf2",
|
||||
"commit": "71733d86d6d27d1f1ebb75205d5e0599f253f791",
|
||||
"schema_version": 8,
|
||||
"store": "bench/runs/leg_check/data/ely_eye.sqlite",
|
||||
"source_store": "bench/runs/dense_scale",
|
||||
"dense_vectors": 153857,
|
||||
"picture_index": {
|
||||
"path": "bench/runs/leg_check_index",
|
||||
"atoms": 53668,
|
||||
"dim": 512,
|
||||
"bytes": 109912192
|
||||
},
|
||||
"encode_rate": {
|
||||
"pictures": 200,
|
||||
"seconds": 11.912494190008147,
|
||||
"per_second": 16.7890952818054
|
||||
},
|
||||
"probes": 300,
|
||||
"arms": [
|
||||
{
|
||||
@@ -13,8 +26,8 @@
|
||||
"mrr": 0.6832447089947089,
|
||||
"hit_at_10": 0.8333333333333334,
|
||||
"wilson_lower": 0.7870033503873167,
|
||||
"p50_seconds": 0.0654734540003119,
|
||||
"p95_seconds": 0.13490134734311146,
|
||||
"p50_seconds": 0.0674771249978221,
|
||||
"p95_seconds": 0.14665940315098855,
|
||||
"median_distinct_sources": 4.0,
|
||||
"by_modality": {
|
||||
"code": 0.953125,
|
||||
@@ -40,8 +53,8 @@
|
||||
"mrr": 0.6875571326821327,
|
||||
"hit_at_10": 0.85,
|
||||
"wilson_lower": 0.8051823888546709,
|
||||
"p50_seconds": 0.07251374649786158,
|
||||
"p95_seconds": 0.15588207065375176,
|
||||
"p50_seconds": 0.0675995074925595,
|
||||
"p95_seconds": 0.12988004180151622,
|
||||
"median_distinct_sources": 6.0,
|
||||
"by_modality": {
|
||||
"code": 0.953125,
|
||||
@@ -67,8 +80,8 @@
|
||||
"mrr": 0.6895710215710216,
|
||||
"hit_at_10": 0.8566666666666667,
|
||||
"wilson_lower": 0.8124994228503402,
|
||||
"p50_seconds": 0.06468957299512113,
|
||||
"p95_seconds": 0.1297007056004077,
|
||||
"p50_seconds": 0.07200110949634109,
|
||||
"p95_seconds": 0.15601039270331973,
|
||||
"median_distinct_sources": 6.0,
|
||||
"by_modality": {
|
||||
"code": 0.953125,
|
||||
@@ -94,8 +107,8 @@
|
||||
"mrr": 0.6632195767195768,
|
||||
"hit_at_10": 0.8466666666666667,
|
||||
"wilson_lower": 0.8015339429418983,
|
||||
"p50_seconds": 0.4729458744986914,
|
||||
"p95_seconds": 0.5775450093562541,
|
||||
"p50_seconds": 0.4748052095019375,
|
||||
"p95_seconds": 0.5791264187995694,
|
||||
"median_distinct_sources": 5.0,
|
||||
"by_modality": {
|
||||
"code": 0.9375,
|
||||
@@ -121,8 +134,8 @@
|
||||
"mrr": 0.6686637344137344,
|
||||
"hit_at_10": 0.86,
|
||||
"wilson_lower": 0.8161683876490509,
|
||||
"p50_seconds": 0.41367531901050825,
|
||||
"p95_seconds": 0.5258364057008293,
|
||||
"p50_seconds": 0.41600067100080196,
|
||||
"p95_seconds": 0.5433694835548523,
|
||||
"median_distinct_sources": 7.0,
|
||||
"by_modality": {
|
||||
"code": 0.9375,
|
||||
@@ -148,8 +161,8 @@
|
||||
"mrr": 0.5525886243386243,
|
||||
"hit_at_10": 0.8166666666666667,
|
||||
"wilson_lower": 0.7689713823226987,
|
||||
"p50_seconds": 0.13248339149868116,
|
||||
"p95_seconds": 0.2713221473451995,
|
||||
"p50_seconds": 0.1381109430003562,
|
||||
"p95_seconds": 0.3035783412473393,
|
||||
"median_distinct_sources": 8.0,
|
||||
"by_modality": {
|
||||
"code": 0.9296875,
|
||||
@@ -175,8 +188,8 @@
|
||||
"mrr": 0.5507809648412589,
|
||||
"hit_at_10": 0.8266666666666667,
|
||||
"wilson_lower": 0.7797738592974309,
|
||||
"p50_seconds": 0.16283495400421089,
|
||||
"p95_seconds": 0.2837207488548302,
|
||||
"p50_seconds": 0.16669872149941511,
|
||||
"p95_seconds": 0.3142305556466454,
|
||||
"median_distinct_sources": 20.0,
|
||||
"by_modality": {
|
||||
"code": 0.8984375,
|
||||
@@ -200,9 +213,9 @@
|
||||
"cpus": 32,
|
||||
"threads_per_query": 4,
|
||||
"load_average_1_5_15": [
|
||||
25.51,
|
||||
9.62,
|
||||
5.7
|
||||
22.19,
|
||||
9.9,
|
||||
6.39
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user