Stop the checklist from quoting numbers no artifact carries
A review of the p2-1 record found seven places where the prose had drifted away from the reports it cites. None were product defects; all were record keeping, and record keeping is what a memory system is judged on. The quantization row mis-transcribed nine latencies, and the two worst made dimension truncation look twice as fast as the full scan when the report measures six percent. The extrapolation table scaled latency by atoms and bytes by vectors under one heading that named neither switch. The picture index stats existed only in prose, so the harness now reads the index shape off the artifact and times the encoder on a sample, which reproduces the hand-timed build within seven percent. The dense arms named a store that holds no vectors, so the report names the store the vectors came from and refuses to run where a dense leg would rank an empty index. The frozen baseline was migrated in place by the product CLI while the picture index was built, so its pinned mtime, digest and schema version are re-pinned and the step that moved it is named. The sqlite_vec import added two mypy errors; the fix is the missing stub declaration, not an inline ignore, and the count is back to the recorded baseline of twenty-four.
This commit is contained in:
1 parent
6cc5640856
commit
71733d86d6
7 files changed
+153
-31
No files matched your search
+16
-14
@@ -280,8 +280,8 @@
|
||||
- `backend/src/ely_eye/maintenance.py` + `cli_maintenance.py`(走仓库已有的 `register(app, settings)` 模式,因为 `cli.py` 已 490 行,再塞三个命令就破 500 规则)。backup 用 SQLite 自己的在线备份 API(服务在跑也能备,不是 `shutil.copy` 拷一个正在写的 WAL 库),清单记 schema 版本 / 行数 / sha256 / 体积 / 版本,行数是从**副本**读回来的
|
||||
- restore 的顺序是校验在先:sha256 对不上清单 → 拒绝;`PRAGMA integrity_check` 不是 ok → 拒绝;缺必需表 → 拒绝;schema 比本构建新 → 拒绝;被替换的库连同 `-wal`/`-shm` 留成 `.before-restore`;然后 `Database(settings).init()`,所以旧备份是「进来时被迁移」而不是要求人再跑一次
|
||||
- doctor 分两级:problems 让库停摆(integrity、打不开、挂载的卡带文件没了、ui_dir 设了但没建),warnings 只说明降级(快照缺失、视觉索引落后、dense 索引落后)
|
||||
- 真数据实测(`bench/runs/baseline`,948,822,016 字节 = 904.8 MiB / 165,973 atoms / 50,278 sources / schema 7,库文件 mtime `2026-10-08 02:33:36`):`backup` 清单里的 sha256 `c815c6049c5bb562…` 用 `sha256sum` 独立核对一致;`restore` 到新 home 后读回 165,973 atoms / 50,278 sources / schema 7 全等;`doctor` ok=true、0 problems,并给出真降级警告:**「the store holds 21362 picture atoms but no CLIP snapshot is set (ELY_EYE_CLIP_MODEL_DIR), so they are findable only by the text already on them」**
|
||||
- 那个 sha256 是**那份快照**的指纹,不是目录的:同一状态下连备两次哈希逐字节相同(所以它是确定的),但库在提交之后又被写过,现在再备会得到 `498b0cd92c1be78d…`。秒数不写死:我这次 backup 4.0 秒 / restore 2.8 秒,reviewer 复跑 2.24 秒 / 5.65 秒,同一台机器负载不同就差这么多,只有行数和哈希是可复现的
|
||||
- 真数据实测(`bench/runs/baseline`,948,822,016 字节 = 904.8 MiB / 165,973 atoms / 50,278 sources):**当时**库是 schema 7、mtime `2026-10-08 02:33:36`,`backup` 清单里的 sha256 `c815c6049c5bb562…` 用 `sha256sum` 独立核对一致;`restore` 到新 home 后读回 165,973 atoms / 50,278 sources / schema 7 全等;`doctor` ok=true、0 problems,并给出真降级警告:**「the store holds 21362 picture atoms but no CLIP snapshot is set (ELY_EYE_CLIP_MODEL_DIR), so they are findable only by the text already on them」**
|
||||
- 那个 sha256 是**那份快照**的指纹,不是目录的:同一状态下连备两次哈希逐字节相同(所以它是确定的),但库被写过,哈希就跟着走。p2-1 之后这个文件是 mtime `2026-10-08 15:54:54`、948,822,016 字节、sha256 `61244b6de73d478c4210ef79af2c4a21ef753cbc90b2806bb25a3617f236fe26`、schema **8**;现在再 `backup` 得到 `0d2d31bcda9d790b3d03cd9094e11af25664cf05d209fe7e060bc142fb783150`(2.56 秒,读回 165,973 atoms / 50,278 sources / schema 8)。把它推到 schema 8 的是**产品 CLI**而不是 harness:p2-1.4 建 CLIP 图片索引时 `ely-eye index-visual` 在 15:54 打开了这个库,而 `bench/scratch.py` 那条「harness 只动副本」的规则管不到产品命令;harness 那侧复核过,`python -m bench.scale_check` 跑完文件逐字节不变。秒数不写死:我这次 backup 4.0 秒 / restore 2.8 秒,reviewer 复跑 2.24 秒 / 5.65 秒,p2-1 这次 backup 2.56 秒,同一台机器负载不同就差这么多,只有行数和哈希是可复现的
|
||||
- doctor 第一版是假健康:它只对「路径设了但目录不在」报警,对「库里有 21,362 个图片 atom 却压根没配 CLIP」报 ok=true。现在按库里的 modality 计数判,`test_doctor_names_the_senses_a_store_holds_but_cannot_use` 钉住这条
|
||||
- 测试写出来才发现的两条:`Database.__init__` 每次构造都跑迁移,所以 doctor 永远看不到旧 schema,我原先那句 remediation「run `ely-eye status` to migrate」是编的 → 改成 catch `SystemExit` 报「the store cannot be opened by this build: …」;另一条是 fixture 假设错(`_store()` 往已有库里加,被替换的库确实有 3 个 atom),改测试不改产品
|
||||
- 验收:CI 绿;干净 clone 上 `docker build` 成功(已实测 `09803cd49add` + 容器 curl 通);桩 `nvidia-smi` 打印 RTX 5090 证明设备名不再致命(已实测)
|
||||
@@ -346,22 +346,24 @@
|
||||
- 实测出来的 vec0 0.1.9 坑(都是跑出来的,不是猜的):`metadata json` 不是合法表选项(只有 `metadata text`);KNN 是 `WHERE <列> MATCH ? AND k = ?`(匹配列不是表);int8/bit 的插入与查询向量必须套 SQL 构造器 `vec_int8(blob)` / `vec_bit(blob)`,`sqlite_vec.serialize_int8()` 返回的裸 BLOB 会被拒(`expected to be of type int8, but a float32 vector was provided`);float32 不需要包装,也没有 `vec_float32()`;默认距离是 L2,cosine 是表选项。
|
||||
- 偏差:0.1.9 的 vec0 **没有 ANN,逐条扫**,所以延迟与字节随规模线性——DENSE.md 把这条假设印在外推表旁边,不假装是对数。
|
||||
- [x] p2-1.2 量化与截断:测了,没上
|
||||
- `python -m bench.quant_probe` → `bench/reports/QUANT.md`(165,973 向量、200 探针、top 10,tie-aware 一致度):sqlite-vec float 40.3ms/1.0、+holder 过滤 29.3ms/0.599、int8 35.3ms/**0.977**、int8+holder 23.0/0.595、bit 2.7ms/0.728、bit+holder 4.2/0.513、numpy float32 全扫(真值)8.0ms/1.0、numpy 符号位 hamming 19.1/0.011、float32 前 192/128/64 维 4.4/0.722、4.0/0.57、4.0/0.43、**bit 粗排@200 → numpy 里 float32 精排 23.7ms/0.983**。每向量字节 f32 1536 / int8 384 / bit 48。
|
||||
- 结论:int8 省 4× 空间几乎不掉质量(0.977)但不省时间;bit 省 32× 空间和 14× 时间、只剩 0.728,只能当粗排;两级方案必须在 vec0 **外面**精排(在库里用 `rowid IN (…)` 回捞反而更慢)。MRL 截断是在**不是 MRL 训练**的 MiniLM 上测的,所以那是真实代价不是白拿;切片必须是存下来的连续维,不是查询时切(非连续视图点集会掉速度)。
|
||||
- `python -m bench.quant_probe` → `bench/reports/QUANT.md`(165,973 向量、200 探针、top 10,tie-aware 一致度;下面每个 p50 逐字抄自那张表):sqlite-vec float **41.2ms**/1.0、+holder 过滤 29.8/0.599、int8 35.5/**0.977**、int8+holder 22.3/0.595、bit **2.6ms**/0.728、bit+holder 4.1/0.513、numpy float32 全扫(真值)8.0/1.0、numpy 符号位 hamming 18.9/0.011、float32 前 192/128/64 维 7.5/0.722、7.9/0.57、4.0/0.43、**bit 粗排@200 → numpy 里 float32 精排 23.7/0.983**。每向量字节 f32 1536 / int8 384 / bit 48。
|
||||
- 结论:int8 省 4× 空间几乎不掉质量(0.977)但不省时间(35.5 vs 41.2);bit 省 32× 空间、省 **15.8×** 时间(41.2→2.6),只剩 0.728,只能当粗排;两级方案必须在 vec0 **外面**精排(在库里用 `rowid IN (…)` 回捞反而更慢)。维度截断是在**不是 MRL 训练**的 MiniLM 上测的,所以那是真实代价不是白拿:前 192 维 7.5ms、前 128 维 7.9ms,对全维 8.0ms 只快 1–6%,前 64 维才快一半但一致度掉到 0.43;切片必须是存下来的连续维,不是查询时切(非连续视图点集会掉速度)。
|
||||
- 偏差:精排腿(`bge-reranker-base`)早在基线消融里就有并测过(加了它 recall@10 从 0.565 掉到 0.453),所以这里没再为它编一套故事;粗排+精排这条链测出 0.983 一致度但**没接线**,因为 dense 腿默认关(见下)。
|
||||
- [x] p2-1.3 100k 实测 + 1M 外推:`python -m bench.dense_scale` → `bench/reports/DENSE.md`(stamp dff5977)
|
||||
- 实测 **165,973 atom / 153,857 向量**:建库 1421.0s(108 向量/秒)、**检索 p50 75.6ms / p95 93.0ms / max 123.6ms(300 查询)**、其中编码问题 p50 12.4ms / p95 25.8ms、与暴力扫描 tie-aware 一致度 **0.998**(40 探针)、**往满索引里写一条记忆 p50 518ms / p95 543ms(50 次写)**、失配扫描 1.3s(不在检索路径上)、**每向量 1,729 字节**(dbstat 数 vec0 影子表:266,059,776 B)、机器 load [3.18, 4.58, 5.42]。
|
||||
- 1M 外推:p50 455ms / p95 561ms、向量表 1.6 GiB、首次建库 153.9 分钟、单条写 3123ms(vec0 按 id 删除也要扫表,所以写同样随规模线性),假设随行印出(逐条扫 ⇒ 线性)。
|
||||
- 实测 **165,973 atom / 153,857 向量**(DENSE.md,stamp 6cc564085):建库 1444.8s(106 向量/秒)、**检索 p50 73.1ms / p95 76.1ms / max 87.2ms(300 查询)**、其中编码问题 p50 11.0ms / p95 24.1ms、与暴力扫描 tie-aware 一致度 **0.998**(40 探针)、**往满索引里写一条记忆 p50 509ms / p95 531ms(50 次写)**、失配扫描 1.2s(不在检索路径上)、**每向量 1,729 字节**(dbstat 数 vec0 影子表:266,059,776 B)、机器 load [3.41, 4.52, 4.31]。
|
||||
- 1M 外推(**分母是向量数**,1,000,000 / 153,857,不是 atom 数——两者差 7.9%,所以报告里把分母印在外推表旁边):p50 475ms / p95 494ms、向量表 1.6 GiB、首次建库 156.5 分钟、单条写 3310ms(vec0 按 id 删除也要扫表,所以写同样随规模线性),假设随行印出(逐条扫 ⇒ 线性)。
|
||||
- tie-aware 的必要性:语料里有重复 atom,分数并列 ⇒ top-K 的 id 集合本身有歧义,两个都对引擎按 id 比会读出 0.827 的假召回。判定改成「返回的 atom 真实 cosine ≥ 第 K 名 cosine」。
|
||||
- 验收:100k dense 腿 p50 <100ms → **165,973 atom(bar 的 1.66 倍)p50 75.6ms**(DENSE.md);新 atom 立即可查 → `pytest backend/tests/test_vector_index.py`(写进去当场能搜到、视图藏起来的 dense 腿也藏、备份带走向量恢复后照样应答)。
|
||||
- [x] p2-1.4 每条腿值多少钱:`python -m bench.leg_check` → `bench/reports/LEGS.md`(stamp dff5977,300 条 holder 作用域探针,7 个配置,每臂带 n / MRR / Wilson 下界 / p50 / p95 / 分模态召回)
|
||||
- 只有词的 FTS 页:问 10 个 atom recall@10 **0.832**、MRR 0.683、p50 65.5ms;问产品配置的 24 个 **0.848** / 0.688 / 72.5ms;页放宽到 30 **0.855** / 0.690 / 64.7ms(放宽不要钱,10→24 就白捡 1.6 个点)。
|
||||
- 开 dense 腿:10 个 atom **0.845** / MRR 0.663 / p50 472.9ms;页 30 **0.858** / 0.669 / 413.7ms。多 1.3–2.6 个点召回、MRR 降、慢 6–7 倍。
|
||||
- 开图片腿(CLIP 索引 53,668 张图、建库 2901.7s、512 维):10 个 atom **0.815** / MRR 0.553 / 132.5ms;页 30 **0.825** / 0.551 / 162.8ms。分模态才看得见它在干什么:**video 0.583→0.708、pdf 0.849→0.907 涨,code 0.953→0.898、ui_screenshot 0.583→0.500 跌**——它救的正是金标是图的题,代价是稀释别的。
|
||||
- 验收:100k dense 腿 p50 <100ms → **165,973 atom(bar 的 1.66 倍)p50 73.1ms**(DENSE.md);新 atom 立即可查 → `pytest backend/tests/test_vector_index.py`(写进去当场能搜到、视图藏起来的 dense 腿也藏、备份带走向量恢复后照样应答)。
|
||||
- [x] p2-1.4 每条腿值多少钱:`python -m bench.leg_check` → `bench/reports/LEGS.md`(stamp 见报告首行,300 条 holder 作用域探针,7 个配置,每臂带 n / MRR / Wilson 下界 / p50 / p95 / 分模态召回)。测的是 `bench/runs/dense_scale` 的副本:冻结基线自己**不带向量**,dense 腿在其上只会排一个空索引,所以报告首行就把来源 store 与向量数印出来。
|
||||
- 只有词的 FTS 页:问 10 个 atom recall@10 **0.832**、MRR 0.683、p50 65.1ms;问产品配置的 24 个 **0.848** / 0.688 / 66.5ms;页放宽到 30 **0.855** / 0.690 / 94.0ms(放宽不要钱,10→24 就白捡 1.6 个点)。
|
||||
- 开 dense 腿:10 个 atom **0.845** / MRR 0.663 / p50 481.4ms;页 30 **0.858** / 0.669 / 412.8ms。多 1.3–2.6 个点召回、MRR 降、慢 6–7 倍。
|
||||
- 开图片腿。索引实物印在 LEGS.md 里(不是这里口头说的):**53,668 个 atom、512 维、109,912,192 字节**;编码速度实测 **19.9 张/秒**(200 张样本),据此那 53,668 张 ≈ **45.0 分钟**——和我当时手工记的 2901.7 秒(48.4 分钟)差 7%,两条路对得上。结果:10 个 atom **0.815** / MRR 0.553 / 139.5ms;页 30 **0.825** / 0.551 / 160.0ms。分模态才看得见它在干什么:**video 0.583→0.708、pdf 0.849→0.907 涨,code 0.953→0.898、ui_screenshot 0.583→0.500 跌**——它救的正是金标是图的题,代价是稀释别的。
|
||||
- p50 是负载函数,不是性常数:同一份 harness 两次跑,words 腿 64.7–72.5ms、dense 腿 412.8–481.4ms、图片腿 132.5–160.0ms,**召回一格没动**。所以比较只用召回,延迟看报告当次的 load。
|
||||
- 产品的图片腿不是开关:`retrieval.py:136` `if image is None and not thin: return {}`,稀疏页薄到 2 条以下才问图。所以这几行测的是「腿可用」,不是强行常开融合。
|
||||
- **没达到的验收线,写在这里**:用户定的验收是 recall@10 ≥0.90(n≥300)。测出来最好的一行是 **0.858**(dense 腿开、页 30、p50 414ms),默认配置 **0.848**(p50 72.5ms),CURRENT.md 里按 10 个 atom 问是 0.832;仓内 Mem0/Zep 式对照在 holder 作用域是 **0.866**。差距集中在图(image 0.750、ui_screenshot 0.500–0.583、video 0.583–0.708),文字的题(code 0.953、pdf 0.919)已经过了线。诚实结论:**延迟达标(281 倍于基线、p50 71.7ms),召回没达标**,剩下 4 个点是跨模态召回,不是索引规模问题。
|
||||
- 数字逼出来的决定:**dense 腿默认关**(`config.py:32 dense_embeddings_enabled=False`)。同一 300 条 holder 作用域探针上(LEGS.md),开腿 recall@10 0.832→0.845、MRR 0.683→0.663、p50 65.5ms→472.9ms。多 1.3 个点的召回、慢 7 倍、首位质量还降——默认值就该是关,腿留着(已增量、已入库)等更大的语料或更好的嵌入模型。
|
||||
- 顺带修的测量缺陷:`copy_store()` 只拷 SQLite,所以任何要跑视觉腿的 harness 都得自己带上 `objects/`(25 GB,探针里用符号链接)与 `cache/clip_image_index.npy`;`bench/scale_check.py` 原本直接打开冻结基线库、把它**原地迁移**了,现改成在副本上测(`--scratch`)。
|
||||
- **没达到的验收线,写在这里**:用户定的验收是 recall@10 ≥0.90(n≥300)。测出来最好的一行是 **0.858**(dense 腿开、页 30、p50 412.8ms),默认配置 **0.848**(p50 66.5ms),CURRENT.md 里按 10 个 atom 问是 0.832;仓内 Mem0/Zep 式对照在 holder 作用域是 **0.866**。差距集中在图(image 0.750、ui_screenshot 0.500–0.583、video 0.583–0.708),文字的题(code 0.953、pdf 0.919)已经过了线。诚实结论:**延迟达标(281 倍于基线、p50 71.7ms),召回没达标**,剩下 4 个点是跨模态召回,不是索引规模问题。
|
||||
- 数字逼出来的决定:**dense 腿默认关**(`config.py:32 dense_embeddings_enabled=False`)。同一 300 条 holder 作用域探针上(LEGS.md),开腿 recall@10 0.832→0.845、MRR 0.683→0.663、p50 65.1ms→481.4ms。多 1.3 个点的召回、慢 7 倍、首位质量还降——默认值就该是关,腿留着(已增量、已入库)等更大的语料或更好的嵌入模型。
|
||||
- 门禁:`pytest -q` 全绿;`ruff check` 干净;mypy 冷缓存 **24 errors / 9 files**,与 p1-6 基线(`9cb46c1` 同口径 24/9)**净新增 0**——p2-1 一度引入 +2(`vector_index.py` 与 `quant_probe.py` 的 `import sqlite_vec` 没有 stub),根因修法是在 `pyproject.toml` 的 `[[tool.mypy.overrides]]` 里声明 `sqlite_vec` 无 stub,而不是在代码里加 `# type: ignore`;`./node_modules/.bin/pyright` 对改动文件 0 errors;`bench/filesize_check` 干净;go / npm 未动。
|
||||
- 顺带修的测量缺陷:`copy_store()` 只拷 SQLite,所以任何要跑视觉腿的 harness 都得自己带上 `objects/`(25 GB,探针里用符号链接)与 `cache/clip_image_index.npy`;`bench/scale_check.py` 原本直接打开冻结基线库、把它**原地迁移**了,现改成在副本上测(`--scratch`)。产品 CLI 不在这条规则里:p2-1.4 建图片索引时 `ely-eye index-visual` 就把冻结基线从 schema 7 升到了 8,所以 p1-4.8 那两条钉的是**当前**的 mtime / sha256 / schema,并指名是哪一步改的。
|
||||
- [ ] **p2-2** 安全与隐私
|
||||
- [ ] p2-2.1 FastAPI 鉴权 + Origin 校验
|
||||
- [ ] p2-2.2 写入信任分级与隔离:未佐证不得被引用
|
||||
|
||||
@@ -40,8 +40,7 @@ def _dense_facts(settings: Settings, database: Database) -> tuple[DenseState, in
|
||||
facts = VectorIndex(settings, database).stats()
|
||||
if not facts["built"]:
|
||||
return "missing", 0, expected
|
||||
# A memory with no words belongs to the picture index, so the honest denominator is the
|
||||
# memories this index is meant to hold, not every row in the store.
|
||||
# A wordless memory belongs to the picture index, so the denominator is what this index holds, not every row.
|
||||
return ("current" if facts["pending"] == 0 else "behind"), int(facts["vectors"]), expected
|
||||
|
||||
|
||||
|
||||
@@ -22,8 +22,7 @@ from .db import Database
|
||||
from .embeddings import EmbeddingService
|
||||
from .row_codec import to_json
|
||||
|
||||
# The vec0 schema cannot be altered in place, so the format tag is load-bearing: changing the
|
||||
# vector width or the model means dropping the table and re-encoding, not migrating it.
|
||||
# vec0 cannot be altered in place, so the format tag is load-bearing: a new width or model drops the table and re-encodes.
|
||||
FORMAT = "ely-eye-dense-vectors-v1"
|
||||
TABLE = "vec_atoms"
|
||||
META_TABLE = "vector_index_meta"
|
||||
@@ -183,8 +182,7 @@ class VectorIndex:
|
||||
as_holder=as_holder,
|
||||
)
|
||||
hits = {atom_id: 1.0 - distance for atom_id, distance in ranked if atom_id in allowed}
|
||||
# A filter that keeps little of the wide ask means there are more answers further
|
||||
# out; one wider round buys them, and the ceiling keeps the round bounded.
|
||||
# A filter that keeps little of the ask means more answers sit further out; one wider round buys them, bounded by the ceiling.
|
||||
if len(hits) >= limit or len(ranked) < asked or asked >= CANDIDATE_CEILING:
|
||||
break
|
||||
asked = min(asked * 4, CANDIDATE_CEILING)
|
||||
|
||||
@@ -192,13 +192,16 @@ def measure(home: Path, scratch: Path, probes: int, model_dir: Path) -> dict[str
|
||||
"bytes_method": bytes_method,
|
||||
"extrapolation": {
|
||||
"to": EXTRAPOLATE_TO,
|
||||
"basis": encoded,
|
||||
"assumption": (
|
||||
"sqlite-vec 0.1.9 scans every vector, so latency and bytes grow linearly with the "
|
||||
"count; the model, the machine and the thread budget stay the same"
|
||||
"vector count; every row here is scaled by "
|
||||
f"{EXTRAPOLATE_TO} / {encoded} vectors, and the model, the machine and the thread "
|
||||
"budget stay the same"
|
||||
),
|
||||
"p50_seconds": timing["p50"] * EXTRAPOLATE_TO / atoms,
|
||||
"p95_seconds": timing["p95"] * EXTRAPOLATE_TO / atoms,
|
||||
"write_p50_seconds": writes["p50"] * EXTRAPOLATE_TO / atoms,
|
||||
"p50_seconds": timing["p50"] * EXTRAPOLATE_TO / encoded,
|
||||
"p95_seconds": timing["p95"] * EXTRAPOLATE_TO / encoded,
|
||||
"write_p50_seconds": writes["p50"] * EXTRAPOLATE_TO / encoded,
|
||||
"gibibytes": per_vector * EXTRAPOLATE_TO / 1024**3,
|
||||
"build_seconds": EXTRAPOLATE_TO / texts_per_second if texts_per_second else None,
|
||||
},
|
||||
@@ -254,12 +257,12 @@ def write_markdown(report: dict[str, Any], path: Path) -> None:
|
||||
f"{machine['threads_per_query']} threads per query, load {machine['load_average_1_5_15']}"
|
||||
),
|
||||
"",
|
||||
"## One million memories on this machine",
|
||||
"## One million vectors on this machine",
|
||||
"",
|
||||
f"- p50 {future['p50_seconds'] * 1000:.0f}ms · p95 {future['p95_seconds'] * 1000:.0f}ms",
|
||||
f"- vector table: {future['gibibytes']:.1f} GiB",
|
||||
f"- first build: {future['build_seconds'] / 60:.1f} minutes",
|
||||
f"- writing one memory: {future['write_p50_seconds'] * 1000:.0f}ms — a vec0 delete by id scans the table, so this grows with the count too",
|
||||
f"- writing one vector: {future['write_p50_seconds'] * 1000:.0f}ms — a vec0 delete by id scans the table, so this grows with the count too",
|
||||
f"- assumption: {future['assumption']}",
|
||||
"",
|
||||
(
|
||||
|
||||
+114
-5
@@ -34,13 +34,12 @@ DEFAULT_OBJECTS = Path("bench/runs/baseline/objects")
|
||||
DEFAULT_CLIP_INDEX = Path("bench/runs/leg_check_index")
|
||||
DEFAULT_CLIP_MODEL = Path("bench/models/chinese-clip-vit-base-patch16")
|
||||
DEFAULT_EMBEDDING_MODEL = Path("bench/models/paraphrase-multilingual-MiniLM-L12-v2")
|
||||
ENCODE_SAMPLE = 200
|
||||
|
||||
# The page size the product asks for by default. measure() checks it against the product's own
|
||||
# setting, so a config change here cannot quietly turn the "as shipped" row into a hypothetical.
|
||||
# The page the product asks for by default; measure() checks it against Settings, so "as shipped" cannot rot into a hypothesis.
|
||||
SHIPPED_TOP_K = 24
|
||||
|
||||
# (label, atoms asked, dense leg, picture leg). The picture leg routes itself: it answers only when
|
||||
# the sparse page comes back thin, which is the product's rule, not this harness's.
|
||||
# (label, atoms asked, dense leg, picture leg); the picture leg answers only on a thin page, which is the product's rule.
|
||||
CONFIGS: list[dict[str, Any]] = [
|
||||
{"label": "words only", "top_k": 10, "dense": False, "visual": False},
|
||||
{"label": "words only (the configured 24)", "top_k": SHIPPED_TOP_K, "dense": False, "visual": False},
|
||||
@@ -92,6 +91,74 @@ def prepare(
|
||||
return copy_home
|
||||
|
||||
|
||||
def _dense_vectors(store: Path) -> int:
|
||||
"""Vectors the source store already carries, read without opening it for write."""
|
||||
import sqlite3
|
||||
|
||||
import sqlite_vec
|
||||
from ely_eye import vector_index
|
||||
|
||||
if vector_index.TABLE != "vec_atoms":
|
||||
raise SystemExit(f"the dense table is now {vector_index.TABLE}; update this harness's count")
|
||||
if not store.is_file():
|
||||
raise SystemExit(f"{store}: no product database")
|
||||
with sqlite3.connect(f"file:{store}?mode=ro", uri=True) as connection:
|
||||
connection.enable_load_extension(True)
|
||||
sqlite_vec.load(connection)
|
||||
connection.enable_load_extension(False)
|
||||
if not connection.execute("SELECT 1 FROM sqlite_master WHERE name = 'vec_atoms'").fetchone():
|
||||
return 0
|
||||
return int(connection.execute("SELECT count(*) FROM vec_atoms").fetchone()[0])
|
||||
|
||||
|
||||
def picture_facts(clip_index: Path) -> dict[str, Any]:
|
||||
"""What the picture index really holds, read off the artifact instead of remembered in prose."""
|
||||
import numpy as np
|
||||
|
||||
vectors_path = clip_index / "clip_image_index.npy"
|
||||
atoms_path = clip_index / "clip_image_index_atoms.json"
|
||||
vectors = np.load(vectors_path)
|
||||
with atoms_path.open(encoding="utf-8") as handle:
|
||||
try:
|
||||
atoms = json.load(handle)
|
||||
except json.JSONDecodeError as error:
|
||||
raise ValueError(f"{atoms_path}: the picture index's atom list is unreadable ({error})") from error
|
||||
if len(atoms) != int(vectors.shape[0]):
|
||||
raise SystemExit(
|
||||
f"{clip_index}: the picture index disagrees with itself — {len(atoms)} atom ids for "
|
||||
f"{vectors.shape[0]} vectors"
|
||||
)
|
||||
return {
|
||||
"path": str(clip_index),
|
||||
"atoms": len(atoms),
|
||||
"dim": int(vectors.shape[-1]),
|
||||
"bytes": int(vectors_path.stat().st_size),
|
||||
}
|
||||
|
||||
|
||||
def encode_rate(clip_model: Path, objects: Path, count: int) -> dict[str, Any]:
|
||||
"""Pictures per second the encoder manages, measured on a sample instead of remembered."""
|
||||
import time
|
||||
|
||||
from ely_eye.clip_encoder import ClipEncoder
|
||||
from PIL import Image
|
||||
|
||||
suffixes = {".png", ".jpg", ".jpeg"}
|
||||
sample = sorted(p for p in objects.rglob("*") if p.suffix.lower() in suffixes)[:count]
|
||||
if len(sample) < count:
|
||||
raise SystemExit(f"{objects}: only {len(sample)} pictures to time the encoder on")
|
||||
encoder = ClipEncoder(clip_model)
|
||||
began = time.perf_counter()
|
||||
rows = 0
|
||||
for start in range(0, len(sample), 8):
|
||||
batch = [Image.open(path).convert("RGB") for path in sample[start : start + 8]]
|
||||
rows += len(encoder.encode_images(batch))
|
||||
seconds = time.perf_counter() - began
|
||||
if rows != len(sample):
|
||||
raise SystemExit(f"the encoder returned {rows} vectors for {len(sample)} pictures")
|
||||
return {"pictures": rows, "seconds": seconds, "per_second": rows / seconds}
|
||||
|
||||
|
||||
def _schema(home: Path) -> int:
|
||||
import sqlite3
|
||||
|
||||
@@ -141,7 +208,17 @@ def measure(home: Path, scratch: Path, objects: Path, clip_index: Path, probes:
|
||||
raise SystemExit("the frozen suite carries no answerable query; nothing to score")
|
||||
print(f"{len(answerable)} answerable probes", flush=True)
|
||||
|
||||
vectors = _dense_vectors(home / "data" / "ely_eye.sqlite")
|
||||
if vectors == 0 and any(config["dense"] for config in CONFIGS):
|
||||
raise SystemExit(
|
||||
f"{home}: the store carries no dense vectors, so a dense arm here would rank an empty "
|
||||
"index and the 24-minute build would hide outside the measured latency. Build them with "
|
||||
"python -m bench.dense_scale, or point --home at a store that already has them."
|
||||
)
|
||||
|
||||
arms: list[dict[str, Any]] = []
|
||||
picture: dict[str, Any] | None = None
|
||||
rate: dict[str, Any] | None = None
|
||||
schema = 0
|
||||
for config in CONFIGS:
|
||||
print(f" {config['label']}: dense={config['dense']} visual={config['visual']}", flush=True)
|
||||
@@ -156,6 +233,10 @@ def measure(home: Path, scratch: Path, objects: Path, clip_index: Path, probes:
|
||||
embedding_model=models["embedding"],
|
||||
)
|
||||
schema = _schema(copy_home)
|
||||
if config["visual"]:
|
||||
picture = picture_facts(clip_index)
|
||||
if rate is None:
|
||||
rate = encode_rate(models["clip"], objects, ENCODE_SAMPLE)
|
||||
settings = Settings(
|
||||
home=copy_home,
|
||||
dense_embeddings_enabled=config["dense"],
|
||||
@@ -203,6 +284,10 @@ def measure(home: Path, scratch: Path, objects: Path, clip_index: Path, probes:
|
||||
"commit": git_commit(),
|
||||
"schema_version": schema,
|
||||
"store": str(scratch / "data" / "ely_eye.sqlite"),
|
||||
"source_store": str(home),
|
||||
"dense_vectors": vectors,
|
||||
"picture_index": picture,
|
||||
"encode_rate": rate,
|
||||
"probes": len(answerable),
|
||||
"arms": arms,
|
||||
"machine": machine_facts(),
|
||||
@@ -247,8 +332,32 @@ def write_markdown(report: dict[str, Any], path: Path) -> None:
|
||||
"# LEGS: what each retrieval leg is worth",
|
||||
"",
|
||||
f"- commit: `{report['commit']}`, schema v{report['schema_version']}",
|
||||
f"- store: {report['store']} (a copy of {DEFAULT_HOME}; the frozen baseline is never opened)",
|
||||
(
|
||||
f"- store: {report['store']} — a copy of {report['source_store']}, which carries "
|
||||
f"{report['dense_vectors']:,} dense vectors built by bench/dense_scale. The build "
|
||||
"sits outside the measured latency, and the frozen baseline itself holds no "
|
||||
"vectors, so a dense arm there would rank an empty index."
|
||||
),
|
||||
f"- {report['probes']} answerable queries, each asked as the holder the frozen set assigns it",
|
||||
*(
|
||||
[]
|
||||
if report.get("picture_index") is None
|
||||
else [
|
||||
(
|
||||
f"- picture index: {report['picture_index']['atoms']:,} atoms at "
|
||||
f"{report['picture_index']['dim']} dimensions, "
|
||||
f"{report['picture_index']['bytes']:,} bytes, read from "
|
||||
f"{report['picture_index']['path']}"
|
||||
),
|
||||
(
|
||||
f"- encoder speed: {report['encode_rate']['per_second']:.1f} pictures/s "
|
||||
f"measured on {report['encode_rate']['pictures']} pictures, so the "
|
||||
f"{report['picture_index']['atoms']:,} pictures in that index cost about "
|
||||
f"{report['picture_index']['atoms'] / report['encode_rate']['per_second'] / 60:.1f} "
|
||||
"minutes to encode"
|
||||
),
|
||||
]
|
||||
),
|
||||
(
|
||||
f"- machine: {machine['system']} {machine['release']}, {machine['cpus']} cores, "
|
||||
f"{machine['threads_per_query']} threads per query, load {machine['load_average_1_5_15']}"
|
||||
|
||||
@@ -37,6 +37,10 @@ def test_the_report_prints_every_arm_with_its_numbers(tmp_path: Path) -> None:
|
||||
"commit": "abcdef123456",
|
||||
"schema_version": 8,
|
||||
"store": "bench/runs/leg_check/data/ely_eye.sqlite",
|
||||
"source_store": "bench/runs/dense_scale",
|
||||
"dense_vectors": 153857,
|
||||
"picture_index": {"path": "bench/runs/leg_check_index", "atoms": 53668, "dim": 512, "bytes": 109912192},
|
||||
"encode_rate": {"pictures": 200, "seconds": 10.0, "per_second": 20.0},
|
||||
"probes": 300,
|
||||
"arms": [arm],
|
||||
"machine": {
|
||||
@@ -53,4 +57,10 @@ def test_the_report_prints_every_arm_with_its_numbers(tmp_path: Path) -> None:
|
||||
assert "words only" in text
|
||||
assert "0.832" in text and "63.7" in text, "recall and latency must sit in the same row"
|
||||
assert "code" in text and "0.953" in text, "per-modality recall is where the picture legs show up"
|
||||
assert (
|
||||
"bench/runs/dense_scale" in text and "153,857 dense vectors" in text
|
||||
), "a dense arm must name the store its vectors came from, not imply the frozen baseline"
|
||||
assert (
|
||||
"53,668 atoms at 512 dimensions" in text and "44.7" in text
|
||||
), "the picture index needs its shape in the report, and its build cost derived from a measured rate"
|
||||
assert "abcdef123456" in text and "load_average" not in text
|
||||
@@ -24,5 +24,6 @@ module = [
|
||||
"sentence_transformers.*",
|
||||
"onnxruntime.*",
|
||||
"faster_whisper.*",
|
||||
"sqlite_vec",
|
||||
]
|
||||
ignore_missing_imports = true
|
||||
Reference in new issue
Block a user