Measure what each retrieval leg is worth before choosing a default

The product fuses a sparse page, a dense vector leg, and a picture leg that
only speaks when the words come back thin. Two of the three are off by
default, and until now no committed artifact said why.

bench/leg_check.py asks the frozen suite's 300 answerable queries as their
own holder, once per configuration, and prints recall, first-hit quality,
latency and per-modality recall in one table. The picture index and the
CLIP snapshot are 48 minutes of encoding, so they are carried in rather
than rebuilt per row; the harness refuses to run a leg whose snapshot is
missing instead of quietly scoring nothing.

The numbers that came back:
- words only, 10 atoms: recall@10 0.832, MRR 0.683, p50 64ms
- words only, the configured 24: 0.848, 0.688, 69ms
- words only, page 30: 0.855, 0.690, 67ms
- dense leg on: 0.845 / 0.858, MRR down, p50 469ms / 411ms
- picture leg available: 0.815 / 0.825, MRR 0.553 / 0.551, p50 148ms / 164ms

So both defaults stay off, and the acceptance bar (recall@10 >= 0.90) is
not met by the product path yet: 0.858 is the best measured row, the
in-repo counterpart at holder scope is 0.866. The loss concentrates in
picture golds, and the current CLIP leg pays more in dilution than it
returns there.

CHECKLIST p2-1 and p0-3.7 now quote the committed reports (DENSE.md,
QUANT.md, SCALE.md, TERMS.md) instead of scratch scripts.

CHECKLIST.md

@@ -75,9 +75,9 @@
   - [x] p0-3.4 七个过滤器（holder/scenario/source/modality/since/until/min_trust）+ base64 游标；`since > until` 报 "search since must not be after until"，游标坏了报 "search cursor is unreadable"
   - [x] p0-3.5 RRF（`RRF_K = 60`）取代 `0.55/0.35/0.10`；顺手删掉无人调用的 `tokenize()` / `lexical_score()` / `_graph_score()`（graph 项恒为 0，因为证据边从不参与排序）
   - [x] p0-3.6 Go 的 `search_atoms` / `compile_context` 改走 `POST /api/search`，删掉 Go 侧第二套分词器（`store.SearchAtoms` / `ftsQuery` / `tokenPattern`）——两面不可能再漂开，因为只剩一个排序实现
-  - [x] p0-3.7 100k atom 下 p50 <100ms：`python -m bench.scale_check` → `bench/reports/SCALE.md`。实测 **165,973 atom（bar 的 1.66 倍）、300 探针、p50 57ms / p95 138ms**，机器 load 1.76 / 32 核 / 每查询 4 线程；基线 p50 20.16s ⇒ **快 354 倍**。带 holder 作用域再测一遍（n=173）：recall@10 0.870、MRR 0.740、p50 62ms / p95 117ms。
+  - [x] p0-3.7 100k atom 下 p50 <100ms：`python -m bench.scale_check` → `bench/reports/SCALE.md`。实测 **165,973 atom（bar 的 1.66 倍）、300 探针、p50 57ms / p95 138ms**，机器 load 1.76 / 32 核 / 每查询 4 线程；基线 p50 20.16s ⇒ **快 343 倍**（SCALE.md 重跑：p50 58.7ms / p95 143.0ms）。同一探针集上的质量与延迟扫描见 `bench/reports/TERMS.md`（n=173 holder 作用域，0.05 档 recall@10 0.803、MRR 0.652、p50 64.8ms / p95 121.7ms）。
     - 根因不是 SQL 慢，是 OR 表达式里那些「每个文档都有」的虚词：`"parsing"* OR "error"* OR "in"* OR "the"* …` 命中 127,145 行、305ms，而中文二元组短语查询 32 命中只要 1.0ms。bm25 必须给每条命中打分，延迟就跟着命中数走。
-    - 修法：用索引自己当 IDF 源，按文档频率丢掉虚词（`text_index.rare_terms`，`COMMON_TERM_FRACTION = 0.05`，无手写停用词表，换语料自动换答案）。0.05 是扫出来的：n=173 上 0.20→0.893/112.9ms、**0.05→0.899/58.6ms**、0.02→0.870/27.2ms、0.01→0.844/13.4ms、0.005→0.783/6.5ms —— 召回峰值就在 0.05。
+    - 修法：用索引自己当 IDF 源，按文档频率丢掉虚词（`text_index.rare_terms`，`COMMON_TERM_FRACTION = 0.05`，无手写停用词表，换语料自动换答案）。0.05 是扫出来的，扫描器已入库成证据：`bench/term_sweep.py` → `bench/reports/TERMS.md`（n=173，冻结库的副本）：0.20→0.792/385.9ms、0.10→0.792/372.9ms、**0.05→0.803/64.8ms**、0.02→0.792/30.3ms、0.01→0.775/16.8ms、0.005→0.740/9.3ms —— 召回峰值就在 0.05。早先 scratch 脚本扫出的 0.893/0.899/0.870 已被取代：那些数字没有入库的产物，且是在检索改写（候选放宽规则、向量腿入库）之前测的。为此 `rare_terms(connection, terms, *, fraction=None)` 改成调用时解析上界——def 期默认值扫不动。
     - 计数缓存 `term_df`（schema v6）：一个词只数一次，任何 atom 写入都清空它，所以缓存不会比它所描述的库更老。冷缓存 p50 73ms / 热 63ms，1,324 个词被缓存。
   - 验收：`pytest backend/tests/test_text_index.py backend/tests/test_search_api.py` 绿（18 条，真 FTS5 表 + TestClient）：「钥匙」命中「我们的钥匙放在玄关抽屉里」、`search atoms` 命中 `func SearchAtoms(ctx context.Context)`、`forg` 命中 forgotten、全角 `Ｓｅａｒｃｈ　Ａｔｏｍｓ` 命中、游标翻页无重复、supersede 后旧记忆不再应答；MCP 一致性由 `ely-eye-mcp/internal/mcpserver/search_tools_test.go`（3 条）+ `bash scripts/mcp_write_smoke.sh` PASS 证明；偏差：checklist 写的 `tests/test_retrieval_parity.py` 不存在，实际在 `backend/tests/` 与 Go 测试里
 - [x] **p0-4** 卡带变成可挂载作用域
@@ -340,11 +340,22 @@
 
 ## 阶段 3 — P2：规模化、安全、分层与共享
 
-- [ ] **p2-1** 规模
-  - [ ] p2-1.1 `sqlite-vec` `vec0` 增量索引
-  - [ ] p2-1.2 MRL 截断 + int8/binary 粗排 + reranker 精排（reranker ≤2GB）
-  - [ ] p2-1.3 100k 实测；1M 外推表 + 假设
-  - 验收：100k dense 腿 p50 <100ms 实测；新 atom 立即可查有测试
+- [x] **p2-1** 规模
+  - [x] p2-1.1 `sqlite-vec` `vec0` 增量索引（`backend/src/ely_eye/vector_index.py`，schema v8）
+    - 事实：向量与 atom 同库同事务，写路径（`memory.py` / `ingestion`）在同一提交里维护它；旧的 `dense_index.npy` + `dense_index_atoms.json` 那对会腐烂的旁文件删掉了，`build_index()` 全量重编码也没了。`sync()` 只在失配时补，`sync_atoms()` 按批补删。
+    - 实测出来的 vec0 0.1.9 坑（都是跑出来的，不是猜的）：`metadata json` 不是合法表选项（只有 `metadata text`）；KNN 是 `WHERE <列> MATCH ? AND k = ?`（匹配列不是表）；int8/bit 的插入与查询向量必须套 SQL 构造器 `vec_int8(blob)` / `vec_bit(blob)`，`sqlite_vec.serialize_int8()` 返回的裸 BLOB 会被拒（`expected to be of type int8, but a float32 vector was provided`）；float32 不需要包装，也没有 `vec_float32()`；默认距离是 L2，cosine 是表选项。
+    - 偏差：0.1.9 的 vec0 **没有 ANN，逐条扫**，所以延迟与字节随规模线性——DENSE.md 把这条假设印在外推表旁边，不假装是对数。
+  - [x] p2-1.2 量化与截断：测了，没上
+    - `python -m bench.quant_probe` → `bench/reports/QUANT.md`（165,973 向量、200 探针、top 10，tie-aware 一致度）：sqlite-vec float 40.3ms/1.0、+holder 过滤 29.3ms/0.599、int8 35.3ms/**0.977**、int8+holder 23.0/0.595、bit 2.7ms/0.728、bit+holder 4.2/0.513、numpy float32 全扫（真值）8.0ms/1.0、numpy 符号位 hamming 19.1/0.011、float32 前 192/128/64 维 4.4/0.722、4.0/0.57、4.0/0.43、**bit 粗排@200 → numpy 里 float32 精排 23.7ms/0.983**。每向量字节 f32 1536 / int8 384 / bit 48。
+    - 结论：int8 省 4× 空间几乎不掉质量（0.977）但不省时间；bit 省 32× 空间和 14× 时间、只剩 0.728，只能当粗排；两级方案必须在 vec0 **外面**精排（在库里用 `rowid IN (…)` 回捞反而更慢）。MRL 截断是在**不是 MRL 训练**的 MiniLM 上测的，所以那是真实代价不是白拿；切片必须是存下来的连续维，不是查询时切（非连续视图点集会掉速度）。
+    - 偏差：精排腿（`bge-reranker-base`）早在基线消融里就有并测过（加了它 recall@10 从 0.565 掉到 0.453），所以这里没再为它编一套故事；粗排+精排这条链测出 0.983 一致度但**没接线**，因为 dense 腿默认关（见下）。
+  - [x] p2-1.3 100k 实测 + 1M 外推：`python -m bench.dense_scale` → `bench/reports/DENSE.md`（stamp cd9851a）
+    - 实测 **165,973 atom / 153,857 向量**：建库 1599.1s（96 向量/秒）、**检索 p50 74.6ms / p95 77.5ms / max 89.0ms（300 查询）**、其中编码问题 p50 11.3ms、与暴力扫描 tie-aware 一致度 **0.998**（40 探针）、**往满索引里写一条记忆 p50 514ms / p95 536ms（50 次写）**、失配扫描 1.7s（不在检索路径上）、**每向量 1,729 字节**（dbstat 数 vec0 影子表：266,059,776 B）。
+    - 1M 外推：p50 443ms / p95 463ms、向量表 1.6 GiB、首次建库 154.2 分钟、单条写 3132ms，假设随行印出（逐条扫 ⇒ 线性）。
+    - tie-aware 的必要性：语料里有重复 atom，分数并列 ⇒ top-K 的 id 集合本身有歧义，两个都对引擎按 id 比会读出 0.827 的假召回。判定改成「返回的 atom 真实 cosine ≥ 第 K 名 cosine」。
+  - 验收：100k dense 腿 p50 <100ms → **165,973 atom（bar 的 1.66 倍）p50 74.6ms**（DENSE.md）；新 atom 立即可查 → `pytest backend/tests/test_vector_index.py`（写进去当场能搜到、视图藏起来的 dense 腿也藏、备份带走向量恢复后照样应答）。
+  - 数字逼出来的决定：**dense 腿默认关**（`config.py:32 dense_embeddings_enabled=False`）。同一 300 条 holder 作用域探针上，开腿 recall@10 0.832→0.845、MRR 0.683→0.663、p50 64ms→482ms。多 1.3 个点的召回、慢 7 倍、首位质量还降——默认值就该是关，腿留着（已增量、已入库）等更大的语料或更好的嵌入模型。
+  - 顺带修的测量缺陷：`copy_store()` 只拷 SQLite，所以任何要跑视觉腿的 harness 都得自己带上 `objects/`（25 GB，探针里用符号链接）与 `cache/clip_image_index.npy`；`bench/scale_check.py` 原本直接打开冻结基线库、把它**原地迁移**了，现改成在副本上测（`--scratch`）。
 - [ ] **p2-2** 安全与隐私
   - [ ] p2-2.1 FastAPI 鉴权 + Origin 校验
   - [ ] p2-2.2 写入信任分级与隔离：未佐证不得被引用

bench/leg_check.py

@@ -0,0 +1,315 @@
+"""What each retrieval leg is actually worth, on the frozen suite, asked as its holder.
+
+The product fuses three legs with reciprocal rank fusion: a sparse page, a dense vector leg, and a
+picture leg that only speaks up when the words come back thin. Each one costs latency and a model
+snapshot, and the suite is the only honest referee. This harness turns each leg on and off over the
+same probes and prints recall, first-hit quality, latency, and per-modality recall in one table, so
+a default like `dense_embeddings_enabled=False` is a measurement rather than a preference.
+
+Run: python -m bench.leg_check [--probes 300]
+Writes: bench/reports/LEGS.md, bench/reports/leg_check.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import shutil
+import statistics
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+from bench.metrics import aggregate, score_query
+from bench.query_set import load_queries, scale_probes
+from bench.report import git_commit
+from bench.run import latency
+from bench.systems import copy_store, machine_facts
+
+DEFAULT_HOME = Path("bench/runs/dense_scale")
+DEFAULT_SCRATCH = Path("bench/runs/leg_check")
+DEFAULT_OBJECTS = Path("bench/runs/baseline/objects")
+DEFAULT_CLIP_INDEX = Path("bench/runs/leg_check_index")
+DEFAULT_CLIP_MODEL = Path("bench/models/chinese-clip-vit-base-patch16")
+DEFAULT_EMBEDDING_MODEL = Path("bench/models/paraphrase-multilingual-MiniLM-L12-v2")
+
+# The page size the product asks for by default. measure() checks it against the product's own
+# setting, so a config change here cannot quietly turn the "as shipped" row into a hypothetical.
+SHIPPED_TOP_K = 24
+
+# (label, atoms asked, dense leg, picture leg). The picture leg routes itself: it answers only when
+# the sparse page comes back thin, which is the product's rule, not this harness's.
+CONFIGS: list[dict[str, Any]] = [
+    {"label": "words only", "top_k": 10, "dense": False, "visual": False},
+    {"label": "words only (the configured 24)", "top_k": SHIPPED_TOP_K, "dense": False, "visual": False},
+    {"label": "words only, wider page", "top_k": 30, "dense": False, "visual": False},
+    {"label": "+ dense leg", "top_k": 10, "dense": True, "visual": False},
+    {"label": "+ dense leg, wider page", "top_k": 30, "dense": True, "visual": False},
+    {"label": "+ picture leg", "top_k": 10, "dense": False, "visual": True},
+    {"label": "+ picture leg, wider page", "top_k": 30, "dense": False, "visual": True},
+]
+
+
+def prepare(
+    home: Path,
+    scratch: Path,
+    objects: Path,
+    clip_index: Path,
+    *,
+    dense: bool,
+    visual: bool,
+    clip_model: Path,
+    embedding_model: Path,
+) -> Path:
+    """Build a scratch store that can run every leg: vectors, the picture index, and the bytes.
+
+    The picture index and the CLIP snapshot are 48 minutes of encoding, so they are carried in from
+    wherever they were built rather than rebuilt per row.
+    """
+    copy_home, _ = copy_store(home, scratch)
+    if visual:
+        for name in ("clip_image_index.npy", "clip_image_index_atoms.json"):
+            source = clip_index / name
+            if not source.exists():
+                raise SystemExit(
+                    f"{source}: the picture index has not been built; run `ely-eye index-visual` "
+                    "with ELY_EYE_CLIP_MODEL_DIR set, or point --clip-index at its cache"
+                )
+            (copy_home / "cache").mkdir(parents=True, exist_ok=True)
+            shutil.copy2(source, copy_home / "cache" / name)
+        if not (clip_model / "onnx" / "model_fp16.onnx").exists():
+            raise SystemExit(f"{clip_model / 'onnx' / 'model_fp16.onnx'}: the CLIP snapshot is missing")
+        if not objects.is_dir():
+            raise SystemExit(f"{objects}: the picture leg needs the object store to check image bytes")
+        link = copy_home / "objects"
+        if not link.exists() and not link.is_symlink():
+            link.symlink_to(objects.resolve())
+    if dense and not (embedding_model / "config.json").exists():
+        raise SystemExit(f"{embedding_model}: the embedding snapshot is missing; the dense leg cannot run")
+
+    return copy_home
+
+
+def _schema(home: Path) -> int:
+    import sqlite3
+
+    with sqlite3.connect(home / "data" / "ely_eye.sqlite") as connection:
+        return int(connection.execute("PRAGMA user_version").fetchone()[0])
+
+
+def rank_all(service: Any, ids: dict[str, str], probes: list[Any], top_k: int) -> tuple[list[list[str]], list[int], list[float]]:
+    """Ask the product as each probe's own holder and collapse atoms to sources by first hit."""
+    from ely_eye.schemas import SearchFilters
+
+    rankings: list[list[str]] = []
+    distinct: list[int] = []
+    timings: list[float] = []
+    for index, query in enumerate(probes):
+        began = time.perf_counter()
+        hits = service.search(query.text, top_k=top_k, filters=SearchFilters(as_holder=query.holder)).hits
+        timings.append(time.perf_counter() - began)
+        sources: list[str] = []
+        for hit in hits:
+            source = ids[hit.atom.source_id]
+            if source not in sources:
+                sources.append(source)
+        distinct.append(len(sources))
+        rankings.append(sources)
+        if (index + 1) % 100 == 0:
+            print(f"    {index + 1}/{len(probes)}", flush=True)
+    return rankings, distinct, timings
+
+
+def measure(home: Path, scratch: Path, objects: Path, clip_index: Path, probes: int, models: dict[str, Path]) -> dict[str, Any]:
+    """Score every configuration on the same probes, one warm service per configuration."""
+    from ely_eye.config import Settings
+    from ely_eye.db import Database
+    from ely_eye.retrieval import RetrievalService
+
+    shipped = Settings().retrieval_top_k
+    if shipped != SHIPPED_TOP_K:
+        raise SystemExit(
+            f"the product asks for {shipped} atoms per search but this harness pins "
+            f"SHIPPED_TOP_K={SHIPPED_TOP_K}; update the rows before trusting the 'as shipped' label"
+        )
+
+    chosen = scale_probes(load_queries(), probes)
+    answerable = [query for query in chosen if query.gold]
+    if not answerable:
+        raise SystemExit("the frozen suite carries no answerable query; nothing to score")
+    print(f"{len(answerable)} answerable probes", flush=True)
+
+    arms: list[dict[str, Any]] = []
+    schema = 0
+    for config in CONFIGS:
+        print(f"  {config['label']}: dense={config['dense']} visual={config['visual']}", flush=True)
+        copy_home = prepare(
+            home,
+            scratch,
+            objects,
+            clip_index,
+            dense=config["dense"],
+            visual=config["visual"],
+            clip_model=models["clip"],
+            embedding_model=models["embedding"],
+        )
+        schema = _schema(copy_home)
+        settings = Settings(
+            home=copy_home,
+            dense_embeddings_enabled=config["dense"],
+            embedding_model_dir=models["embedding"] if config["dense"] else None,
+            clip_model_dir=models["clip"] if config["visual"] else None,
+        )
+        database = Database(settings)
+        service = RetrievalService(settings=settings, database=database)
+        ids = _source_ids(database)
+        rankings, distinct, timings = rank_all(service, ids, answerable, int(config["top_k"]))
+        scores = [
+            score_query(query.query_id, ranking, set(query.gold), query.holder)
+            for query, ranking in zip(answerable, rankings, strict=True)
+        ]
+        summary = aggregate(scores)
+        timing = latency(timings)
+        by_modality: dict[str, list[float]] = {}
+        for query, score in zip(answerable, scores, strict=True):
+            by_modality.setdefault(query.modality, []).append(score.recall_at_10)
+        arms.append(
+            {
+                "label": config["label"],
+                "top_k": config["top_k"],
+                "dense": config["dense"],
+                "visual": config["visual"],
+                "recall_at_10": summary["recall_at_10"],
+                "mrr": summary["mrr"],
+                "hit_at_10": summary["hit_at_10"],
+                "wilson_lower": summary["hit_at_10_wilson_lower"],
+                "p50_seconds": timing["p50"],
+                "p95_seconds": timing["p95"],
+                "median_distinct_sources": statistics.median(distinct),
+                "by_modality": {
+                    modality: sum(rows) / len(rows) for modality, rows in sorted(by_modality.items())
+                },
+                "modality_counts": {modality: len(rows) for modality, rows in sorted(by_modality.items())},
+            }
+        )
+        print(
+            f"    recall@10 {summary['recall_at_10']:.3f}, mrr {summary['mrr']:.3f}, "
+            f"p50 {timing['p50'] * 1000:.1f}ms",
+            flush=True,
+        )
+    return {
+        "commit": git_commit(),
+        "schema_version": schema,
+        "store": str(scratch / "data" / "ely_eye.sqlite"),
+        "probes": len(answerable),
+        "arms": arms,
+        "machine": machine_facts(),
+    }
+
+
+def _source_ids(database: Any) -> dict[str, str]:
+    """Store source id to benchmark source id, the same mapping every other arm uses."""
+    from bench.systems import _corpus_ids
+
+    return _corpus_ids(database)
+
+
+def write_markdown(report: dict[str, Any], path: Path) -> None:
+    machine = report["machine"]
+    header = [
+        "| leg configuration | atoms asked | recall@10 | MRR | hit@10 | 95% lower | p50 ms | p95 ms | distinct sources |",
+        "| --- | --- | --- | --- | --- | --- | --- | --- | --- |",
+    ]
+    rows = []
+    for arm in report["arms"]:
+        rows.append(
+            f"| {arm['label']} | {arm['top_k']} | {arm['recall_at_10']:.3f} | {arm['mrr']:.3f} | "
+            f"{arm['hit_at_10']:.3f} | {arm['wilson_lower']:.3f} | {arm['p50_seconds'] * 1000:.1f} | "
+            f"{arm['p95_seconds'] * 1000:.1f} | {arm['median_distinct_sources']} |"
+        )
+    modalities = sorted({modality for arm in report["arms"] for modality in arm["by_modality"]})
+    modality_rows = [
+        "| leg configuration | " + " | ".join(modalities) + " |",
+        "| --- |" + " --- |" * len(modalities),
+    ]
+    for arm in report["arms"]:
+        modality_rows.append(
+            f"| {arm['label']} | "
+            + " | ".join(f"{arm['by_modality'].get(modality, float('nan')):.3f}" for modality in modalities)
+            + " |"
+        )
+    counts = report["arms"][0]["modality_counts"]
+    path.write_text(
+        "\n".join(
+            [
+                "# LEGS: what each retrieval leg is worth",
+                "",
+                f"- commit: `{report['commit']}`, schema v{report['schema_version']}",
+                f"- store: {report['store']} (a copy of {DEFAULT_HOME}; the frozen baseline is never opened)",
+                f"- {report['probes']} answerable queries, each asked as the holder the frozen set assigns it",
+                (
+                    f"- machine: {machine['system']} {machine['release']}, {machine['cpus']} cores, "
+                    f"{machine['threads_per_query']} threads per query, load {machine['load_average_1_5_15']}"
+                ),
+                "",
+                "Recall is scored on the first ten **sources** after atoms are collapsed to their source",
+                "by first hit, which is how every other arm in this repo is scored.",
+                "",
+                *header,
+                *rows,
+                "",
+                "## recall@10 by modality",
+                "",
+                *modality_rows,
+                "",
+                f"_Query counts per modality: {', '.join(f'{name} {count}' for name, count in counts.items())}._",
+                "",
+                (
+                    "The picture leg is not a switch this harness flips per row: the product runs it only "
+                    "when the sparse page comes back thin, so these rows measure the product with the leg "
+                    "available, not a forced always-on fusion."
+                ),
+                "",
+                (
+                    "_Latency is wall time per search on the machine named above with the thread budget "
+                    "the benchmark pins; a run on a loaded machine is not comparable._"
+                ),
+                "",
+            ]
+        ),
+        encoding="utf-8",
+    )
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--home", type=Path, default=DEFAULT_HOME)
+    parser.add_argument("--scratch", type=Path, default=DEFAULT_SCRATCH)
+    parser.add_argument("--objects", type=Path, default=DEFAULT_OBJECTS)
+    parser.add_argument("--clip-index", type=Path, default=DEFAULT_CLIP_INDEX)
+    parser.add_argument("--clip-model", type=Path, default=DEFAULT_CLIP_MODEL)
+    parser.add_argument("--embedding-model", type=Path, default=DEFAULT_EMBEDDING_MODEL)
+    parser.add_argument("--probes", type=int, default=300)
+    parser.add_argument("--out", type=Path, default=Path("bench/reports/LEGS.md"))
+    parser.add_argument("--json-out", type=Path, default=Path("bench/reports/leg_check.json"))
+    args = parser.parse_args()
+
+    report = measure(
+        args.home,
+        args.scratch,
+        args.objects,
+        args.clip_index,
+        args.probes,
+        {"clip": args.clip_model, "embedding": args.embedding_model},
+    )
+    args.json_out.parent.mkdir(parents=True, exist_ok=True)
+    args.json_out.write_text(json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8")
+    write_markdown(report, args.out)
+    print(f"wrote {args.out}")
+
+
+if __name__ == "__main__":
+    if sys.version_info < (3, 11):
+        raise SystemExit("bench needs Python 3.11+")
+    main()

bench/tests/test_leg_check.py

@@ -0,0 +1,56 @@
+"""The leg ablation must stay an ablation: every leg on and off, and a report that says so."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+from bench import leg_check
+
+
+def test_every_leg_appears_switched_both_ways() -> None:
+    dense = {config["dense"] for config in leg_check.CONFIGS}
+    visual = {config["visual"] for config in leg_check.CONFIGS}
+    assert dense == {True, False}, "the dense leg is only an ablation if both states are scored"
+    assert visual == {True, False}, "the picture leg is only an ablation if both states are scored"
+    pages = {int(config["top_k"]) for config in leg_check.CONFIGS}
+    assert leg_check.SHIPPED_TOP_K in pages, "the page size the product ships must be one of the rows"
+
+
+def test_the_report_prints_every_arm_with_its_numbers(tmp_path: Path) -> None:
+    arm = {
+        "label": "words only",
+        "top_k": 10,
+        "dense": False,
+        "visual": False,
+        "recall_at_10": 0.832,
+        "mrr": 0.683,
+        "hit_at_10": 0.833,
+        "wilson_lower": 0.787,
+        "p50_seconds": 0.0637,
+        "p95_seconds": 0.13,
+        "median_distinct_sources": 4,
+        "by_modality": {"code": 0.953, "pdf": 0.849},
+        "modality_counts": {"code": 128, "pdf": 86},
+    }
+    report: dict[str, Any] = {
+        "commit": "abcdef123456",
+        "schema_version": 8,
+        "store": "bench/runs/leg_check/data/ely_eye.sqlite",
+        "probes": 300,
+        "arms": [arm],
+        "machine": {
+            "system": "Linux",
+            "release": "6.18",
+            "cpus": 32,
+            "threads_per_query": 4,
+            "load_average_1_5_15": [1.0, 2.0, 3.0],
+        },
+    }
+    out = tmp_path / "LEGS.md"
+    leg_check.write_markdown(report, out)
+    text = out.read_text(encoding="utf-8")
+    assert "words only" in text
+    assert "0.832" in text and "63.7" in text, "recall and latency must sit in the same row"
+    assert "code" in text and "0.953" in text, "per-modality recall is where the picture legs show up"
+    assert "abcdef123456" in text and "load_average" not in text