diff --git a/CHECKLIST.md b/CHECKLIST.md index b5a4f97..9bc8378 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -439,11 +439,13 @@ - 验收：`pytest backend/tests/test_holder_acl.py backend/tests/test_visibility_rule.py backend/tests/test_transactive.py` → **13 + 2 + 19** 全绿（两 holder 互不可见、无 token 读不到私钥、`everyone` 无 token 可搜、`visible_to` 只放行被指名的人、声称别人 403 且点名两人、token 只出现一次且库里只有 sha256、吊销后旧 token 401、卡无 token 可读且不含正文、陌生人的图片 404 而无主的 200、`compile(as_holder="")` 返回 0 hits 且 `cache_trace_id` 与 `as_holder="wife"` / `None` 都不同）。跨进程：`bash scripts/mcp_write_smoke.sh` → 两行 PASS，第二行是「a caller with no token sees only public memory, a holder's token sees her own, naming somebody else is refused, and the capability card answers without a token」——它先证明无 token 搜不到私钥，再用 operator token 铸出妻子的 token、证明她能搜到自己的、搜不到 `local` 写的 `the oven needs descaling`，最后**把 MCP 进程不带 `ELY_EYE_AUTH_TOKEN` 重启一遍**，证明 Go 那条路守同一条规则（不是假设）。 - 顺手修的两条自己写的断言缺陷（都是跑出来的）：smoke 里 `if secret in found(None, token=token)` 是同义反复（搜的还是那句话，当然命中），改成拿**别人写的**记忆当探针；指纹探测 `GET /api/memory/working?holder=fingerprint` 现在返回 **403**（路由在、但匿名不能声称 holder），所以探测必须带 operator token——旧树上没有这条路由仍是 404，判别力保住。 - **评审第 1 轮 FAIL**（同一独立 pi，`209f2d0..83dfc14`，736 工具轮）：**功能面它全部实测成立**，记录面 8 条对不上，已全部按实测改掉——D1 「比对用 `compare_digest`」对 holder token 是假的（库里只有摘要，`verify_token` 是 SQL 等值，`compare_digest` 只用在 operator 的明文上）；D2 指针错（`register_minimal` 在 `memory.py:94`，不是 `:120`）；D3 「先前 `as_holder or ""` 两边哈希 `""`」描述的机制从来没存在过（旧 key 根本没有身份成分）；D4 `browse.py:14` 注释说「four — two here」而实际五份、这里三份（`:52`/`:66`/`:93`）；D5 指针被 as_of 改动挪了行（`vector_index.py:288`、`store_search.py:103`）；D6 我 prompt 里写「33 条路由」而实测父提交 34 个装饰器 / 31 条不同路径、HEAD 38 / 35；D7 semgrep 那条在它的环境里无法复现（`which semgrep` / `which opengrep` 都空），已改成「当时观察到的」并把防漂移的责任交给不依赖那个工具的测试；D8 「holder 也只能是 caller 自己…由 token 保证」与实测相反（只有 operator 能写，归属来自 `default_holder`，真正的保护是 `RememberRequest` 没有 holder 字段）。它同时实测确认：三条漏在父提交上是真的、在 HEAD 关上；38 条路由里会吐正文/字节的 7 条全带眼睛；11 个攻击全关（含吊销 token 401、holder token 不能写）；三态没被照瞎（CLI 命中、`bench.leg_check --probes 20` recall@10 0.900 / p50 72.3 ms、smoke PASS）；五份副本它自己数过，把 `? <> ''` 改掉立刻让 `test_visibility_rule.py` 红；缓存 key 两种顺序四个 trace id 全不同；smoke 新增那段是真判别子（父提交树会把私钥搜出来）。另采纳四条精度说明（吊销硬拒、匿名空串 403、能力卡还带时间戳与 `token_issued`、`leg_check` 的 commit 行是 cwd 的 HEAD）。**D7 后续补上的证据**：那个门禁在本机确实跑得起来，只是不是独立 CLI——pi-lens 的 semgrep runner 在 `migrations.py:245`（`f"PRAGMA table_info({table})"`，`git show HEAD` 同位置同样存在）上 fired `python.sqlalchemy.security.sqlalchemy-execute-raw-query`，所以「插值变量会被报」不是口述，可重跑；那个站点是只拿字面量调的私有 helper。 - - **评审第 2 轮 FAIL**（同一独立 pi，`df812f3`，7 项全跑）：D1–D8 全部实测确认已修（它在 `209f2d0` worktree 里重现了旧 key 无身份成分、`git log -S` 重现了 `as_holder or ""` 只存在于 `83dfc14`）。as_of 的行为面它自己写脚本测通：退役的记忆在 `as_of` 下应答、下周的事今天不应答、归档不挡 as_of、不跨 holder 借看。`scripts/key_test.sh` 在 `df812f3` 上 exit 0（6 PASS + 1 NOT MEASURED），把 `store_search.py` 换回父提交就红在场景 3。新缺陷六条，三条行为三条记录：**D9** 那条 dense 测试名字不副实、as_of 的向量侧零覆盖（包一层计数实测 0 次调用；改 `_allowed` 的 as_of 分支、删 `retrieval.py:119` 都全绿）；**D10** 真开 dense 时两条腿不一致，词腿回旧事实、dense 腿回空，因为退役记忆根本没向量（`_expected`/追赶都读 `v_atoms_current`），commit message 里「cannot disagree」写过头；**D11** 排期到未来的 supersede 不披露且不对称：不带 as_of 的搜索 0 命中、`GET /api/atoms` total 0、dense 腿 1 行；**D12** `:430` 那个旧行号括号仍是错的；**D13** p2-5 验收 grep 实测 **2 命中**不是 0（它扫到了 CHECKLIST 自己引用的被删字符串）；**D14** `:455` 写「10 项闭合、2 项部分闭合」与 Gaps §0.5 自己那张表对不上（实为 8 / 3 / 1）。六条已全部按机制修完，见上面 p2-3 那条 as_of 与下面本轮门禁格。**读错评审答案的教训（真实代价）**：我第一次读的是它 session JSONL 里最后一个文本块的 PASS 表格，据此拆出一份错的 D-list（compare_digest / 历史行没钉提交 / 401 vs 406 / 缓存 key 后果 / video harness 写死报告），`cf2de31` 就是按那份错的清单修的——那些修法本身都成立（它们是评审的精度说明与上一轮残留），但不是本轮的 D9–D14，而且第 3 轮 prompt 已按错清单发出去了（作废重发）。以后读 verdict：先按 index 数文本块、只读它自己的「缺陷清单」那一节，不读 PASS 表。另采纳它的四条精度说明（不算缺陷）：本轮范围是 3 个 commit 不是 2；naive `--as-of` 被接受（与 `--since` 同策略，awareness 校验只在 `schemas.py:128-145` 的 `EvidenceAtom` 上，`SearchFilters` 没有）；smoke 的端口守卫只在启动前查（它曾拿到一个正在死的 server：「FAIL: the session did not reload in the order it was recorded: []」）；`browse.py` 那条子句只被 pytest 卡住（删掉它 key_test 与 smoke 都绿，因为场景 5 测的是隔离不是 ACL）。 + - **评审第 2 轮 FAIL**（同一独立 pi，`df812f3`，7 项全跑）：D1–D8 全部实测确认已修。as_of 的行为面它自己写脚本测通：退役的记忆在 `as_of` 下应答、下周的事今天不应答、归档不挡 as_of、不跨 holder 借看。`scripts/key_test.sh` 在 `df812f3` 上 exit 0（6 PASS + 1 NOT MEASURED），把 `store_search.py` 换回父提交就红在场景 3。新缺陷六条，三条行为三条记录：**D9** 那条 dense 测试名字不副实、as_of 的向量侧零覆盖（包一层计数实测 0 次调用；改 `_allowed` 的 as_of 分支、删 `retrieval.py:119` 都全绿）；**D10** 真开 dense 时两条腿不一致，词腿回旧事实、dense 腿回空，因为退役记忆根本没向量（`_expected`/追赶都读 `v_atoms_current`），commit message 里「cannot disagree」写过头；**D11** 排期到未来的 supersede 不披露且不对称：不带 as_of 的搜索 0 命中、`GET /api/atoms` total 0、dense 腿 1 行；**D12** `:430` 那个旧行号括号仍是错的；**D13** p2-5 验收 grep 实测 **2 命中**不是 0（它扫到了 CHECKLIST 自己引用的被删字符串）；**D14** `:455` 写「10 项闭合、2 项部分闭合」与 Gaps §0.5 自己那张表对不上（实为 8 / 3 / 1）。六条已全部按机制修完，见上面 p2-3 那条 as_of 与下面本轮门禁格。**读错评审答案的教训（真实代价）**：我第一次读的是它 session JSONL 里最后一个文本块的 PASS 表格，据此拆出一份错的 D-list（compare_digest / 历史行没钉提交 / 401 vs 406 / 缓存 key 后果 / video harness 写死报告），`cf2de31` 就是按那份错的清单修的——那些修法本身都成立（它们是评审的精度说明与上一轮残留），但不是本轮的 D9–D14，而且第 3 轮 prompt 已按错清单发出去了（作废重发）。以后读 verdict：先按 index 数文本块、只读它自己的「缺陷清单」那一节，不读 PASS 表。另采纳它的四条精度说明（不算缺陷）：本轮范围是 3 个 commit 不是 2；naive `--as-of` 被接受（与 `--since` 同策略，awareness 校验只在 `schemas.py:128-145` 的 `EvidenceAtom` 上，`SearchFilters` 没有）；smoke 的端口守卫只在启动前查（它曾拿到一个正在死的 server：「FAIL: the session did not reload in the order it was recorded: []」）；`browse.py` 那条子句只被 pytest 卡住（删掉它 key_test 与 smoke 都绿，因为场景 5 测的是隔离不是 ACL）。 - **第 3 轮作废重发**：`cf2de31` 那一轮是按我读错的 D-list 发的（见上面那条教训），它验的仍是记录面那五条；本轮真正的 D9–D14 已在 `eab6bd7` 之后修完，重发的 prompt 把三条行为缺陷的复现方式写死（dense 腿零覆盖的包层计数、`_expected` 的向量计数、排期 supersede 的三种读法）。 + - **评审第 3 轮 FAIL**（`cf2de31`，D15–D22，全部实测）：D15 = 我写「dense 腿也验了」时它其实是零覆盖——`test_as_of_search.py:105` 没开 `dense_embeddings_enabled`、没建 `VectorIndex`，在 `VectorIndex.search` 里加 `print("[CALL]")` 对该文件 **0 次**（`test_vector_index.py` 17 次），真开 dense 后 words 腿 `as_of=上周+1` 答抽屉、`index.search(as_of=…)` 返空，`expected_vectors` 只算出 1、`vec_atoms` 只有一行；把 `_allowed` 里 6 个 `as_of` 参数全绑成 `None` 仍 `5 passed`。⇒ 第 2 轮的 D9/D10 既没修也没记，却被写成已验。**已由 `92237c8` 修**（D9 测试真开 dense 并断言两条腿读同一窗口、D10 向量改读 `atoms`、D11 币种统一 + schema v12，五条变异各打红一条）。D16/D17 = 我上一行把功劳写错了对象：「在 `209f2d0` 重现旧 key 无身份成分」是**第 3 轮**做的，且实测比这句话多一层——`209f2d0:compiler.py:43` 根本不传 filters，那里没有「陌生人那份」可串，`compiler.py:33 if compiled.hits:` 说明空 pack 从不被端出；`as_holder or ""` 这个串**代码里从来没有**，只在 CHECKLIST 里出现过（`git log -S` 列出 `83dfc14`/`df812f3`/`cf2de31` 三个提交，都是改 CHECKLIST 的）。D18 = D-list 被换掉（已由 `92237c8` 恢复）。D19 = 「当前 411」是我自己工作树的数，被审的 `cf2de31` 是 410。D20 = 我那句「视频 harness 会拒写冻结报告」三处不实：`VIDEO_AUDIO.md`/`video_audio.json` 当时**未入库**（worktree 里根本没有可弄脏的冻结件）、worktree 跑不到写那步（死在 `no store to copy` / `123 store sources are not corpus files`）、守卫按 cwd 相对比较所以**绝对路径指向冻结件本身也能过**；且该 harness 是 `cf2de31` 自己引入的，「已修」修的是同一提交刚造的东西。D21 = `--limit 1 --probes 1`（含 `--probes 5`）死在 `aggregate needs at least one answerable query`、exit 1、零产物——冒烟旋钮在小数值时不是「少测」而是崩。D22 = 第 2 轮的 D13/D14 未动（已由 `92237c8` 修）。 + - 本轮（D15–D22 收尾）harness 三修：`refuse_frozen_overwrite` 改成比较 `resolve()` 后的目录（绝对路径指向冻结件也拒，D20）；新增 `probe_sample(probes, limit)`，`--probes 0` 关掉该臂；当 `--limit` 把样本收到只剩视频时小 count 会整块跳过该模态（评审量的 `--limit 1 --probes 1` 与 `--probes 5` 就是），此时在**二十分钟 ingest 之前**就 `SystemExit` 点名「picked no answerable video query…Pass --probes 0 to skip the probe arm, or the shipped 300」（D21：分层样本按全库跨步，小 count 会整块跳过本模态）；`rank_arm` 空样本返 `None`，`_required` 保证视频臂不会静默变成一行零，`_score_of`/markdown 打印 `skipped`。新增 3 条测试（绝对路径守卫、`probe_sample` 两态、`_required`/`_score_of`）。 - 本轮另修的两处产品事实（都是 D10/D11 带出来的）：一个还存在的记忆就保留它的向量（`vector_index.py:397`/`:451`/`:474` 改读 `atoms`），谁能看由 `_allowed` 决定；「现在开着」的谓词在视图与两条搜索腿里统一成 `valid_to IS NULL OR valid_to > now`，schema 以 **v12** 记录这次语义改变，升级时 `DROP VIEW`（`CREATE VIEW IF NOT EXISTS` 不会换身体）。`db.close_atoms` 的 `WHERE valid_to IS NULL` 不是现行读法而是写闸门（别去闭一条已经闭上的），故意保留。 - 本轮门禁（D9–D14 修完，数字钉在本轮的 commit 上）：`pytest -q` **411 passed**（+1 排期 supersede 测试；`test_superseding_moves_the_vector_rather_than_keeping_two` 改成新契约 `test_a_superseded_memory_keeps_its_vector_but_leaves_the_current_answer_alone`：vectors **2**、现行搜索只回替换者、`as_of` 下回旧者）；`ruff check backend/src/ely_eye backend/tests bench` 干净；`bench/filesize_check` 干净；`bench.audit --product` → `audited 33 harness files, 2 reports, 6 set files` / `clean`；`./node_modules/.bin/pyright` 对 6 个改动文件 → **0 errors**；mypy 冷缓存对同 6 个文件 `Success: no issues found`；五个变异 M1–M5 各自只红一条测试（见 p2-3 那条 as_of）。 - - 门禁（`83dfc14` 那轮量的；as_of 之后 406，D9–D14 修完当前 **411**）：`pytest -q` **401 passed**；`ruff check` 对 22 个改动文件干净；`bench/filesize_check` 干净；`bench.audit --product` → `audited 32 harness files, 2 reports, 6 set files` / `clean`；mypy 冷缓存 **24 errors in 9 files (checked 112 source files)**，与 `9cb46c1` 基线（24/9）**净新增 0**；`./node_modules/.bin/pyright` 对 17 个后端源文件 + 5 个测试文件 → **0 errors**；`go build` / `go vet` / `go test`（apiclient ok、mcpserver ok）+ `gofmt -l` 空；`npm run build` → `dist/assets/index-CSDHCFNo.js 221.48 kB`（bundle 哈希与 `246dae9` 相同，与「本轮没动前端」一致）+ eslint exit 0；`bash scripts/mcp_write_smoke.sh` 3.3s 两行 PASS。 + - 门禁（`83dfc14` 那轮量的 401；`eab6bd7` 加 5 条 as_of 测试后 406，`cf2de31` 加 4 条视频 harness 测试后 **410**（评审第 3 轮 D19：我原先写「当前 411」是我工作树里的数，被审 commit 是 410），`92237c8` 加 1 条排期 supersede 测试后 **411**）：`pytest -q` **401 passed**；`ruff check` 对 22 个改动文件干净；`bench/filesize_check` 干净；`bench.audit --product` → `audited 32 harness files, 2 reports, 6 set files` / `clean`；mypy 冷缓存 **24 errors in 9 files (checked 112 source files)**，与 `9cb46c1` 基线（24/9）**净新增 0**；`./node_modules/.bin/pyright` 对 17 个后端源文件 + 5 个测试文件 → **0 errors**；`go build` / `go vet` / `go test`（apiclient ok、mcpserver ok）+ `gofmt -l` 空；`npm run build` → `dist/assets/index-CSDHCFNo.js 221.48 kB`（bundle 哈希与 `246dae9` 相同，与「本轮没动前端」一致）+ eslint exit 0；`bash scripts/mcp_write_smoke.sh` 3.3s 两行 PASS。 - [x] **p2-5** 文档单一真相 - [x] p2-5.1 `AGENTS.md` 重写并停止 gitignore - `.gitignore` 删了 `AGENTS.md` 那一行（`CLAUDE.md` 仍忽略，它不是本仓库的真相源），`git check-ignore AGENTS.md` 现在 exit 1。新文件 5.5KB，写的不是风格偏好而是**这个仓库付过学费的约束**：门禁命令清单（含 mypy 基线是 24 errors in 9 files、必须在分支点的 worktree 里量而不是猜）；一个 writer 一条规则（Go 不写 SQLite）；fail fast 必须点名（引真实报错 `nothing to forget: no atom or source was given`）；≤500 行、无死代码、注释单行、docstring 是契约；「缺席是一种状态」——`v_atoms_current` 隐藏而不删除。 @@ -460,7 +462,7 @@ - 诚实处理两处派生数字：`+0.454` 不在 `BASELINE_EXTERNAL.md` 里，是它两行（0.866 / 0.412）相减得来，Gaps 里写明「该报告只给两行与‘这段差值是 holder 边界的价值’这句话，差值是这里减出来的」；ASR 速度改成当前报告的数（258.18s 音频 / 3.12s，每视频 68.4–111.2×），不再引旧报告的 63.9–112.2。 - 偏差（顺手补的真缺口）：CLI `search` 只有 `--holder`（过滤器），**没有**办法用某个 holder 的眼睛看库——即无法回答「我的 agent 会看到什么」。加了 `--as-holder`（`SearchFilters.as_holder`）与两条断言（`--as-holder stranger` → `no_match`，`--as-holder wife` → 命中），否则三态眼睛只在 HTTP/MCP 两面成立而 CLI 面缺一个口。 - 验收：`grep -rn "MMMU-Pro=\|OCRBench=\|1-hop=\|2-hop=" README.md TRAINING_ARTIFACTS.md Gaps.md AGENTS.md backend/README.md bench/README.md ely-eye-mcp/README.md` → **0 命中**（评审第 2 轮 D13：我原先写的命令带 `*.md`，会把 CHECKLIST 自己扫进去，实测 **2 命中**，两处都在本文件里——`:452` 与 `:458` 引用的正是被删掉的字符串；结论不变，命令与数字得改）。README 里每个数字都能在报告里找到（0.863→LEGS、0.866/0.412→BASELINE_EXTERNAL、0.0717/0.2177→SCALE、165973/153857/106 per second→DENSE、0.242/533/68.4–111.2→ASR、模态行→LEGS、148.9GB→Gaps 附录 B）。唯一留下的模型分数是 `TRAINING_ARTIFACTS.md:84` 的 `citation accuracy 1.00`：它在证据块里（那是当天产出的工件原文），且紧跟的段落写明了它测的是什么、不测什么。 - - 门禁（`89c880f` 那轮量的；`eab6bd7` 加了 5 条 as_of 测试后 406，D9–D14 修完当前 **411**）：`pytest -q` **401 passed**；`ruff check backend/src/ely_eye backend/tests bench` 干净；`bench.filesize_check` 干净；`bench.audit --product` → `audited 32 harness files, 2 reports, 6 set files` / `clean`；mypy 冷缓存 **24 errors in 9 files (checked 112 source files)** = `9cb46c1` 基线，净新增 0；`./node_modules/.bin/pyright backend/src/ely_eye/cli.py backend/tests/test_search_api.py` → **0 errors**；本轮未动 Go/前端/报告，所以 go/npm 套件沿用 `83dfc14` 的绿。 + - 门禁（`89c880f` 那轮量的 401；`eab6bd7` 加了 5 条 as_of 测试后 406，`cf2de31` 后 410，`92237c8` 加 1 条排期 supersede 测试后 **411**）：`pytest -q` **401 passed**；`ruff check backend/src/ely_eye backend/tests bench` 干净；`bench.filesize_check` 干净；`bench.audit --product` → `audited 32 harness files, 2 reports, 6 set files` / `clean`；mypy 冷缓存 **24 errors in 9 files (checked 112 source files)** = `9cb46c1` 基线，净新增 0；`./node_modules/.bin/pyright backend/src/ely_eye/cli.py backend/tests/test_search_api.py` → **0 errors**；本轮未动 Go/前端/报告，所以 go/npm 套件沿用 `83dfc14` 的绿。 --- diff --git a/bench/scratch.py b/bench/scratch.py index 032b5b5..3ef8495 100644 --- a/bench/scratch.py +++ b/bench/scratch.py @@ -3,10 +3,40 @@ migrates whatever database it opens.""" from __future__ import annotations +import atexit +import os import shutil import time from pathlib import Path +HELD = "held_by" + + +def _alive(pid: str) -> bool: + """A live process holds the copy. Without /proc there is nothing to check, so a stale lock cannot block a run.""" + return pid.isdigit() and Path(f"/proc/{pid}").exists() + + +def claim(scratch: Path) -> None: + """Refuse a copy another run is already using: two harnesses on one scratch overwrite each other's store mid-measurement.""" + lock = scratch / HELD + if lock.exists(): + holder = lock.read_text(encoding="utf-8").strip() + if _alive(holder) and holder != str(os.getpid()): + raise SystemExit( + f"{scratch} is held by pid {holder}; wait for that run or pass a different scratch path" + ) + scratch.mkdir(parents=True, exist_ok=True) + lock.write_text(str(os.getpid()), encoding="utf-8") + atexit.register(_release, scratch) + + +def _release(scratch: Path) -> None: + """Drop the claim, but only if it is still ours.""" + lock = scratch / HELD + if lock.exists() and lock.read_text(encoding="utf-8").strip() == str(os.getpid()): + lock.unlink(missing_ok=True) + def copy_store(home: Path, scratch: Path) -> tuple[Path, float]: """Work on a copy: the baseline store is frozen evidence and must not be migrated in place.""" @@ -15,6 +45,7 @@ def copy_store(home: Path, scratch: Path) -> tuple[Path, float]: raise SystemExit( f"{source}: no product database, build it with python -m bench.ingest_baseline" ) + claim(scratch) data = scratch / "data" data.mkdir(parents=True, exist_ok=True) began = time.perf_counter() diff --git a/bench/tests/test_scratch.py b/bench/tests/test_scratch.py new file mode 100644 index 0000000..8f5c0c9 --- /dev/null +++ b/bench/tests/test_scratch.py @@ -0,0 +1,31 @@ +"""A scratch copy is one run's working store; two runs on one path silently measure each other's data.""" + +import os +from pathlib import Path + +import pytest + +from bench import scratch + + +def test_a_live_holder_refuses_the_same_scratch(tmp_path: Path) -> None: + """The collision I hit: a second run re-copied the store under a run that was ingesting into it.""" + if not Path("/proc/1").exists(): + pytest.skip("liveness is read from /proc") + (tmp_path / scratch.HELD).write_text("1", encoding="utf-8") + with pytest.raises(SystemExit, match="is held by pid 1"): + scratch.claim(tmp_path) + + +def test_a_stale_claim_is_taken_over(tmp_path: Path) -> None: + """A crashed run must not lock the scratch away forever.""" + (tmp_path / scratch.HELD).write_text("999999", encoding="utf-8") + scratch.claim(tmp_path) + assert (tmp_path / scratch.HELD).read_text(encoding="utf-8") == str(os.getpid()) + + +def test_one_run_may_copy_more_than_once(tmp_path: Path) -> None: + """Harnesses that score several arms re-copy on purpose; that is not a collision.""" + scratch.claim(tmp_path) + scratch.claim(tmp_path) + assert (tmp_path / scratch.HELD).read_text(encoding="utf-8") == str(os.getpid()) diff --git a/bench/tests/test_video_audio_check.py b/bench/tests/test_video_audio_check.py index 6bac428..0d06950 100644 --- a/bench/tests/test_video_audio_check.py +++ b/bench/tests/test_video_audio_check.py @@ -11,8 +11,9 @@ from bench import video_audio_check def test_a_frozen_report_is_not_overwritten_by_a_harness( tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: - """A reviewer runs harnesses in a worktree of its own; the committed reports stay put.""" + """A reviewer runs harnesses in a worktree of its own; the committed evidence stays put.""" monkeypatch.setattr(video_audio_check, "PAYLOAD", tmp_path / "video_audio.json") + monkeypatch.setattr(video_audio_check, "_tracked", lambda path: True) frozen = tmp_path / "VIDEO_AUDIO.md" frozen.write_text("committed evidence", encoding="utf-8") with pytest.raises(SystemExit, match="is a frozen report"): @@ -23,6 +24,47 @@ def test_a_frozen_report_is_not_overwritten_by_a_harness( video_audio_check.refuse_frozen_overwrite(elsewhere) +def test_a_harness_may_write_its_own_untracked_output( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The default destination is the reports directory; refusing it would stop the shipped run.""" + monkeypatch.setattr(video_audio_check, "PAYLOAD", tmp_path / "video_audio.json") + monkeypatch.setattr(video_audio_check, "_tracked", lambda path: False) + fresh = tmp_path / "VIDEO_AUDIO.md" + fresh.write_text("not yet evidence", encoding="utf-8") + video_audio_check.refuse_frozen_overwrite(fresh) + + +def test_an_absolute_path_to_the_frozen_directory_is_refused_too( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The guard compares where the file really lands, not how the path was spelled.""" + monkeypatch.setattr(video_audio_check, "PAYLOAD", tmp_path / "video_audio.json") + monkeypatch.setattr(video_audio_check, "_tracked", lambda path: True) + frozen = tmp_path / "VIDEO_AUDIO.md" + frozen.write_text("committed evidence", encoding="utf-8") + with pytest.raises(SystemExit, match="is a frozen report"): + video_audio_check.refuse_frozen_overwrite(Path(frozen.resolve())) + + +def test_the_probe_arm_switches_off_rather_than_ranking_nothing() -> None: + """--probes 0 means "skip that arm"; a small count with the smoke knob means "you measured nothing".""" + assert video_audio_check.probe_sample(0, 0) == [] + # The shipped sample keeps every modality; the smoke knob narrows it to video, where a + # small count can miss the modality entirely (the reviewer's --limit 1 --probes 1 crash). + assert any(query.modality == video_audio_check.VIDEO_MODALITY for query in video_audio_check.probe_sample(300, 0)) + for probes in (1, 5): + with pytest.raises(SystemExit, match=f"--probes {probes} picked no answerable video query"): + video_audio_check.probe_sample(probes, probes) + + +def test_an_empty_video_arm_is_a_lie_not_a_zero() -> None: + """The video arm is the point of the harness; a row of zeros there must stop the run.""" + with pytest.raises(SystemExit, match="nothing to rank"): + video_audio_check._required(None, "every video question") + assert video_audio_check._score_of(None) == "skipped" + + def test_a_missing_speech_snapshot_says_so_before_ranking(tmp_path: Path) -> None: """Without a snapshot every video defers, and the after row would equal the before row.""" with pytest.raises(SystemExit, match="no speech snapshot"): diff --git a/bench/video_audio_check.py b/bench/video_audio_check.py index 4a40d8d..ba8dd8b 100644 --- a/bench/video_audio_check.py +++ b/bench/video_audio_check.py @@ -14,6 +14,7 @@ from __future__ import annotations import argparse import json import statistics +import subprocess import sys import time from pathlib import Path @@ -75,12 +76,14 @@ def speech_facts(store: Path) -> dict[str, Any]: return {"atoms": int(atoms), "characters": int(characters)} -def rank_arm(home: Path, probes: list[Any], top_k: int) -> dict[str, Any]: - """Rank the probes through the shipped words-only path and score them.""" +def rank_arm(home: Path, probes: list[Any], top_k: int) -> dict[str, Any] | None: + """Rank the probes through the shipped words-only path and score them; None skips the arm.""" from ely_eye.config import Settings from ely_eye.db import Database from ely_eye.retrieval import RetrievalService + if not probes: + return None settings = Settings( home=home, dense_embeddings_enabled=False, @@ -140,15 +143,33 @@ def add_speech(home: Path, files: list[Path], asr: Path, limit: int) -> dict[str return {"files": batches, "seconds": time.perf_counter() - began} +def probe_sample(probes: int, limit: int) -> list[Any]: + """The standard stratified sample, asked about video only when the smoke knob is on. + + A small `--probes` is not "measure a little": the sample strides the whole suite, so a + count under a few dozen can miss this modality entirely and the arm would rank nothing. + """ + if probes <= 0: + return [] + picked = [query for query in scale_probes(load_queries(), probes) if query.gold] + if limit: + picked = [query for query in picked if query.modality == VIDEO_MODALITY][:limit] + if not picked: + raise SystemExit( + f"--probes {probes} picked no answerable video query: the sample strides the whole " + "suite, so a small count can skip this modality and the arm would rank nothing. " + "Pass --probes 0 to skip the probe arm, or the shipped 300 to keep every modality." + ) + return picked + + def measure(home: Path, scratch: Path, videos: Path, asr: Path, probes: int, limit: int) -> dict[str, Any]: """Score the video questions and the standard probe sample before and after the speech pass.""" if not (home / "data" / "ely_eye.sqlite").exists(): raise SystemExit(f"{home}: no store to copy; run `python -m bench.dense_scale` first") files = guard_videos(videos, home) whole = video_queries() if not limit else video_queries()[:limit] - sample = [query for query in scale_probes(load_queries(), probes) if query.gold] - if limit: - sample = [query for query in sample if query.modality == VIDEO_MODALITY][:limit] + sample = probe_sample(probes, limit) print(f"{len(whole)} answerable video queries, {len(sample)} stratified probes, {len(files)} files", flush=True) copy_home, copied = copy_store(home, scratch) @@ -156,11 +177,11 @@ def measure(home: Path, scratch: Path, videos: Path, asr: Path, probes: int, lim before_speech = speech_facts(copy_home) if before_speech["atoms"]: print(f"note: the copy already carries {before_speech['atoms']} speech atoms", flush=True) - before_video = rank_arm(copy_home, whole, SHIPPED_TOP_K) + before_video = _required(rank_arm(copy_home, whole, SHIPPED_TOP_K), "every video question") before_sample = rank_arm(copy_home, sample, SHIPPED_TOP_K) print( f" before: video recall@10 {before_video['recall_at_10']:.3f}, " - f"probes {before_sample['recall_at_10']:.3f}", + f"probes {_score_of(before_sample)}", flush=True, ) @@ -171,11 +192,11 @@ def measure(home: Path, scratch: Path, videos: Path, asr: Path, probes: int, lim f"re-ingesting {len(files)} files added no speech atoms; the snapshot at {asr} is " "available but produced nothing, so the after row would be a measurement of nothing" ) - after_video = rank_arm(copy_home, whole, SHIPPED_TOP_K) + after_video = _required(rank_arm(copy_home, whole, SHIPPED_TOP_K), "every video question") after_sample = rank_arm(copy_home, sample, SHIPPED_TOP_K) print( f" after: video recall@10 {after_video['recall_at_10']:.3f}, " - f"probes {after_sample['recall_at_10']:.3f}, speech atoms {after_speech['atoms']}", + f"probes {_score_of(after_sample)}, speech atoms {after_speech['atoms']}", flush=True, ) return { @@ -193,6 +214,18 @@ def measure(home: Path, scratch: Path, videos: Path, asr: Path, probes: int, lim } +def _required(row: dict[str, Any] | None, label: str) -> dict[str, Any]: + """The video arm is never empty; a row of zeros there would be a silent lie.""" + if row is None: + raise SystemExit(f"{label}: nothing to rank, so this run would print a row of zeros") + return row + + +def _score_of(row: dict[str, Any] | None) -> str: + """One number for the progress line, or the reason there is none.""" + return "skipped" if row is None else f"{row['recall_at_10']:.3f}" + + def write_markdown(report: dict[str, Any], destination: Path) -> None: """Print the two rows beside each other, with the load they were measured under.""" load = report["machine"].get("load_average_1_5_15") @@ -217,15 +250,19 @@ def write_markdown(report: dict[str, Any], destination: Path) -> None: for stage in ("before", "after"): for arm, label in (("video", "every video question"), ("probes", "the stratified probe sample")): row = report[stage][arm] + if row is None: + lines.append(f"| {label} ({stage}) | 0 | skipped (--probes 0) | — | — | — | — |") + continue lines.append( f"| {label} ({stage}) | {row['queries']} | {row['recall_at_10']:.3f} | {row['mrr']:.3f} | " f"{row['hit_at_10']:.3f} | {row['wilson_lower']:.3f} | {row['p50_seconds']:.3f} |" ) gained = report["after"]["video"]["recall_at_10"] - report["before"]["video"]["recall_at_10"] - overall = report["after"]["probes"]["recall_at_10"] - report["before"]["probes"]["recall_at_10"] + probe_row = report["after"]["probes"] + overall = "not measured (--probes 0)" if probe_row is None else f"{probe_row['recall_at_10'] - report['before']['probes']['recall_at_10']:+.3f}" lines += [ "", - f"Speech moved video recall@10 by {gained:+.3f} and the whole probe sample by {overall:+.3f}.", + f"Speech moved video recall@10 by {gained:+.3f} and the whole probe sample by {overall}.", "", "Latency is read from the same pass and moves with machine load; the recall columns do not.", "", @@ -234,9 +271,21 @@ def write_markdown(report: dict[str, Any], destination: Path) -> None: destination.write_text("\n".join(lines), encoding="utf-8") +def _tracked(path: Path) -> bool: + """Only committed evidence is frozen; a harness's own untracked output is fair game.""" + return ( + subprocess.run( + ["git", "ls-files", "--error-unmatch", str(path)], capture_output=True, check=False + ).returncode + == 0 + ) + + def refuse_frozen_overwrite(destination: Path) -> None: """A reviewer running this in a worktree of its own must not overwrite the committed evidence.""" - if destination.parent == PAYLOAD.parent and destination.exists(): + if destination.exists() and destination.resolve().parent == PAYLOAD.resolve().parent and _tracked( + destination + ): raise SystemExit( f"{destination} is a frozen report; point --report and --payload somewhere else, or " "delete the frozen copy on purpose rather than under a harness"