Measure the last key-test scenario instead of writing it off
Scenario 7 claimed nothing was observable. It is: compiling the same question twice hands back the same pack and the second compile is served from the semantic cache (cache_events layer=semantic hit=1 miss=1), so the row is now PARTLY MEASURED with those numbers and only the time-to-first-token half stays unmeasured. Scenario 4 gains the settling step: an unresolved disagreement shows the newer side, and after resolve-conflicts one page shows both readings (measured 2 hits) — settlement is a retrieval behavior, not a deletion, which is also why my first draft of that scenario was wrong. Gaps.md §6 now carries a verdict table with what each scenario actually measured, and §0.5 quotes whole-suite recall (0.850 words-only over 2,928 answerable queries) instead of the 300-probe sample's 0.863, naming why the sample reads high. Gates: 422 passed, ruff clean, filesize OK, bash -n clean, key test 6 PASS + 1 PARTLY MEASURED with exit 0.
This commit is contained in:
3 files changed
+76
-11
No files matched your search
+4
-4
@@ -478,12 +478,12 @@
|
||||
- [x] p2-6.2 给截图一个局部的图片腿：**假设被实测推翻，默认路径不动**。池实验（`/tmp/region_clip.py`，日志 `bench/runs/region_clip.log`）：275 张 gold 截图、320 条查询、chance@10 = 0.036，三臂只比排名——屏上文字（空白分词的 BM25，比现行 CJK 词腿弱）0.584（text 135/169 = 0.799、icon 52/151 = 0.344）；整屏 CLIP 0.269（text 0.272、icon 0.265）；OCR 框裁块 CLIP（每屏取 ~36 块的最高分）0.475（text 0.663、icon 40/151 = 0.265）。读法：裁块**确实**好过整屏（0.475 vs 0.269，VISUAL.md 那条「picture 腿把 ui_screenshot 从 0.583 拉到 0.528」的整屏结论没有被推翻），但两条图片腿都打不过屏上的文字，而在真正需要图片的 icon 半边，裁块与整屏同为 0.265、文字反而最高（0.344）——**裁块没有补上 icon 的缺口**，所以不加 `bench/visual_recall.py` 的新臂、不进默认路径，按 p2-6.2 自己写的「不涨就删掉该臂并把不涨写进去」处理。口径交代：该实验在 275 张池内排名，不是 5 万源的现行排序；文字臂是空白分词，故它是被低估的基线，图片臂输给的是一个更弱的对手。
|
||||
- [ ] p2-6.3 若仍不达 0.90：把「不达」与原因写进 README/Gaps（已有的 VLM 区域描述、widget tree 两条外部能力，本轮按 m00217 未下 8B VLM），不用改验收线口径硬凑。
|
||||
- [ ] **final-accept** Gaps.md §6「钥匙测试」7 条场景逐条跑通
|
||||
- 命令：`bash scripts/key_test.sh`（自建临时 home、假模型快照、起 API、Go MCP 走 stdio、CLI 挂卡带；一张表，任何 FAIL 退出码非 0）。实测：**6 PASS + 1 NOT MEASURED**，exit 0。
|
||||
- 命令：`bash scripts/key_test.sh`（自建临时 home、假模型快照、起 API、Go MCP 走 stdio、CLI 挂卡带；一张表，任何 FAIL 退出码非 0）。实测：**6 PASS + 1 PARTLY MEASURED**，exit 0（第 7 项只测命中率，省算力未测）。
|
||||
- 1 随手记 PASS：`POST /api/memory` 一次往返 → `POST /api/search` 答出来，holder=wife origin=self scenario=home event_time=2026-09-30 citable=True。
|
||||
- 2 找老婆 PASS：MCP `who_knows(护照)` → holder=wife、confidence 0.683、basis 三条；`ask(护照在哪)`（无 token）→ `status=referred`、`answered_by=wife`、`atom_ids` 为空——**过去的是指针不是正文**；`ask(土星的光环…)` → `no_holder_knows`，不是硬编一个答案。
|
||||
- 3 知识更新 PASS：supersede 之后 `现在` → 「钥匙现在在车里」，`as_of 2026-10-06` → 「钥匙放在玄关抽屉里」，旧记忆从现行列表里消失（它那个 source 0 行）但没被删。这条是终验逼出来的新能力，见 p2-3 那条 as_of。
|
||||
- 4 分歧共存 PASS：两条日期不同的冲突记忆都在库里，一页只给新的一条（`retrieval.py:_settle_conflicts` 摘掉未裁决的旧侧），`GET /api/memory/conflicts?holder=wife` 记下双值（1 条 open），`as_of 36 小时前` 拿回 22 度那条。
|
||||
- 4 分歧共存 PASS：两条日期不同的冲突记忆都在库里，一页只给新的一条（`retrieval.py:196` `lifecycle_store.losing_sides` 摘掉未裁决的旧侧——**裁决是检索行为，不是删除**），`GET /api/memory/conflicts?holder=wife` 记下双值（1 条 open），`as_of 36 小时前` 拿回 22 度那条；`resolve-conflicts` 裁定之后一页同时给两条（实测 2 hits）。口径纠正：本轮自己写的场景预期「两条都在结果里」与既定设计相反（未裁决时只给新边），是测试错不是产品错，改的是测试。
|
||||
- 5 抗污染 PASS：`origin=third_party` 的无佐证说法可被主人搜到但 `quarantined=True`（编译器把它挡在 prompt 外，`CompiledContext.quarantined_excluded`，由 `test_trust_quarantine.py` 钉住），换个 holder 问则完全答不出。
|
||||
- 6 卡带即作用域 PASS：`ingest --cartridge-name 甲/乙` + `finalize-cartridge` + `mount/switch/unmount`，挂载后搜索只从卡带出（月岩灰 1/花岗岩 0 → 切换后 0/1 → 卸载后 1），把模型快照的 rope factor 改成 2.0 之后 `mount 甲` 拒绝并点名 `rope: factor: cartridge 4.0 vs local 2.0`（还带 model_hash）。
|
||||
- 7 卡带省算力 **NOT MEASURED**：这条断言是 TTFT 与命中率，本仓从没跑过 9B 生成器（树里没有模型快照），而且没有 `/api/compile*`、`/api/cache*` 路由（`/api/chat` 无运行时 503），所以连「命中率可观测」这半边也没有面可达。今天存在的是作用域（mount/switch/unmount/append）和 `profiles.py` 里那些尚未接地的启动旗标。
|
||||
- 未标绿的原因（两条都量着）：场景 7 未测；验收线 recall@10 ≥ 0.90 仍未达（shipped 0.863 / 300 探针，外部对手 holder 作用域 0.866，chance 0.0002；缺口全在视觉与音频腿：video 0.455、image 0.720、ui_screenshot 0.818，冻结库里 0 个 audio atom）。报告并排：`bench/reports/BASELINE.md`（本项目基线）、`BASELINE_EXTERNAL.md`（Mem0/Zep 式对手 + chance 行）、`LEGS.md`（当前各腿）、`QUANT.md`/`DENSE.md`/`SCALE.md`（向量与规模）、`VISUAL.md`（图片腿）、`ASR.md`（语音）。
|
||||
- 7 卡带省算力 **PARTLY MEASURED**：可观测那半边已实测——同一题编译两次返回同一份正文（`cache_trace_id` 相同，1 个 atom），第二次落在语义缓存（`cache_events` layer=semantic：hit=1 miss=1）。省的那笔是首字时间（TTFT），本仓从没跑过 9B 生成器（树里没有模型快照），`/api/chat` 无运行时 503，所以 TTFT 那半边仍未测。今天存在的是作用域（mount/switch/unmount/append）和 `profiles.py` 里那些尚未接地的启动旗标。
|
||||
- 未标绿的原因（两条都量着）：场景 7 的省算力半边未测；验收线 recall@10 ≥ 0.90 仍未达（**全库可答题 2,928 题实测 0.850**（words-only，top_k 24，各自 holder），dense 页 24 → 0.859、页 30 → 0.862；300 分层探针读 0.863，比全库高是因为抽样把 ui_screenshot 抽重了：样本 0.818 vs 全量 0.659；外部对手 holder 作用域 0.866，chance 0.0002；缺口全在视觉与音频腿：video 0.493、image 0.783、ui_screenshot 0.659，冻结库里 0 个 audio atom）。报告并排：`bench/reports/BASELINE.md`（本项目基线）、`BASELINE_EXTERNAL.md`（Mem0/Zep 式对手 + chance 行）、`LEGS.md`（当前各腿）、`QUANT.md`/`DENSE.md`/`SCALE.md`（向量与规模）、`VISUAL.md`（图片腿）、`ASR.md`（语音）。
|
||||
@@ -28,11 +28,11 @@
|
||||
|---|---|---|
|
||||
| 3.1 记忆模型：谁、何时、何事、对谁可见 | **闭合** | schema 已到 v12；`holder_id/origin/scenario/event_time(+precision)/valid_from/valid_to/supersedes/visibility/retention_until` 全部落库并有往返测试（`backend/tests/test_memory_schema.py`）；会话、重要度、归档层、冲突清单在 p2-3 落地（`test_memory_tiers.py`、`test_memory_conflicts.py`）；「当时为真」能问回来（`SearchFilters.as_of`，`test_as_of_search.py`） |
|
||||
| 3.2 没有写入回路 | **闭合** | `remember/supersede/forget` 一个 Python writer，HTTP/CLI/MCP 三面共用（`3d79241`）；写入需 token，第三方来源无人佐证即隔离不可引用（p2-2，`test_trust_quarantine.py`） |
|
||||
| 3.3 O(语料库) 检索 + 两套矛盾排序 | **闭合，但验收线未达** | 检索走 SQL FTS + RRF 融合（`674c88e`），向量在库内 vec0、与 atom 同一事务写入（`cd9851a`）；165,973 atoms 上 p50 **0.0717 s**，替换掉的是 **20.16 s**（`bench/reports/SCALE.md`）。recall@10 **0.863**，验收线 0.90 **未达**（`bench/reports/LEGS.md`） |
|
||||
| 3.3 O(语料库) 检索 + 两套矛盾排序 | **闭合，但验收线未达** | 检索走 SQL FTS + RRF 融合（`674c88e`），向量在库内 vec0、与 atom 同一事务写入（`cd9851a`）；165,973 atoms 上 p50 **0.0717 s**，替换掉的是 **20.16 s**（`bench/reports/SCALE.md`）。recall@10 **全库 2,928 道可答题实测 0.850**（words-only top_k 24；dense 页 24 → 0.859、页 30 → 0.862），验收线 0.90 **未达**；外部对手同批条目 holder 作用域 0.866，chance 0.0002（`bench/reports/BASELINE_EXTERNAL.md`）。注：300 分层探针读 0.863，比全库高，因为抽样偏重了 ui_screenshot（样本 0.818 vs 全量 0.659） |
|
||||
| 3.4 卡带是导出包不是卡带 | **作用域闭合，算力未闭合** | `mount/switch/unmount/append` 让挂载真的改变检索域，指纹不符拒绝挂载（`9fe13ff`）。原文里「省下真实算力（前缀/KV 复用）」**没做**：`profiles.py` 的 YaRN/量化/CPU-offload 仍是引擎启动参数，没有前缀复用实现 |
|
||||
| 3.5 没有 holder，就没有「找老婆」 | **闭合** | `who_knows` / `ask` / 目录 / 「没人知道」终态（`dfc6cd4`）；p2-4 用 token 把「只给指针」变成真边界：无 token 搜不到私钥，MCP 进程不带 token 重启一遍仍守规则（`83dfc14`、`bash scripts/mcp_write_smoke.sh` 两行 PASS） |
|
||||
| 3.6 没有工作记忆与巩固 | **闭合** | `working / consolidate / archive / archived / memory-restore / conflicts / resolve-conflicts`（p2-3） |
|
||||
| 3.7 无文字图像结构性不可见 | **部分闭合** | CLIP 图文互查（`03ca357`）与语音+词级时间戳（`9f0f799`）已实现，但实测 recall@10 仍是 video **0.455** / image **0.720** / ui_screenshot **0.818**（code 0.945、pdf 0.885、conversation 1.000），且冻结库**一个 audio atom 都没有**——视频是抽帧+OCR，靠说话回答的问题无从匹配；CLIP 也无法在整屏截图里定位一个图标 |
|
||||
| 3.7 无文字图像结构性不可见 | **部分闭合** | CLIP 图文互查（`03ca357`）与语音+词级时间戳（`9f0f799`）已实现，但全库实测 recall@10 仍是 video **0.493** / image **0.783** / ui_screenshot **0.659**（code 0.939、pdf 0.888、conversation 0.953），且冻结库**一个 audio atom 都没有**——视频是抽帧+OCR，靠说话回答的问题无从匹配；CLIP 也无法在整屏截图里定位一个图标（OCR 框裁块试过，0.475 仍输给屏上文字 0.584，见 CHECKLIST p2-6.2） |
|
||||
| 3.8 6 个 adapter 只有 1 个真被加载 | **加载闭合，效果未测** | `adapters.py` 统一加载/校验/缓存（p1-3）。9B 生成、HashHop、标准基准在本仓库**没跑过**，所以 README 与 `TRAINING_ARTIFACTS.md` 不再引用任何未跑分数 |
|
||||
| 3.9 证明外壳比能力跑得远 | **闭合** | `evals.py` 重写：跑不动的检查报告它跑不动，不再静默通过；第三方语料 50,789 sources / 5,308 queries 冻结在 `bench/sets/`，`python -m bench.validate_sets` 校验；`python -m bench.audit --product` 专抓「harness 自证」 |
|
||||
| 3.10 工程与可复现性 | **闭合** | 15 个 `.sh` 双胞胎 + Docker + Linux CI + `backup/restore/doctor`（p1-4）；每个 tracked 源文件 ≤500 行、死代码清零、注释单行（p1-6，`bench/filesize_check`） |
|
||||
@@ -250,6 +250,18 @@ frontend (React, 8 panels, 只读为主)
|
||||
6. **卡带即作用域**：`mount(cartridge_A)` 后检索只命中 A；换 mount B 结果域随之改变；指纹/签名不符 → **拒绝挂载并指名差异字段**。
|
||||
7. **卡带省算力**：同一卡带第二次提问的 TTFT 显著低于首次，且命中率是**可观测指标**（否则「卡带」只是打包格式）。
|
||||
|
||||
实测结果（`bash scripts/key_test.sh`，一张表、任何 FAIL 退出码非 0；逐条证据在 `CHECKLIST.md` 的 final-accept）：
|
||||
|
||||
| 场景 | 判定 | 测到的东西 |
|
||||
|---|---|---|
|
||||
| 1 随手记 | PASS | 一次 HTTP 往返存进去，检索答出来，带 holder=wife / origin=self / scenario=home / event_time，且 citable |
|
||||
| 2 找老婆 | PASS | MCP `who_knows(护照)` → holder=wife（带 basis/confidence）；`ask(护照在哪)` 无 token → `referred`、`answered_by=wife`、**atom_ids 为空**：过去的是指针不是正文；无人持有 → `no_holder_knows` |
|
||||
| 3 知识更新 | PASS | supersede 后「现在」答车里、「上周」答抽屉；旧条从现行列表消失但没被删 |
|
||||
| 4 分歧共存 | PASS | 两条都在库；未裁决时一页只给新边；冲突清单记下双值；`as_of` 拿回旧值；`resolve-conflicts` 之后两条都给 |
|
||||
| 5 抗污染 | PASS | 第三方说法可被本人搜到但 `quarantined=True`，编译器把它挡在正文外；换个 holder 完全答不出 |
|
||||
| 6 卡带即作用域 | PASS | 挂载后只命中卡带内容、切换随之变、卸载恢复；把快照 RoPE 改成 2.0 后拒绝挂载并点名 `factor: cartridge 4.0 vs local 2.0` |
|
||||
| 7 卡带省算力 | **PARTLY MEASURED** | 命中率半边实测：同题编译两次同一份正文（`cache_trace_id` 相同），第二次落在语义缓存（hit=1 miss=1）；TTFT 那半边要生成器，本仓没跑过 9B |
|
||||
|
||||
---
|
||||
|
||||
## 7. 建议直接删掉的东西
|
||||
|
||||
+58
-5
@@ -214,12 +214,25 @@ if "恒温器调到 24 度" not in page:
|
||||
older_then = texts(search("恒温器", as_holder="wife", as_of=(NOW - datetime.timedelta(hours=36)).isoformat())["hits"])
|
||||
if "恒温器放在书房，设定 22 度" not in older_then:
|
||||
raise SystemExit(f"FAIL: the older reading is unreachable for the time it was true: {older_then}")
|
||||
# Settling is the holder's call: until it happens retrieval shows the newer side, and once it happens both sides answer.
|
||||
from ely_eye.config import Settings
|
||||
from ely_eye.db import Database
|
||||
from ely_eye.memory import MemoryService
|
||||
|
||||
keeper = MemoryService(Settings(), Database(Settings()))
|
||||
settled = keeper.resolve_conflicts([item["conflict_id"] for item in conflicts])
|
||||
if settled != len(conflicts):
|
||||
raise SystemExit(f"FAIL: settling reported {settled}, expected {len(conflicts)}")
|
||||
both = texts(search("恒温器", as_holder="wife")["hits"])
|
||||
if not {warm["text"], hot["text"]} <= set(both):
|
||||
raise SystemExit(f"FAIL: settling did not hand both readings back to the reader: {both}")
|
||||
record(
|
||||
4,
|
||||
"PASS",
|
||||
f"two dated memories that disagree: both stay in the store (each fetched back from the store), one page shows the newer "
|
||||
"reading instead of merging them into one vague sentence, GET /api/memory/conflicts?holder=wife records the "
|
||||
f"disagreement with both values ({len(conflicts)} open), and asking as_of 36h ago returns the 22 度 reading",
|
||||
f"disagreement with both values ({len(conflicts)} open), asking as_of 36h ago returns the 22 度 reading, and "
|
||||
f"after resolve-conflicts settles that {settled} disagreement one page shows both ({len(both)} hits)",
|
||||
)
|
||||
|
||||
# 5. 抗污染: a third-party sentence cannot make itself a trusted fact.
|
||||
@@ -370,8 +383,48 @@ write_snapshot 4.0
|
||||
printf '6\tPASS\tely-eye ingest --cartridge-name 甲/乙 + finalize-cartridge + switch: mounted 甲 → 月岩灰=%s 花岗岩=%s; switch 乙 → 月岩灰=%s 花岗岩=%s; unmount → 月岩灰=%s; rewriting the model snapshot to rope factor 2.0 makes `mount 甲` refuse: %s\n' \
|
||||
"$a_only" "$b_hidden" "$a_gone" "$b_found" "$whole_a" "$moved" >> "$results"
|
||||
|
||||
# 7. 卡带省算力: the claim is a time-to-first-token saving, which needs the generator.
|
||||
printf '7\tNOT MEASURED\tno command measures it here: the saving is time-to-first-token and cache-hit rate on a real generation, and this repo has never run the 9B generator (no model snapshot in the tree). The observable half — a compile that writes a cache trace — has no route either: there is no /api/compile or /api/cache endpoint, and /api/chat returns 503 without a runtime. What exists today is the scope (ely-eye mount/switch/unmount/append) and the launch flags in profiles.py that would deliver it\n' >> "$results"
|
||||
# 7. 卡带省算力: the cache-hit half is measurable here, the time-to-first-token half is not.
|
||||
cache_stats="$(
|
||||
"$python_bin" - "车库的漆是什么颜色" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
from ely_eye.compiler import ContextCompiler
|
||||
from ely_eye.config import get_settings
|
||||
from ely_eye.db import Database
|
||||
from ely_eye.schemas import RuntimeProfile
|
||||
|
||||
database = Database(get_settings())
|
||||
compiler = ContextCompiler(get_settings(), database)
|
||||
question = sys.argv[1]
|
||||
first = compiler.compile(question, RuntimeProfile.live_demo)
|
||||
second = compiler.compile(question, RuntimeProfile.live_demo)
|
||||
with database.connect() as connection:
|
||||
rows = connection.execute(
|
||||
"SELECT event, count(*) AS n FROM cache_events WHERE layer = 'semantic' GROUP BY event"
|
||||
).fetchall()
|
||||
counts = {row["event"]: int(row["n"]) for row in rows}
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"same_pack": first.cache_trace_id == second.cache_trace_id,
|
||||
"atoms": len(first.hits),
|
||||
"hit": counts.get("hit", 0),
|
||||
"miss": counts.get("miss", 0),
|
||||
}
|
||||
)
|
||||
)
|
||||
PY
|
||||
)"
|
||||
cache_same="$(printf '%s' "$cache_stats" | "$python_bin" -c 'import json,sys;print(json.load(sys.stdin)["same_pack"])')"
|
||||
cache_atoms="$(printf '%s' "$cache_stats" | "$python_bin" -c 'import json,sys;print(json.load(sys.stdin)["atoms"])')"
|
||||
cache_hit="$(printf '%s' "$cache_stats" | "$python_bin" -c 'import json,sys;print(json.load(sys.stdin)["hit"])')"
|
||||
cache_miss="$(printf '%s' "$cache_stats" | "$python_bin" -c 'import json,sys;print(json.load(sys.stdin)["miss"])')"
|
||||
if [ "$cache_same" != "True" ] || [ "$cache_atoms" -lt 1 ] || [ "$cache_hit" -lt 1 ]; then
|
||||
echo "FAIL: the second compile did not land on the semantic cache: $cache_stats"
|
||||
exit 1
|
||||
fi
|
||||
printf '7\tPARTLY MEASURED\tthe observable half is measured: compiling the same question twice hands back the same pack (cache_trace_id equal=%s over %s atoms) and the second compile is served from the semantic cache (cache_events layer=semantic: hit=%s miss=%s). The claimed saving is time-to-first-token on a real generation, and this repo has never run the 9B generator (no model snapshot in the tree) and /api/chat returns 503 without a runtime, so that half stays unmeasured. What exists today is the scope (ely-eye mount/switch/unmount/append) and the launch flags in profiles.py that would deliver it\\n' \
|
||||
"$cache_same" "$cache_atoms" "$cache_hit" "$cache_miss" >> "$results"
|
||||
|
||||
echo
|
||||
echo "钥匙测试 — the seven scenarios from Gaps.md §6, run against one live store"
|
||||
@@ -386,5 +439,5 @@ if grep -qP '^\d+\tFAIL' "$results"; then
|
||||
echo "RESULT: at least one scenario failed"
|
||||
exit 1
|
||||
fi
|
||||
fail_count="$(grep -cP '^\d+\tNOT MEASURED' "$results" || true)"
|
||||
echo "RESULT: $fail_count scenario(s) could not be measured here; the rest passed"
|
||||
fail_count="$(grep -cP '^\d+\t(NOT MEASURED|PARTLY MEASURED)' "$results" || true)"
|
||||
echo "RESULT: $fail_count scenario(s) could not be fully measured here; the rest passed"
|
||||
Reference in new issue
Block a user