Record the memory tiers and what they cost in retrieval

The tier report is regenerated against a52c104, which carries the harness
that produced it, and now names the schema the product actually ran on: the
copy arrives at v9, the product migrates it on open, every arm ran on v10.

Every recall cell is identical across three runs on three different machine
loads (0.847 / 0.863 / 0.873 / 0.860 / 0.873 / 0.817 / 0.848), which is the
regression proof p2-3 needs: the tier code adds state and rules, not a
ranker, and this corpus holds no conflicts and no archived rows. Latency is
not comparable across those runs (p50 101.6 / 87.5 / 122.5 ms at load 27.4 /
21.4 / 17.2) and the report says so.

The memory subset is now measured whole — all 150 answerable conversation
queries instead of 15 sampled probes — because a 15-probe sample leaves one
to three queries per LongMemEval question type. It gives the two categories
the contract asks for: knowledge-update 1.000 and temporal-reasoning 0.879
as shipped, 0.950 with the dense leg. The newest-first column is the sharper
question beside them: 0.52 / 0.48 / 0.37 as shipped, so a perfect recall can
still lead with a value the speaker has since replaced.

CHECKLIST p2-3 is marked with the measured cells, the two real defects the
tests caught (a consolidation watermark that double-counted under a lagging
clock, and typer calling a variadic command callback by keyword), and the
deviation that decay is not a stored field because an unmeasured decay
formula would be a story.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-09 07:58:59 -04:00
1 parent a52c10499b
commit deabbf76b8
3 files changed
+982 -152

No files matched your search

+11 -6
View File
@@ -405,12 +405,17 @@
- **评审第 7 轮 FAIL**(同一独立 pi,8 项全跑):D22–D27 六项修法全部实测成立(含 smoke 两条路径与 CI 不受影响、摘掉虚词表的反向实验里恰好 3 条测试变红、其中包含新加那条),全套门禁绿且 mypy 与 `9cb46c1` 基线逐字相同,纪律扫描注释块归 0,p2-2 逐格复算只剩两格文字——D28:`:387` 把 `her` 列为表内词,实测 `'her' in STOPWORDS` 为假,而它恰恰**算**被要求词(`I can meet her tomorrow` → `['tomorrow','meet','her']`);D29:`:383` 把至今如此的 `ai` 槽位写成「修前的现象」,与同行前半句自相矛盾。两条已改成实测。评审另提一条精度建议(不算缺陷):`:381` 的 D15 例子带省略号不唯一可复现,已钉成完整字符串并附上加长后会变 `True` 的反例。本轮只改 `CHECKLIST.md`,代码未动;改后重跑 `pytest -q` 与 `ruff check` 仍绿(数字见上格,未变)。
- **评审第 8 轮 PASS**(同一独立 pi,6 项全跑,本轮 diff 仅 `CHECKLIST.md` +4/−3、代码 blob 全同):D28/D29 已按实测改掉并逐词复验,第 7 轮的精度建议也落实(`:381` 钉完整字符串 + 加长反例);全套门禁重测(325 passed、四文件 60 passed = 8/26/8/18、ruff、filesize、`--product` 31/2/6 clean、pyright 0、mypy 与 `9cb46c1` 基线逐字相同、go 全套、`index-CSDHCFNo.js 221.48 kB` + eslint 0);smoke **三条路径**全测:干净端口 exit 0、被 ely-eye 服务器占用 → 端口守卫 exit 1、被**非** ely-eye 的 HTTP 服务器占用(`/api/status` 404)→ 端口守卫放行但被自身进程存活检查兜住,仍 exit 1。评审另确认第 7 轮记录行引文属实(在 `d56996b` worktree 重跑摘表实验,恰好 3 条测试变红且含新加那条)。两条精度说明(不算缺陷)已采纳:`:387` 补上 `she` 与指示/疑问词一类。
- 顺带修的 harness 缺陷:`bench/audit.py --product` 还指着 `evals.py`,而证明套件在 p1-6 已拆进 `backend/src/ely_eye/proofs/`。它 fail fast 报了「not there」,但那条反自证规则其实已经**审不到产品**。改成 `product_sources(root)` 审整个包,并加测试钉住「必须有 `suite.py`、整包零 re-roll 违规」。现在 `--product` → `audited 31 harness files, 2 reports, 6 set files` / `clean`,顺带在当前树上重新证了 p0-6。
- [ ] **p2-3** 短期/长期分层
- [ ] p2-3.1 工作记忆层
- [ ] p2-3.2 巩固台账 + watermark,可重放不重复计数
- [ ] p2-3.3 `importance` / `recall_count` / `last_recalled_at` / `decay` / `archived_at`
- [ ] p2-3.4 冲突记录持久化
- 验收:LongMemEval 风格的知识更新与时间推理两类有分数;被 supersede 的旧事实不当作现行
- [x] **p2-3** 短期/长期分层
- [x] p2-3.1 工作记忆层:`working <holder> <session>` 按记录顺序返回该 session 记下的记忆,不经检索;HTTP `GET /api/memory/working` 与 MCP `working_set` 走同一份实现(Go 只转发,写规则仍在一处)。空 session 直接拒:`a working set needs a session; without one every memory is working memory`。验收:`pytest backend/tests/test_memory_tiers.py backend/tests/test_memory_cli.py backend/tests/test_memory_api.py`(12 + 14 + 8)+ `bash scripts/mcp_write_smoke.sh` 里跨进程那一段(session 名不合并两个 holder;陌生 session 返回空而不是别人的记忆)。提交 `dc9cfb5`。
- [x] p2-3.2 巩固台账:`consolidations` 表按 `(holder, scenario)` 存 watermark + rollup atom id,一条 lane 只有一份常驻摘要、原地替换;`lane_atoms` 用严格 `ingested_at > watermark`,所以 rollup 自己不会被再算一次。**实测出一个真缺陷**:`watermark_at` 原来写成本轮时钟,用落后于记忆的时钟重跑会把同一批算两遍(第二次 `atom_count=4`),现改成 `max(atom.ingested_at for atom in batch)`。验收:`test_a_lane_rolls_up_once_and_keeps_rolling_from_where_it_stopped`(第二次返回 None,第三次 `atom_count=3` 且 `rollup_atom_id` 不变)。
- [x] p2-3.3 时间/价值字段:schema v10 给 atom 加 `session_id` / `importance` / `recall_count` / `last_recalled_at` / `archived_at`;`v_atoms_current` 加 `archived_at IS NULL`,且 `apply()` 先 `DROP VIEW v_atoms_current`——`CREATE VIEW IF NOT EXISTS` 会留着旧视图体、把归档当没看见,升级测试钉住这点。**`decay` 没有落成字段**:没测过的衰减公式就是编故事,所以归档规则把条件写在命令里(`--older-than-days`、`--max-importance`、`recall_count = 0`),`importance` 是人声明的、`recall_count` 是 store 自己观察的。验收:`test_a_search_counts_the_memory_it_handed_back`、`test_a_memory_a_search_already_returned_is_not_stale`、`test_an_archived_memory_stops_answering_without_leaving_the_store`、`test_the_archive_rule_states_its_own_terms_and_can_be_read_before_it_acts`(dry-run 先点名,`atom_count` 不减)。
- [x] p2-3.4 冲突记录持久化:`conflicts` 表存(旧, 新, 共享标签, 两个值, 是否已裁决),只在同 holder 同 scenario 且**数字**变了时记录;纯措辞分歧不记(换句话就会每次触发)。检索在未裁决的对里只给新的一侧(`losing_sides`,先排宽到 `limit * CANDIDATE_WINDOW` 再裁,因为 `_rank` 内部已经截断),裁决后两侧都回来且 `valid_to` 仍为 None——裁决不等于 supersede。数字按「主语 + 最近标签」双键归一,实测依据:`恒温器放在书房,设定 22 度` 与 `恒温器调到 24 度` 只共享主语,只按最近词会键成 `书房`/`调到` 而找不到冲突。验收:`backend/tests/test_memory_conflicts.py`(10)+ MCP `memory_conflicts`(Go 4 条测试,`go test ./internal/mcpserver/` 按名跑 4 PASS)。
- 验收(原文两条):
- LongMemEval 风格两类**有分数**:`bench/reports/LEGS.md` 的 memory 表把 150 条可答 conversation 查询**整集**测了(不是 15 条抽样),shipped 配置下 knowledge-update **1.000**、temporal-reasoning **0.879**、multi-session 0.857、single-session-preference 0.800、single-session-user 0.960;开 dense 腿后 temporal-reasoning **0.950**、multi-session 0.920、preference 0.960。报告另有一列「最新的那条是否第一个回来」:words-only 0.52 / 0.48 / 0.37,dense 0.40 / 0.46 / 0.47——recall 满分不等于排序把被替换的旧值让开了位。
- 被 supersede 的旧事实不当作现行:`v_atoms_current` 过滤 `valid_to`(`test_the_current_view_hides_closed_retired_and_unbegun_memories`),Go store 读的也是这个视图(p0-2 时实测跨语言断过),smoke 里 supersede 之后旧 atom 不再出现在 `search_atoms` 结果里。
- 分层没动检索质量:同一 harness、同一 store、同一 300 探针,p2-3 前后 LEGS 每个 recall 格子**逐字相同**(0.847 / 0.863 / 0.873 / 0.860 / 0.873 / 0.817 / 0.848,memory 表同样),因为这批语料里没有冲突行也没有归档行——新增的是状态与规则,不是排序器。延迟三次分别 p50 101.6 / 87.5 / 122.5 ms(load [27.4,13.8,7.5] / [21.4,10.4,5.5] / [17.2,10.8,6.5]),报告已写明跨 run 不可比。
- 门禁:`pytest -q` 366 passed;六文件 65 passed;ruff clean;mypy 全仓 24 errors/9 files = `9cb46c1` 基线(净 0);pyright 触碰文件 0 errors;go build/vet/test + gofmt clean;`bench.audit --product` → `audited 32 harness files, 2 reports, 6 set files` / clean;`mcp_write_smoke.sh` PASS。
- 顺带:`cli.py` 495→432 行(写路径拆进 `cli_memory.py`),`bench/leg_check.py` 拆出 `bench/leg_report.py`;typer 0.27.2 用**关键字**调用回调,`*atom_ids: str` 形式的命令一调就 `TypeError`,改成 `list[str]`——这才是之前 `test_maintenance` 那条 TypeError 的真因,命令重名只是表象。报告 stamp 早于 harness 的缺陷也修了(`a52c104`)。
- [ ] **p2-4** 共享记忆
- [ ] p2-4.1 多用户/多 agent 作用域 + ACL
- [ ] p2-4.2 默认可见性:answer-only 转介