diff --git a/.gitignore b/.gitignore index e8ad487..b03e341 100644 --- a/.gitignore +++ b/.gitignore @@ -22,7 +22,6 @@ unsloth_compiled_cache/ *.sqlite-wal *.sqlite-shm PRD.md -AGENTS.md CLAUDE.md designreference.html # Local landing site — deployed to Cloudflare, not part of this repo diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..522ee54 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,98 @@ +# Agent notes for this repository + +Read `CHECKLIST.md` for the ordered work list and `Gaps.md` for the audit the +work started from. Measured results live in `bench/reports/`. If a sentence in +this repository states a capability, there must be a command that demonstrates +it — otherwise the sentence belongs in the "not measured" section. + +## Verify before you claim + +Run the gates, and compare against the recorded baseline rather than "it built": + +```bash +pytest -q # both launch styles must work +ruff check backend/src/ely_eye backend/tests bench +.venv/bin/python -m bench.filesize_check # fails on any tracked file > 500 lines +.venv/bin/python -m bench.audit --product # the harness must not justify the product +.venv/bin/python -m mypy backend/src/ely_eye backend/tests # cold cache; baseline is 24 errors in 9 files +./node_modules/.bin/pyright # must be 0 errors on touched files +cd ely-eye-mcp && go build ./... && go vet ./... && go test ./... && gofmt -l . +cd frontend && npm run build && npx eslint src --max-warnings 0 +bash scripts/mcp_write_smoke.sh # two processes, one store +``` + +mypy carries a pre-existing error baseline. A change is clean when the count and +the file set match the recorded baseline; measure the baseline in a worktree of +the commit you branched from, do not assume it. + +## Rules this repo enforces + +- One writer per fact. Memory writes go through `MemoryService`; the Go MCP + server forwards over HTTP instead of touching SQLite, so the rules cannot + drift into two dialects. +- Fail fast and name the thing. `nothing to forget: no atom or source was given`, + `database schema is version 99, this build understands 11`. A silent default + that hides a caller's mistake is a defect, not a convenience. +- A file over 500 lines gets split before it gets commented. +- No dead code, no `v1`/`v2`/`v3` shadows of a thing that no longer exists. +- A `#` comment is one line. A docstring is a contract and may be longer. +- Absence is a state. `v_atoms_current` hides a closed, lapsed, or not-yet-valid + memory; it does not delete it, and `atom_count()` still counts it. + +## Measurement rules + +- A number that moves with machine load is quoted as a range across runs, or it + points at the report that holds it. Never quote a single load-dependent value + as if it were stable. +- A report's commit stamp must name the commit that carries the harness that + produced it. That is why a harness change and its report land as two commits: a + report cannot stamp its own commit. +- Every recall number sits next to its chance floor and, where one exists, an + external counterpart. `bench/reports/BASELINE_EXTERNAL.md` is the counterpart; + the chance row is printed in the same table. +- One sample is not a benchmark. `next(iter(stream))` with `streaming=True` was + how the old proof suite "scored" MMMU-Pro and OCRBench. +- A probe sample drawn by file order can silently drop a whole modality. + `bench/query_set.py:scale_probes` is modality-stratified for that reason, and a + test pins that every modality appears. +- A harness works on a copy of the frozen store (`bench/scratch.py:copy_store`). + Never migrate `bench/runs/baseline/data/ely_eye.sqlite` in place. +- When two correct engines disagree, suspect the metric before the engine: + byte-identical atoms produce tied scores, so id-level recall can read 0.83 for + no real reason. `bench/reports/QUANT.md` scores a hit as "any returned atom + whose true cosine is at least the K-th best". + +## Environment traps already paid for + +- pi-lens runs pyright with **0 config files**, so its `Import could not be + resolved` findings for `pydantic`, `fastapi`, `torch`, `sqlite_vec` are an + environment artifact. `./node_modules/.bin/pyright` reads `pyrightconfig.json` + and is the one that counts. +- pi-lens auto-runs a `ruff format`-style formatter on files you touch. This + repository is not ruff-format clean and its gate is `ruff check`, so revert + that drift (`git checkout --`) before committing. +- Run `ruff check --fix` on explicit paths, never on the package: it will + reformat a dozen unrelated files and you will commit them by accident. +- `bench/tests` import `bench.*`; the root `pyproject.toml` supplies + `pythonpath = ["."]`, which is why bare `pytest` works. +- `import fitz` prints a deprecation warning to **stdout**, which corrupts every + CLI JSON answer. Import it as `import pymupdf as fitz`. +- SQLite statements must be literals at the call site: the repo's semgrep rule + rejects `connection.execute(variable, …)` even when the variable is a module + constant. Where a rule must be copied, a test compares the copies + (`backend/tests/test_visibility_rule.py`). + +## Review protocol + +Every task lands through an independent reviewer running in a `herdr` pane +against the commit, not against a description of it. Findings are numbered +`D1…Dn`; fix the mechanism, not the wording, and re-measure. A checklist item is +not `[x]` before the reviewer returns PASS. Read the reviewer's verdict from its +session JSONL (the last assistant text block), not from a pane scroll-back that +may show an older round. + +## Cross-language contract + +`backend/src/ely_eye/schemas.py` is the source of truth. `ely-eye-mcp/internal/domain/types.go` +and `frontend/src/types.ts` mirror it; when a field changes there, it changes in +all three, and the smoke script is what proves the Go path agrees. diff --git a/CHECKLIST.md b/CHECKLIST.md index 0883fe9..f82474c 100644 --- a/CHECKLIST.md +++ b/CHECKLIST.md @@ -437,12 +437,23 @@ - 验收：`pytest backend/tests/test_holder_acl.py backend/tests/test_visibility_rule.py backend/tests/test_transactive.py` → **13 + 2 + 19** 全绿（两 holder 互不可见、无 token 读不到私钥、`everyone` 无 token 可搜、`visible_to` 只放行被指名的人、声称别人 403 且点名两人、token 只出现一次且库里只有 sha256、吊销后旧 token 401、卡无 token 可读且不含正文、陌生人的图片 404 而无主的 200、`compile(as_holder="")` 返回 0 hits 且 `cache_trace_id` 与 `as_holder="wife"` / `None` 都不同）。跨进程：`bash scripts/mcp_write_smoke.sh` → 两行 PASS，第二行是「a caller with no token sees only public memory, a holder's token sees her own, naming somebody else is refused, and the capability card answers without a token」——它先证明无 token 搜不到私钥，再用 operator token 铸出妻子的 token、证明她能搜到自己的、搜不到 `local` 写的 `the oven needs descaling`，最后**把 MCP 进程不带 `ELY_EYE_AUTH_TOKEN` 重启一遍**，证明 Go 那条路守同一条规则（不是假设）。 - 顺手修的两条自己写的断言缺陷（都是跑出来的）：smoke 里 `if secret in found(None, token=token)` 是同义反复（搜的还是那句话，当然命中），改成拿**别人写的**记忆当探针；指纹探测 `GET /api/memory/working?holder=fingerprint` 现在返回 **403**（路由在、但匿名不能声称 holder），所以探测必须带 operator token——旧树上没有这条路由仍是 404，判别力保住。 - 门禁：`pytest -q` **401 passed**；`ruff check` 对 22 个改动文件干净；`bench/filesize_check` 干净；`bench.audit --product` → `audited 32 harness files, 2 reports, 6 set files` / `clean`；mypy 冷缓存 **24 errors in 9 files (checked 112 source files)**，与 `9cb46c1` 基线（24/9）**净新增 0**；`./node_modules/.bin/pyright` 对 17 个后端源文件 + 5 个测试文件 → **0 errors**；`go build` / `go vet` / `go test`（apiclient ok、mcpserver ok）+ `gofmt -l` 空；`npm run build` → `dist/assets/index-CSDHCFNo.js 221.48 kB`（bundle 哈希与 `246dae9` 相同，与「本轮没动前端」一致）+ eslint exit 0；`bash scripts/mcp_write_smoke.sh` 3.3s 两行 PASS。 -- [ ] **p2-5** 文档单一真相 - - [ ] p2-5.1 `AGENTS.md` 重写并停止 gitignore - - [ ] p2-5.2 `README.md` 每条宣称指向可重跑命令 - - [ ] p2-5.3 `TRAINING_ARTIFACTS.md` 未跑分数删除 - - [ ] p2-5.4 `Gaps.md` 更新为已闭合状态 - - 验收：`grep` 无未跑模型分数残留；每条数字可重跑 +- [x] **p2-5** 文档单一真相 + - [x] p2-5.1 `AGENTS.md` 重写并停止 gitignore + - `.gitignore` 删了 `AGENTS.md` 那一行（`CLAUDE.md` 仍忽略，它不是本仓库的真相源），`git check-ignore AGENTS.md` 现在 exit 1。新文件 5.5KB，写的不是风格偏好而是**这个仓库付过学费的约束**：门禁命令清单（含 mypy 基线是 24 errors in 9 files、必须在分支点的 worktree 里量而不是猜）；一个 writer 一条规则（Go 不写 SQLite）；fail fast 必须点名（引真实报错 `nothing to forget: no atom or source was given`）；≤500 行、无死代码、注释单行、docstring 是契约；「缺席是一种状态」——`v_atoms_current` 隐藏而不删除。 + - 测量规则单独一节，都是本轮踩过的坑：**跟机器负载走的数字只能写区间或指向报告**；报告的 commit 戳必须指向含该 harness 的那个 commit（所以 harness 改动与报告是两个 commit）；每个 recall 必须与 chance 行和外部对手段并排；**n=1 不是基准**（旧套件就是靠 `next(iter(stream))` 「跑」了 MMMU-Pro 与 OCRBench）；按文件序抽样会静默丢掉整个模态（`bench/query_set.py:scale_probes` 已按模态分层并有测试钉住）；harness 只动副本（`bench/scratch.py:copy_store`），绝不原地迁移冻结库；两个正确引擎不一致时先怀疑度量（字节相同的 atom 造成并列，id 级 recall 会读出 0.83 的假差距）。 + - 环境陷阱一节：pi-lens 跑 pyright 时是 **0 config files**，所以它对 `pydantic/fastapi/torch/sqlite_vec` 的 `Import could not be resolved` 是环境伪影，算数的是读 `pyrightconfig.json` 的 `./node_modules/.bin/pyright`；pi-lens 会在你改过的文件上自动跑 `ruff format` 风格格式化，而本仓库不是 ruff-format clean、门禁是 `ruff check`，所以提交前要 `git checkout --` 回退；`ruff check --fix` 只跑显式路径；`import fitz` 把警告打到 **stdout** 会弄坏每一条 CLI JSON，必须 `import pymupdf as fitz`；semgrep 拒绝 `connection.execute(变量)`，所以规则副本必须存在、并由测试比对（`test_visibility_rule.py`）。 + - [x] p2-5.2 `README.md` 每条宣称指向可重跑命令 + - 十类旧宣称全部替换（不是加脚注）：(1)「built on Qwen3.5-9B … fifteen-check proof suite exercises the whole thing against the live model」→ 生成侧列入「**没在本仓库跑过**」清单，每项一条可重跑命令；(2)「retrieves with sparse and Qwen embedding search」→ dense 默认**关**，并给出 LEGS 实测：24 页上 0.863→0.860 而 p50 122→713ms，宽页两边都 0.873 而 p50 84→579ms，它买的是记忆类问题（multi-session 0.857→0.920、preference 0.800→0.960）、赔的是图像类（0.720→0.680）；(3)「checks every answer with a verifier」→ 引用准确性是 id 存在不是蕴含（写在 `TRAINING_ARTIFACTS.md` 那条证据旁边）；(4)「100M-token memory capsule」→ 实测 165,973 atoms / 153,857 vectors，1M 是外推不是测量；(5) HashHop / Contradiction Lens / 标准基准 → 全部进「未测」清单；(6)「nine read-only tools … never writes」→ 23 个工具，写走 API；(7)「the console … is the only place」→ CLI 做得到同样的事；(8)「Requirements: Windows」→ Linux/macOS `.sh` 双胞胎 + Docker + Linux CI；(9) `.ely_eye/` 布局与卡带内容 → 今天的真实形状（vec0 在库内、卡带是可挂载作用域）；(10) 记忆模型 / bench / ACL / 能力卡 / 验收线状态在旧 README **完全没提**，现在都在。 + - 验收线写成明字的**未达**：0.863 对 0.90，并列出缺口在哪（video 0.455 / image 0.720 / ui_screenshot 0.818，而 code 0.945 / pdf 0.885 / conversation 1.000）以及为什么（冻结库一个 audio atom 都没有；CLIP 不能在整屏里定位图标）。 + - [x] p2-5.3 `TRAINING_ARTIFACTS.md` 未跑分数删除 + - 删了：`MMMU-Pro=1.00` 与 `OCRBench=1.00`（每个基准只读了 **一个** 样本，`next(iter(stream))`）和「Visual HashHop gate passed 7/7 … 1-hop=1.00 2-hop=1.00」的 gate 口径（那个检查是**重试到达标为止**，现在重试已删）。改而写明：文件是 2026-05-28 在 Windows+4090 上从已不存在的 `PRD.md` 生成的，adapter 与所有 `.ely_eye/` 路径**不被 git 跟踪**，所以 clone 不能重现。 + - 保留但说清它到底测了什么：`citation accuracy 1.00` = 引用的 id 在送进去的上下文里存在，不是蕴含（且那道题把答案写在题面里）；100M capsule = 检查自己生成的 capsule 自哈希自洽，不是 100M 真实内容；repository input proof 是当天快照（仓库现在更大，且任何 git 仓库都能满足它）；runtime profile「verified」限定在那台机器的固定 WSL 发行版（现在读 `ELY_EYE_WSL_DISTRO`，且有 `.sh` 双胞胎与 Linux CI）；15 行 `Checks:` 块标成「当天写下的行，不是今天的结论」并指向 Gaps.md 附录 A 的逐条判定；卡带资产清单换成 `cartridge.py` 今天真正写的五个文件，并点名 `kv_snapshots.kivi` / `ttt_vl_adapter.safetensors` 是从未接上 loader 的占位。 + - [x] p2-5.4 `Gaps.md` 更新为已闭合状态 + - 新增 §0.5（不改写 §0–§4 的历史判断，只加今天复核）：12 项差距逐项一行——**10 项闭合、2 项部分闭合**，每行带跑过的命令或报告；四个根因逐一说现在为什么不成立；末尾「还剩什么」只列三件事：验收线 0.90 未达（缺口全在视觉/音频腿）、卡带的算力节省（前缀/KV 复用）没做、生成侧在本仓库一次没跑过。 + - 诚实处理两处派生数字：`+0.454` 不在 `BASELINE_EXTERNAL.md` 里，是它两行（0.866 / 0.412）相减得来，Gaps 里写明「该报告只给两行与‘这段差值是 holder 边界的价值’这句话，差值是这里减出来的」；ASR 速度改成当前报告的数（258.18s 音频 / 3.12s，每视频 68.4–111.2×），不再引旧报告的 63.9–112.2。 + - 偏差（顺手补的真缺口）：CLI `search` 只有 `--holder`（过滤器），**没有**办法用某个 holder 的眼睛看库——即无法回答「我的 agent 会看到什么」。加了 `--as-holder`（`SearchFilters.as_holder`）与两条断言（`--as-holder stranger` → `no_match`，`--as-holder wife` → 命中），否则三态眼睛只在 HTTP/MCP 两面成立而 CLI 面缺一个口。 + - 验收：`grep -rn "MMMU-Pro=\|OCRBench=\|1-hop=\|2-hop=" *.md */README.md` → **0 命中**；README 里每个数字都能在报告里找到（0.863→LEGS、0.866/0.412→BASELINE_EXTERNAL、0.0717/0.2177→SCALE、165973/153857/106 per second→DENSE、0.242/533/68.4–111.2→ASR、模态行→LEGS、148.9GB→Gaps 附录 B）。唯一留下的模型分数是 `TRAINING_ARTIFACTS.md:84` 的 `citation accuracy 1.00`：它在证据块里（那是当天产出的工件原文），且紧跟的段落写明了它测的是什么、不测什么。 + - 门禁：`pytest -q` **401 passed**；`ruff check backend/src/ely_eye backend/tests bench` 干净；`bench.filesize_check` 干净；`bench.audit --product` → `audited 32 harness files, 2 reports, 6 set files` / `clean`；mypy 冷缓存 **24 errors in 9 files (checked 112 source files)** = `9cb46c1` 基线，净新增 0；`./node_modules/.bin/pyright backend/src/ely_eye/cli.py backend/tests/test_search_api.py` → **0 errors**；本轮未动 Go/前端/报告，所以 go/npm 套件沿用 `83dfc14` 的绿。 --- diff --git a/Gaps.md b/Gaps.md index 0abfc2e..b7865ad 100644 --- a/Gaps.md +++ b/Gaps.md @@ -20,6 +20,31 @@ --- +## 0.5 现在的状态（2026-10-09 复核，逐项跑过） + +上面那三句结论在审查当天成立。经过 `CHECKLIST.md` 上的 b→P0→P1→P2 之后，**12 项差距里 10 项已闭合、2 项部分闭合**；未闭合的两项卡在同一个地方：视觉与音频腿的能力，不是架构。 + +| 差距 | 状态 | 证据（跑过的命令 / 报告） | +|---|---|---| +| 3.1 记忆模型：谁、何时、何事、对谁可见 | **闭合** | schema 已到 v11；`holder_id/origin/scenario/event_time(+precision)/valid_from/valid_to/supersedes/visibility/retention_until` 全部落库并有往返测试（`backend/tests/test_memory_schema.py`）；会话、重要度、归档层、冲突清单在 p2-3 落地（`test_memory_tiers.py`、`test_memory_conflicts.py`） | +| 3.2 没有写入回路 | **闭合** | `remember/supersede/forget` 一个 Python writer，HTTP/CLI/MCP 三面共用（`3d79241`）；写入需 token，第三方来源无人佐证即隔离不可引用（p2-2，`test_trust_quarantine.py`） | +| 3.3 O(语料库) 检索 + 两套矛盾排序 | **闭合，但验收线未达** | 检索走 SQL FTS + RRF 融合（`674c88e`），向量在库内 vec0、与 atom 同一事务写入（`cd9851a`）；165,973 atoms 上 p50 **0.0717 s**，替换掉的是 **20.16 s**（`bench/reports/SCALE.md`）。recall@10 **0.863**，验收线 0.90 **未达**（`bench/reports/LEGS.md`） | +| 3.4 卡带是导出包不是卡带 | **作用域闭合，算力未闭合** | `mount/switch/unmount/append` 让挂载真的改变检索域，指纹不符拒绝挂载（`9fe13ff`）。原文里「省下真实算力（前缀/KV 复用）」**没做**：`profiles.py` 的 YaRN/量化/CPU-offload 仍是引擎启动参数，没有前缀复用实现 | +| 3.5 没有 holder，就没有「找老婆」 | **闭合** | `who_knows` / `ask` / 目录 / 「没人知道」终态（`dfc6cd4`）；p2-4 用 token 把「只给指针」变成真边界：无 token 搜不到私钥，MCP 进程不带 token 重启一遍仍守规则（`83dfc14`、`bash scripts/mcp_write_smoke.sh` 两行 PASS） | +| 3.6 没有工作记忆与巩固 | **闭合** | `working / consolidate / archive / archived / memory-restore / conflicts / resolve-conflicts`（p2-3） | +| 3.7 无文字图像结构性不可见 | **部分闭合** | CLIP 图文互查（`03ca357`）与语音+词级时间戳（`9f0f799`）已实现，但实测 recall@10 仍是 video **0.455** / image **0.720** / ui_screenshot **0.818**（code 0.945、pdf 0.885、conversation 1.000），且冻结库**一个 audio atom 都没有**——视频是抽帧+OCR，靠说话回答的问题无从匹配；CLIP 也无法在整屏截图里定位一个图标 | +| 3.8 6 个 adapter 只有 1 个真被加载 | **加载闭合，效果未测** | `adapters.py` 统一加载/校验/缓存（p1-3）。9B 生成、HashHop、标准基准在本仓库**没跑过**，所以 README 与 `TRAINING_ARTIFACTS.md` 不再引用任何未跑分数 | +| 3.9 证明外壳比能力跑得远 | **闭合** | `evals.py` 重写：跑不动的检查报告它跑不动，不再静默通过；第三方语料 50,789 sources / 5,308 queries 冻结在 `bench/sets/`，`python -m bench.validate_sets` 校验；`python -m bench.audit --product` 专抓「harness 自证」 | +| 3.10 工程与可复现性 | **闭合** | 15 个 `.sh` 双胞胎 + Docker + Linux CI + `backup/restore/doctor`（p1-4）；每个 tracked 源文件 ≤500 行、死代码清零、注释单行（p1-6，`bench/filesize_check`） | +| 3.11 能取不能记，能看不能操作 | **闭合** | HTTP 路由 21 → **38**；CLI 命令 **59**；MCP 工具 9 只读 → **23**（写/挂载/分层/目录/能力卡） | +| 3.12 安全与隐私 | **闭合** | 写入门禁 + 每 holder 独立 token（库里只存 sha256）+ 服务器决定眼睛 + 能力卡不含正文（p2-2、p2-4） | + +四个根因现在的状态：**「命名即实现」**被 `bench.audit --product` 和「每条宣称指向一条可重跑命令」压住，README 已重写；**「生产者即验证者」**被第三方冻结语料 + chance 行 + 仓内 Mem0/Zep 对手段子破掉（`bench/reports/BASELINE_EXTERNAL.md`：同一 ranker 同一批条目，holder 作用域 0.866 对全库 0.412，**相差 0.454**——该报告只给两行与「这段差值是 holder 边界的价值、不是更好排序器的价值」这句话，差值是这里减出来的）；**「语料库而非记忆」**随 3.1/3.5/3.6 闭合不再成立；**「单用户·单轮·只读」**随 3.2/3.5/3.11/3.12 闭合不再成立。 + +**还剩什么**：验收线 0.90 未达，缺口全在视觉/音频腿；卡带的算力节省（前缀/KV 复用）没做；生成侧（9B + adapter）在本仓库一次都没跑过。§6 的钥匙测试逐条跑通是终验，未跑通不标绿。 + +--- + ## 1. 目的对齐（我理解的目标，以及「做到」长什么样） 你要的机制有两层，仓库目前只碰到了第一层的边： diff --git a/README.md b/README.md index 9e641c1..789364d 100644 --- a/README.md +++ b/README.md @@ -1,133 +1,150 @@ # Ely-Eye -Ely-Eye is a local multimodal context operating system built on Qwen3.5-9B. It -runs on a single consumer GPU and gives a small model three tiers of context: a -262K native window, a 1.01M extended window, and a 100M-token memory capsule it -can retrieve from, cite, and recall across sessions. +Ely-Eye is a local memory store for AI. Every memory it keeps carries who it +belongs to, where it came from, when it was true, in what situation, and who may +read it. Retrieval answers as the person asking, so a store shared by several +people and several agents does not blend them into one voice. A Context +Cartridge is a scope you mount, not only a bundle you export. Coding agents +reach the same store over MCP. -Under the hood it ingests PDFs, screenshots, video, and code into Evidence -Atoms, packs them into a Context Cartridge, retrieves with sparse and Qwen -embedding search, runs Qwen3.5-9B with trained LoRA adapters, and checks every -answer with a verifier. A fifteen-check proof suite exercises the whole thing -against the live model, so a real run can be told apart from a fabricated one. +The difference between this and a vector database is the referral: "I don't know +where the keys are, I know my wife knows, and I can find my wife." When the +store cannot answer, it says nobody knows and names who does — instead of +inventing a plausible answer. -The landing site lives at https://model.elydora.com/ -(its source is under `ely-eye-web/`, which this repository does not track). +Everything below is either measured in this repository, with the report and the +command that re-measures it, or it is marked as not measured. -## Requirements +## What is measured -- Windows with an NVIDIA GPU and CUDA. Training and the proof suite were built - on an RTX 4090; deployment targets a 16GB card such as an RTX 4080. -- The extended-context runtimes (SGLang, vLLM, KTransformers, llama.cpp) are - optional and run under WSL. The default Windows runtime uses Transformers and - needs none of them. +| question | answer | report | +| --- | --- | --- | +| can it find the memory that answers a question? | recall@10 **0.863**, MRR 0.682, over 300 answerable probes drawn from six third-party corpora, each asked as the holder that corpus assigns it | `bench/reports/LEGS.md` | +| is that any good? | a Mem0-style counterpart on the same corpus scores **0.866** inside one holder's memory and **0.412** across all of them; chance is 0.0002 whole-corpus and 0.369 inside one holder | `bench/reports/BASELINE_EXTERNAL.md` | +| does the holder boundary matter? | **+0.454 recall@10** with an identical ranker and identical items — the gain is scoping, not ranking | same report | +| how fast at corpus size? | p50 **0.072 s**, p95 0.218 s over 165,973 atoms; the search this replaced took 20.16 s on the same corpus | `bench/reports/SCALE.md` | +| how much does it hold? | 165,973 atoms and 153,857 vectors in one SQLite file; the vector build runs 106 memories/s. One million is an extrapolation, not a measurement | `bench/reports/DENSE.md` | +| can a wordless picture be found? | yes for icon crops, weakly for whole screenshots — the report shows every arm, including the ones that hurt | `bench/reports/VISUAL.md` | +| what does speech cost? | WER **0.242** over 533 reference words; 258.18 s of audio transcribed in 3.12 s (68.4–111.2× realtime per video, CPU int8), byte-identical across two passes | `bench/reports/ASR.md` | +| is the acceptance bar met? | **no.** The bar is recall@10 ≥ 0.90; shipped is 0.863 | `bench/reports/LEGS.md` | -## Run it locally +Where the deficit lives, by modality (recall@10, same run): code 0.945, +conversation 1.000, pdf 0.885, ui_screenshot 0.818, image 0.720, video 0.455. +Video is the worst because the frozen stores hold no audio atoms — a video is +sampled frames plus OCR, so a question answered by speech has nothing to match. +Images and screenshots are limited because CLIP cannot localize an icon inside a +screenshot. `bench/reports/LEGS.md` names the arms; nothing here rounds those +numbers up. -```powershell -.\scripts\install.ps1 # virtual env, PyTorch + CUDA, Transformers, PEFT, bitsandbytes -.\scripts\download-models.ps1 # Qwen3.5-9B, the 0.8B planner, Qwen3-VL-Embedding-8B -.\scripts\serve-backend.ps1 # API on http://127.0.0.1:8765 -.\scripts\serve-frontend.ps1 # dashboard on http://127.0.0.1:5173 +Re-measure any of them: + +```bash +python -m bench.validate_sets # the frozen corpus and its gold +python -m bench.leg_check --probes 300 # the shipped retrieval path +python -m bench.run_external # the counterparts and the chance floor +python -m bench.dense_scale # the vector leg at corpus size ``` -When `ELY_EYE_ADAPTER_DIR` is empty, the runtime picks the latest trained -HME-Core-LoRA adapter for `Qwen/Qwen3.5-9B` on its own; `ely-eye status` prints -which adapter it resolved. +## What is not measured here -For the 262K and 1.01M extended-context profiles, set up the Linux runtimes once -and launch one: +The generator side is written and runnable but has not been run in this +repository's reports, so no score from it is quoted anywhere in this README: -```powershell -.\scripts\setup-linux-runtimes.ps1 -.\scripts\launch-sglang-live.ps1 # 262K -.\scripts\launch-sglang-extreme.ps1 # 1.01M via YaRN +- generation with Qwen3.5-9B and the trained LoRA adapters — `ely-eye runtime-generation-proof` +- Visual HashHop multi-hop visual addressing — `ely-eye hashhop --kind visual` +- the Visual Contradiction Lens — `ely-eye visual-contradiction-proof` +- standard benchmarks (MMMU-Pro, OCRBench, and the rest) — `ely-eye run-benchmark` +- the fifteen-check proof suite as a whole — `ely-eye proof-suite` + +Each needs a CUDA machine and a model download. They are marked DEFERRED in +`CHECKLIST.md`, and a run that has not happened is not allowed to look like a +result: `ely-eye proof-suite` writes what it observed, and the checks that +depend on a model report what they could not do rather than passing quietly. + +`TRAINING_ARTIFACTS.md` keeps the six-adapter training table, which is +artifact-backed on the machine that trained them, and no longer quotes benchmark +scores that were never run. + +## What a memory carries + +`holder_id`, `origin` (self / observed / artifact / third_party), `scenario`, +`event_time` with its precision and how it was learned, `valid_from` / +`valid_to`, `supersedes`, `visibility` (holder / visible_to / everyone), +`retention_until`, plus ingestion time and trust scores. Ingestion stamps +provenance; a client cannot claim to be somebody else, because the server +decides who a caller is from the token it presents. + +Four verbs, one writer: `remember`, `supersede`, `forget`, `search`. They exist +over HTTP, the CLI, and MCP, and they all go through the same Python writer, so +the rules cannot drift apart. `forget` cascades to everything derived — vectors, +FTS rows, edges, cache entries — and reports which cartridges contained the +memory rather than silently rewriting an export whose digest somebody already +quoted. + +On top of that: a working tier (what one session recorded, in order), +consolidation into a standing summary, an archive tier for memories that are +neither used nor valued, and a conflict list for dated memories that disagree +and that nobody has settled. A third-party memory that no other memory vouches +for is quarantined: it stays in the store, and it is not citable. + +## Run it + +Linux and macOS: + +```bash +bash scripts/install.sh +bash scripts/serve-backend.sh # API on http://127.0.0.1:8765 +bash scripts/serve-frontend.sh # dashboard on http://127.0.0.1:5173 ``` -## Core commands +Windows: the same names as `.ps1`. There is also a container image +(`docker build -t ely-eye .`), and CI runs the whole suite on Linux. -```powershell -.\.venv\Scripts\ely-eye.exe status -.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project -.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144 -.\.venv\Scripts\ely-eye.exe proof-suite +```bash +ely-eye ingest ~/some/package --cartridge-name project +ely-eye search "where did we put the archive key" --as-holder wife +ely-eye remember "the archive key is in the second drawer" --holder wife --scenario home +ely-eye who-knows "archive key" +ely-eye ask "where is the archive key" --as-holder husband +ely-eye card +ely-eye mounts && ely-eye mount project && ely-eye unmount +ely-eye backup ./backup && ely-eye doctor +bash scripts/mcp_write_smoke.sh # two processes, one store, one SQLite ``` -The console is served by the API itself once `ELY_EYE_UI_DIR` points at -`frontend/dist`. It shows the state the CLI prints — runtime, GPU, library, -cartridges, senses, adapters, proof suite — and it is the only place to search -the store as somebody, page through atoms, forget one, and mount a cartridge. -`scripts/ui_smoke.sh` drives that console in a real browser against a fresh -store and fails if any of it stops working. +The dashboard is served by the API itself once `ELY_EYE_UI_DIR` points at +`frontend/dist`. `scripts/ui_smoke.sh` drives it in a real browser against a +fresh store and fails if any of it stops working. The CLI does everything the +console does, so nothing requires a browser. -## Fine-tuning +## Retrieval -Training runs real BF16 LoRA updates on the GPU through Unsloth-patched -Transformers, PEFT, and bitsandbytes with paged 8-bit AdamW. Each run writes the -PEFT adapter alongside a manifest, a per-step training trace, a summary, and a -proof file that records dataset and weight hashes, per-step loss, optimizer-update -norms, and the CUDA/BF16 environment. - -The adapter set: - -```text -HME-Core-LoRA Qwen3.5-9B attention/MLP -HME-Vision-LoRA Qwen3.5-9B visual projector and selected vision layers -HME-TTT-VL Qwen3.5-9B mutable MLP and visual adapter -HME-Visual-MTP Qwen3.5-9B draft-head LoRA -HME-Router Qwen3.5-0.8B context planner -HME-Retrieval Qwen3-VL-Embedding-8B contrastive retrieval adapter -``` - -## Proof suite - -`ely-eye proof-suite` runs fifteen checks against the live model and writes the -evidence under `.ely_eye/data/eval_proofs/`. Among them: - -- real CUDA BF16 LoRA training — changed weight hashes, nonzero optimizer - updates, per-step loss -- the runtime loading the model and adapter and generating a cited answer -- Visual HashHop: multi-hop visual addressing the model has to actually solve, - with no OCR shortcut -- the 100M-token memory capsule committing to a reproducible Memory DNA -- the Visual Contradiction Lens localizing cross-version drift (design-token, - layout, typography, copy, visual-code, temporal) at >=80% recall with no false - positives -- standard benchmark samples (MMMU-Pro, OCRBench, and others) scored against - ground truth - -```powershell -.\.venv\Scripts\ely-eye.exe proof-suite -.\.venv\Scripts\ely-eye.exe latest-proof-suite -.\.venv\Scripts\ely-eye.exe runtime-generation-proof -.\.venv\Scripts\ely-eye.exe visual-contradiction-proof -``` +Three legs, fused by rank. The words leg always runs. The dense leg runs only +when `ELY_EYE_DENSE_EMBEDDINGS_ENABLED=1` and an embedding model is configured. +On the shipped page that leg moves recall from 0.863 to 0.860 while p50 goes +from 122 ms to 713 ms; on a wider page both reach 0.873 while p50 goes from +84 ms to 579 ms. It buys the memory questions (multi-session 0.857 → 0.920, +preference 0.800 → 0.960) and costs the picture ones (image 0.720 → 0.680), +which is why it is opt-in rather than the default. The picture leg runs only +when the words page comes back thin and `ELY_EYE_CLIP_MODEL_DIR` is set. +Vectors live in the database beside the atoms, written in the same transaction +as the memory itself, so a memory is findable the moment it is stored and there +is no index to rebuild. ## MCP server for coding agents -`ely-eye-mcp/` is a local Model Context Protocol server that exposes the -`.ely_eye` memory store to coding agents such as Codex and Claude Code. One -command builds a native Go binary (or downloads a prebuilt one), installs it -under `.ely_eye/bin/`, and registers it with both clients. No Docker. - -Windows: - -```powershell -powershell -ExecutionPolicy Bypass -File .\ely-eye-mcp\install.ps1 -Client both -``` - -Linux and macOS: +`ely-eye-mcp/` is a Go MCP server. One command builds a native binary and +registers it with Codex and Claude Code. Reads mirror the store; writes go +through the API, so the write rules live in one place. A read with no +`ELY_EYE_AUTH_TOKEN` sees only memories nobody owns and memories marked for +everyone; `ely_eye_card` answers "can this store answer my kind of question" +before anything sensitive is sent, and it carries no memory text. ```bash bash ./ely-eye-mcp/install.sh --client both ``` -The agent then has nine read-only tools over the live memory — status, -cartridge listing and detail, Evidence Atom search and fetch, evidence-pack -compilation, proof suites, and HashHop proofs. The server opens the database -read-only and never writes to it. The same installer works against any -`.ely_eye` store, not only this repository. See `ely-eye-mcp/README.md` for the -full tool list and the release pipeline. +See `ely-eye-mcp/README.md` for the full tool list. ## Data layout @@ -135,27 +152,31 @@ Runtime state lives under `.ely_eye/` (git-ignored): ```text .ely_eye/ -├─ adapters/ trained LoRA adapters -├─ data/ sqlite database and eval proofs -├─ training/ datasets and training traces +├─ data/ the SQLite store: atoms, FTS, vectors (vec0), edges, mounts ├─ objects/ ingested source files -├─ cache/ cache fabric -└─ cartridges/ materialized Context Cartridges +├─ cache/ compiled-context cache and the picture index +├─ cartridges/ materialized Context Cartridges +├─ adapters/ trained LoRA adapters +└─ training/ datasets and training traces ``` -A Context Cartridge is a self-contained memory package: the Evidence Atoms, a -sparse index and vector indexes, a temporal graph, KV snapshots, the memory -capsule, any attached adapters, and the eval proofs. +A Context Cartridge is a self-contained memory package: the Evidence Atoms, the +sparse and vector indexes, the temporal graph, and a DNA digest of the files. +Mounting one scopes every subsequent search to it; switching is one command. ## Project layout ```text -backend/ Python: ingestion, retrieval, runtime, training, proof suite, FastAPI -frontend/ the local dashboard (Vite + React) -scripts/ install, model download, serving, runtimes, fine-tuning (.ps1 on Windows, .sh on Linux) -ely-eye-mcp/ Go: local MCP server exposing the memory store to coding agents +backend/ Python: ingestion, retrieval, memory, runtime, training, FastAPI +bench/ the benchmark suite: frozen third-party corpora, harnesses, reports +frontend/ the local dashboard (Vite + React) +scripts/ install, serving, runtimes, smoke (.sh on Linux/macOS, .ps1 on Windows) +ely-eye-mcp/ Go: the MCP server ``` +`CHECKLIST.md` is the ordered work list with the acceptance line for every item; +`Gaps.md` is the audit this work started from. + ## License AGPL-3.0-only. Copyright © 2026 雷电芽衣 . diff --git a/TRAINING_ARTIFACTS.md b/TRAINING_ARTIFACTS.md index 2c9fee9..735160d 100644 --- a/TRAINING_ARTIFACTS.md +++ b/TRAINING_ARTIFACTS.md @@ -1,5 +1,21 @@ # Ely-Eye Training Artifacts +## What this file is, and what it is not + +It is a record of what was produced on **2026-05-28** on the Windows + RTX 4090 +machine that trained the adapters, from a `PRD.md` that no longer exists in this +repository. The adapters themselves, the cartridge, and every `.ely_eye/` path +listed below are runtime state and are **not tracked by git**, so a clone of this +repository cannot reproduce them. + +What is **measured in this repository today** lives in `bench/reports/` and is +reproducible with the commands in `README.md`. Nothing in this file belongs to +that set. Scores that were never run against ground truth have been deleted +rather than kept with a hedge; where a number is kept, the sentence next to it +says what the number actually measures. + +## Artifacts + Generated on 2026-05-28 from the local `PRD.md`, ingested Evidence Atom `PRD_0b88b12114634a56:text`, visual Evidence Atom `ely_eye_dashboard_0180927ffdd3f8b0:image`, local UI proof images, and Qwen @@ -68,6 +84,11 @@ Runtime generation proof: transformers generated 218 tokens with hme_core_lora citation accuracy 1.00 ``` +"Citation accuracy 1.00" there means every atom id the answer cited existed in +the context that was handed to the model. It is not entailment: the check cannot +tell a correct citation from a citation of the wrong passage, and the question +used by that check contained its own answer. + 100M memory capsule proof: ```text @@ -75,6 +96,13 @@ transformers generated 218 tokens with hme_core_lora citation accuracy 1.00 .ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/memory_capsule_index.json .ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/memory_capsule.json The Context Cartridge materialized a 100,000,000 token-equivalent memory capsule with 4,096 committed segments and a verified segment Merkle root. + +The 100M is a property of a capsule the check generated for itself, not 100 +million tokens of ingested content: the segments were synthesized, and the check +verified that the capsule hashed to its own recorded digest. What the store +actually holds today is measured in `bench/reports/DENSE.md` (165,973 atoms, +153,857 vectors), and `Gaps.md` appendix B prices one million tokens of live KV +cache at 148.9 GB in fp16. ``` Model-solved HashHop proofs: @@ -87,7 +115,12 @@ Model-solved HashHop proofs: .ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/4f24efd13342694c/model_visual_hashhop.json .ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/a3f49402ef2710a3/model_visual_hashhop.json .ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/0bd17526ab51392b/model_visual_hashhop.json -Qwen/Qwen3.5-9B with HME-Core-LoRA solved a 1.01M token-equivalent 2-hop text HashHop and 6 Visual HashHop arenas at 262K token-equivalent. The Visual HashHop gate passed 7/7 model-solved proofs, with 262K 1-hop=1.00 and 2-hop=1.00. +Qwen/Qwen3.5-9B with HME-Core-LoRA solved a 1.01M token-equivalent 2-hop text HashHop and 6 Visual HashHop arenas at 262K token-equivalent. + +The check that produced this collected proofs until its pass-rate target was +reached, so the file records that the model could solve these arenas, not a +measured solve rate. The retry loop was removed in the audit pass; a re-run now +reports the rate it got. ``` Benchmark source and sample proof: @@ -96,7 +129,14 @@ Benchmark source and sample proof: .ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/standard_benchmark_sources.json .ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/standard_benchmark_samples.json .ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/standard_benchmark_predictions.json -MMMU-Pro, OCRBench, MMLongBench-Doc, VideoMME, LongVideoBench, RULER, and LongBench v2 loaded official streaming samples or official public metadata samples with sample hashes and source dataset SHA values. LongVideoBench uses `longvideobench/LongVideoBench-Meta` validation metadata with canonical `longvideobench/LongVideoBench` registry tracking. MMMU-Pro and OCRBench generated local Qwen/Qwen3.5-9B + HME-Core-LoRA predictions from official image samples, with local sample scores MMMU-Pro=1.00 and OCRBench=1.00. +MMMU-Pro, OCRBench, MMLongBench-Doc, VideoMME, LongVideoBench, RULER, and LongBench v2 loaded official streaming samples or official public metadata samples with sample hashes and source dataset SHA values. LongVideoBench uses `longvideobench/LongVideoBench-Meta` validation metadata with canonical `longvideobench/LongVideoBench` registry tracking. + +No benchmark score is quoted here. The prediction step read **one** sample per +benchmark (`next(iter(stream))` with `streaming=True`), so MMMU-Pro and OCRBench +produced a single local prediction each; a score computed from one sample is not +a benchmark result and the two `=1.00` lines that used to sit in this file have +been deleted. `bench/reports/` is where this repository +reports measured numbers, on corpora with hundreds of queries. ``` Repository input proof: @@ -104,6 +144,10 @@ Repository input proof: ```text .ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/repository_input.json The repository input proof indexed 94 files, 49 code files, 11,977 call graph edges, and 26 local git commits for PRD 10.1 code repository coverage. + +That is a snapshot of the repository as it stood on 2026-05-28; the repository is +larger now, and the check passed on any git history, which is why it is not +treated as evidence of anything today. ``` Runtime profile readiness: @@ -111,6 +155,11 @@ Runtime profile readiness: ```text .ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/runtime_profiles.json SGLang 0.5.12.post1, vLLM 0.21.0, KTransformers 0.6.2.post4, WSL RTX 4090 GPU visibility, launcher module probes, and winget llama-server help output are verified. + +Verified inside WSL on that machine, against the fixed distro name the launcher +used at the time. The launch profiles now read `ELY_EYE_WSL_DISTRO`, every +`.ps1` has a `.sh` twin for Linux and macOS, and CI runs the suite on Linux — so +this line is a record of one machine, not a portability claim. ``` Checks: @@ -123,7 +172,7 @@ Cartridge Physical Assets passed 100M Memory Capsule passed Context Cartridge Replay passed Visual HashHop passed # 7/7 model-solved proofs -Standard Benchmark Registry passed # MMMU-Pro=1.00, OCRBench=1.00 local samples +Standard Benchmark Registry passed # one local sample per benchmark Repository Input Coverage passed # 94 files + 11977 call graph edges + local git history Runtime Profile Delivery passed Research Module Delivery passed # executable module gates, evidence graph edge, probability sum @@ -132,6 +181,13 @@ Memory DNA passed Cache Trace passed ``` +These are the rows that suite wrote on 2026-05-28, not a current verdict. `Gaps.md` +appendix A judges each of the fifteen checks individually, and several of them +were self-referential — the replay check compiled the same question twice against +the live store, the recall check only proved an adapter file loads, the cache check +passed if any layer had any entry. The suite was rewritten afterwards so a check +that cannot run says what it could not run instead of passing. + Cartridge physical assets: ```text @@ -142,3 +198,10 @@ Cartridge physical assets: .ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/ttt_vl_adapter.safetensors .ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/visual_mtp_head.safetensors ``` + +A cartridge written today is not this list. `cartridge.py` now writes +`manifest.json`, `atoms.parquet`, `sources.json`, `sparse_index.bm25.zst` and +`temporal_graph.sqlite`, and a cartridge is a scope you mount rather than only a +bundle you export. The files above are what that 2026-05-28 export contained; +`kv_snapshots.kivi` and `ttt_vl_adapter.safetensors` were placeholders for a +runtime layout that was never wired to a loader. diff --git a/backend/src/ely_eye/cli.py b/backend/src/ely_eye/cli.py index 1e66aaa..47377ec 100644 --- a/backend/src/ely_eye/cli.py +++ b/backend/src/ely_eye/cli.py @@ -88,6 +88,7 @@ def search( query: str, top_k: int = typer.Option(0, "--top-k", help="hits to return; 0 uses the configured default"), holder: str = typer.Option("", "--holder", help="only this holder's memories may answer"), + as_holder: str = typer.Option("", "--as-holder", help="see the store through this holder's eyes, the way their agent would"), scenario: str = typer.Option("", "--scenario"), source_id: str = typer.Option("", "--source-id"), since: str = typer.Option("", "--since", help="event time at or after, ISO with offset"), @@ -103,6 +104,7 @@ def search( top_k or None, filters=SearchFilters( holder=holder or None, + as_holder=as_holder or None, scenario=scenario or None, source_id=source_id or None, since=moment(since, "--since"), diff --git a/backend/tests/test_search_api.py b/backend/tests/test_search_api.py index 7de6b23..c6f90ce 100644 --- a/backend/tests/test_search_api.py +++ b/backend/tests/test_search_api.py @@ -112,3 +112,8 @@ def test_the_cli_answers_in_json_a_machine_can_read(tmp_path: Path, monkeypatch: empty = json.loads(runner.invoke(app, ["search", "钥匙", "--scenario", "travel"]).stdout) assert empty["status"] == "no_match" + + stranger = json.loads(runner.invoke(app, ["search", "钥匙", "--as-holder", "stranger"]).stdout) + assert stranger["status"] == "no_match", "--as-holder must show what a stranger would see" + owner = json.loads(runner.invoke(app, ["search", "钥匙", "--as-holder", "wife"]).stdout) + assert [hit["atom"]["text"] for hit in owner["hits"]] == [KEYS]