Say what the store actually does, and put a receipt under each claim

The README described a project that no longer exists: a Windows-only Qwen
runtime with nine read-only tools, a console as the only way to operate the
store, and a 100M-token capsule as if it were measured. It is now a description
of what is here, with the report and the command that re-measures each number,
and a section that names what has not been run instead of leaving the reader to
discover it.

TRAINING_ARTIFACTS.md keeps its artifact tables and loses the scores that were
never run: MMMU-Pro and OCRBench each read one sample, and the HashHop gate
collected proofs until it hit its target. What stays is annotated with what it
actually measures — citation accuracy there means the cited id existed in the
context that was handed over, not that the passage supports the answer.

Gaps.md gains a re-audit section: ten of the twelve gaps closed, two partial,
each row naming the command or report behind it, and the remaining work stated
plainly — the 0.90 recall bar is unmet and the deficit is entirely in the
visual and audio legs.

AGENTS.md stops being ignored. It records the constraints this repository paid
for: the measurement rules, the environment traps, and the review protocol, so
the next agent does not pay them again.

The CLI could filter by holder but could not look through a holder's eyes, so
search gains --as-holder: three states of eyes on every face, not two.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-09 16:11:53 -04:00
1 parent 83dfc14daa
commit 89c880fd36
8 files changed
+349 -125

No files matched your search

+136 -115
View File
@@ -1,133 +1,150 @@
# Ely-Eye
Ely-Eye is a local multimodal context operating system built on Qwen3.5-9B. It
runs on a single consumer GPU and gives a small model three tiers of context: a
262K native window, a 1.01M extended window, and a 100M-token memory capsule it
can retrieve from, cite, and recall across sessions.
Ely-Eye is a local memory store for AI. Every memory it keeps carries who it
belongs to, where it came from, when it was true, in what situation, and who may
read it. Retrieval answers as the person asking, so a store shared by several
people and several agents does not blend them into one voice. A Context
Cartridge is a scope you mount, not only a bundle you export. Coding agents
reach the same store over MCP.
Under the hood it ingests PDFs, screenshots, video, and code into Evidence
Atoms, packs them into a Context Cartridge, retrieves with sparse and Qwen
embedding search, runs Qwen3.5-9B with trained LoRA adapters, and checks every
answer with a verifier. A fifteen-check proof suite exercises the whole thing
against the live model, so a real run can be told apart from a fabricated one.
The difference between this and a vector database is the referral: "I don't know
where the keys are, I know my wife knows, and I can find my wife." When the
store cannot answer, it says nobody knows and names who does — instead of
inventing a plausible answer.
The landing site lives at https://model.elydora.com/
(its source is under `ely-eye-web/`, which this repository does not track).
Everything below is either measured in this repository, with the report and the
command that re-measures it, or it is marked as not measured.
## Requirements
## What is measured
- Windows with an NVIDIA GPU and CUDA. Training and the proof suite were built
on an RTX 4090; deployment targets a 16GB card such as an RTX 4080.
- The extended-context runtimes (SGLang, vLLM, KTransformers, llama.cpp) are
optional and run under WSL. The default Windows runtime uses Transformers and
needs none of them.
| question | answer | report |
| --- | --- | --- |
| can it find the memory that answers a question? | recall@10 **0.863**, MRR 0.682, over 300 answerable probes drawn from six third-party corpora, each asked as the holder that corpus assigns it | `bench/reports/LEGS.md` |
| is that any good? | a Mem0-style counterpart on the same corpus scores **0.866** inside one holder's memory and **0.412** across all of them; chance is 0.0002 whole-corpus and 0.369 inside one holder | `bench/reports/BASELINE_EXTERNAL.md` |
| does the holder boundary matter? | **+0.454 recall@10** with an identical ranker and identical items — the gain is scoping, not ranking | same report |
| how fast at corpus size? | p50 **0.072 s**, p95 0.218 s over 165,973 atoms; the search this replaced took 20.16 s on the same corpus | `bench/reports/SCALE.md` |
| how much does it hold? | 165,973 atoms and 153,857 vectors in one SQLite file; the vector build runs 106 memories/s. One million is an extrapolation, not a measurement | `bench/reports/DENSE.md` |
| can a wordless picture be found? | yes for icon crops, weakly for whole screenshots — the report shows every arm, including the ones that hurt | `bench/reports/VISUAL.md` |
| what does speech cost? | WER **0.242** over 533 reference words; 258.18 s of audio transcribed in 3.12 s (68.4–111.2× realtime per video, CPU int8), byte-identical across two passes | `bench/reports/ASR.md` |
| is the acceptance bar met? | **no.** The bar is recall@10 ≥ 0.90; shipped is 0.863 | `bench/reports/LEGS.md` |
## Run it locally
Where the deficit lives, by modality (recall@10, same run): code 0.945,
conversation 1.000, pdf 0.885, ui_screenshot 0.818, image 0.720, video 0.455.
Video is the worst because the frozen stores hold no audio atoms — a video is
sampled frames plus OCR, so a question answered by speech has nothing to match.
Images and screenshots are limited because CLIP cannot localize an icon inside a
screenshot. `bench/reports/LEGS.md` names the arms; nothing here rounds those
numbers up.
```powershell
.\scripts\install.ps1 # virtual env, PyTorch + CUDA, Transformers, PEFT, bitsandbytes
.\scripts\download-models.ps1 # Qwen3.5-9B, the 0.8B planner, Qwen3-VL-Embedding-8B
.\scripts\serve-backend.ps1 # API on http://127.0.0.1:8765
.\scripts\serve-frontend.ps1 # dashboard on http://127.0.0.1:5173
Re-measure any of them:
```bash
python -m bench.validate_sets # the frozen corpus and its gold
python -m bench.leg_check --probes 300 # the shipped retrieval path
python -m bench.run_external # the counterparts and the chance floor
python -m bench.dense_scale # the vector leg at corpus size
```
When `ELY_EYE_ADAPTER_DIR` is empty, the runtime picks the latest trained
HME-Core-LoRA adapter for `Qwen/Qwen3.5-9B` on its own; `ely-eye status` prints
which adapter it resolved.
## What is not measured here
For the 262K and 1.01M extended-context profiles, set up the Linux runtimes once
and launch one:
The generator side is written and runnable but has not been run in this
repository's reports, so no score from it is quoted anywhere in this README:
```powershell
.\scripts\setup-linux-runtimes.ps1
.\scripts\launch-sglang-live.ps1 # 262K
.\scripts\launch-sglang-extreme.ps1 # 1.01M via YaRN
- generation with Qwen3.5-9B and the trained LoRA adapters — `ely-eye runtime-generation-proof`
- Visual HashHop multi-hop visual addressing — `ely-eye hashhop --kind visual`
- the Visual Contradiction Lens — `ely-eye visual-contradiction-proof`
- standard benchmarks (MMMU-Pro, OCRBench, and the rest) — `ely-eye run-benchmark`
- the fifteen-check proof suite as a whole — `ely-eye proof-suite`
Each needs a CUDA machine and a model download. They are marked DEFERRED in
`CHECKLIST.md`, and a run that has not happened is not allowed to look like a
result: `ely-eye proof-suite` writes what it observed, and the checks that
depend on a model report what they could not do rather than passing quietly.
`TRAINING_ARTIFACTS.md` keeps the six-adapter training table, which is
artifact-backed on the machine that trained them, and no longer quotes benchmark
scores that were never run.
## What a memory carries
`holder_id`, `origin` (self / observed / artifact / third_party), `scenario`,
`event_time` with its precision and how it was learned, `valid_from` /
`valid_to`, `supersedes`, `visibility` (holder / visible_to / everyone),
`retention_until`, plus ingestion time and trust scores. Ingestion stamps
provenance; a client cannot claim to be somebody else, because the server
decides who a caller is from the token it presents.
Four verbs, one writer: `remember`, `supersede`, `forget`, `search`. They exist
over HTTP, the CLI, and MCP, and they all go through the same Python writer, so
the rules cannot drift apart. `forget` cascades to everything derived — vectors,
FTS rows, edges, cache entries — and reports which cartridges contained the
memory rather than silently rewriting an export whose digest somebody already
quoted.
On top of that: a working tier (what one session recorded, in order),
consolidation into a standing summary, an archive tier for memories that are
neither used nor valued, and a conflict list for dated memories that disagree
and that nobody has settled. A third-party memory that no other memory vouches
for is quarantined: it stays in the store, and it is not citable.
## Run it
Linux and macOS:
```bash
bash scripts/install.sh
bash scripts/serve-backend.sh # API on http://127.0.0.1:8765
bash scripts/serve-frontend.sh # dashboard on http://127.0.0.1:5173
```
## Core commands
Windows: the same names as `.ps1`. There is also a container image
(`docker build -t ely-eye .`), and CI runs the whole suite on Linux.
```powershell
.\.venv\Scripts\ely-eye.exe status
.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project
.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144
.\.venv\Scripts\ely-eye.exe proof-suite
```bash
ely-eye ingest ~/some/package --cartridge-name project
ely-eye search "where did we put the archive key" --as-holder wife
ely-eye remember "the archive key is in the second drawer" --holder wife --scenario home
ely-eye who-knows "archive key"
ely-eye ask "where is the archive key" --as-holder husband
ely-eye card
ely-eye mounts && ely-eye mount project && ely-eye unmount
ely-eye backup ./backup && ely-eye doctor
bash scripts/mcp_write_smoke.sh # two processes, one store, one SQLite
```
The console is served by the API itself once `ELY_EYE_UI_DIR` points at
`frontend/dist`. It shows the state the CLI prints — runtime, GPU, library,
cartridges, senses, adapters, proof suite — and it is the only place to search
the store as somebody, page through atoms, forget one, and mount a cartridge.
`scripts/ui_smoke.sh` drives that console in a real browser against a fresh
store and fails if any of it stops working.
The dashboard is served by the API itself once `ELY_EYE_UI_DIR` points at
`frontend/dist`. `scripts/ui_smoke.sh` drives it in a real browser against a
fresh store and fails if any of it stops working. The CLI does everything the
console does, so nothing requires a browser.
## Fine-tuning
## Retrieval
Training runs real BF16 LoRA updates on the GPU through Unsloth-patched
Transformers, PEFT, and bitsandbytes with paged 8-bit AdamW. Each run writes the
PEFT adapter alongside a manifest, a per-step training trace, a summary, and a
proof file that records dataset and weight hashes, per-step loss, optimizer-update
norms, and the CUDA/BF16 environment.
The adapter set:
```text
HME-Core-LoRA Qwen3.5-9B attention/MLP
HME-Vision-LoRA Qwen3.5-9B visual projector and selected vision layers
HME-TTT-VL Qwen3.5-9B mutable MLP and visual adapter
HME-Visual-MTP Qwen3.5-9B draft-head LoRA
HME-Router Qwen3.5-0.8B context planner
HME-Retrieval Qwen3-VL-Embedding-8B contrastive retrieval adapter
```
## Proof suite
`ely-eye proof-suite` runs fifteen checks against the live model and writes the
evidence under `.ely_eye/data/eval_proofs/`. Among them:
- real CUDA BF16 LoRA training — changed weight hashes, nonzero optimizer
updates, per-step loss
- the runtime loading the model and adapter and generating a cited answer
- Visual HashHop: multi-hop visual addressing the model has to actually solve,
with no OCR shortcut
- the 100M-token memory capsule committing to a reproducible Memory DNA
- the Visual Contradiction Lens localizing cross-version drift (design-token,
layout, typography, copy, visual-code, temporal) at >=80% recall with no false
positives
- standard benchmark samples (MMMU-Pro, OCRBench, and others) scored against
ground truth
```powershell
.\.venv\Scripts\ely-eye.exe proof-suite
.\.venv\Scripts\ely-eye.exe latest-proof-suite
.\.venv\Scripts\ely-eye.exe runtime-generation-proof
.\.venv\Scripts\ely-eye.exe visual-contradiction-proof
```
Three legs, fused by rank. The words leg always runs. The dense leg runs only
when `ELY_EYE_DENSE_EMBEDDINGS_ENABLED=1` and an embedding model is configured.
On the shipped page that leg moves recall from 0.863 to 0.860 while p50 goes
from 122 ms to 713 ms; on a wider page both reach 0.873 while p50 goes from
84 ms to 579 ms. It buys the memory questions (multi-session 0.857 → 0.920,
preference 0.800 → 0.960) and costs the picture ones (image 0.720 → 0.680),
which is why it is opt-in rather than the default. The picture leg runs only
when the words page comes back thin and `ELY_EYE_CLIP_MODEL_DIR` is set.
Vectors live in the database beside the atoms, written in the same transaction
as the memory itself, so a memory is findable the moment it is stored and there
is no index to rebuild.
## MCP server for coding agents
`ely-eye-mcp/` is a local Model Context Protocol server that exposes the
`.ely_eye` memory store to coding agents such as Codex and Claude Code. One
command builds a native Go binary (or downloads a prebuilt one), installs it
under `.ely_eye/bin/`, and registers it with both clients. No Docker.
Windows:
```powershell
powershell -ExecutionPolicy Bypass -File .\ely-eye-mcp\install.ps1 -Client both
```
Linux and macOS:
`ely-eye-mcp/` is a Go MCP server. One command builds a native binary and
registers it with Codex and Claude Code. Reads mirror the store; writes go
through the API, so the write rules live in one place. A read with no
`ELY_EYE_AUTH_TOKEN` sees only memories nobody owns and memories marked for
everyone; `ely_eye_card` answers "can this store answer my kind of question"
before anything sensitive is sent, and it carries no memory text.
```bash
bash ./ely-eye-mcp/install.sh --client both
```
The agent then has nine read-only tools over the live memory — status,
cartridge listing and detail, Evidence Atom search and fetch, evidence-pack
compilation, proof suites, and HashHop proofs. The server opens the database
read-only and never writes to it. The same installer works against any
`.ely_eye` store, not only this repository. See `ely-eye-mcp/README.md` for the
full tool list and the release pipeline.
See `ely-eye-mcp/README.md` for the full tool list.
## Data layout
@@ -135,27 +152,31 @@ Runtime state lives under `.ely_eye/` (git-ignored):
```text
.ely_eye/
├─ adapters/ trained LoRA adapters
├─ data/ sqlite database and eval proofs
├─ training/ datasets and training traces
├─ data/ the SQLite store: atoms, FTS, vectors (vec0), edges, mounts
├─ objects/ ingested source files
├─ cache/ cache fabric
└─ cartridges/ materialized Context Cartridges
├─ cache/ compiled-context cache and the picture index
├─ cartridges/ materialized Context Cartridges
├─ adapters/ trained LoRA adapters
└─ training/ datasets and training traces
```
A Context Cartridge is a self-contained memory package: the Evidence Atoms, a
sparse index and vector indexes, a temporal graph, KV snapshots, the memory
capsule, any attached adapters, and the eval proofs.
A Context Cartridge is a self-contained memory package: the Evidence Atoms, the
sparse and vector indexes, the temporal graph, and a DNA digest of the files.
Mounting one scopes every subsequent search to it; switching is one command.
## Project layout
```text
backend/ Python: ingestion, retrieval, runtime, training, proof suite, FastAPI
frontend/ the local dashboard (Vite + React)
scripts/ install, model download, serving, runtimes, fine-tuning (.ps1 on Windows, .sh on Linux)
ely-eye-mcp/ Go: local MCP server exposing the memory store to coding agents
backend/ Python: ingestion, retrieval, memory, runtime, training, FastAPI
bench/ the benchmark suite: frozen third-party corpora, harnesses, reports
frontend/ the local dashboard (Vite + React)
scripts/ install, serving, runtimes, smoke (.sh on Linux/macOS, .ps1 on Windows)
ely-eye-mcp/ Go: the MCP server
```
`CHECKLIST.md` is the ordered work list with the acceptance line for every item;
`Gaps.md` is the audit this work started from.
## License
AGPL-3.0-only. Copyright © 2026 雷电芽衣 <zhangyanghaha0407@outlook.com>.