Say what the store actually does, and put a receipt under each claim
The README described a project that no longer exists: a Windows-only Qwen runtime with nine read-only tools, a console as the only way to operate the store, and a 100M-token capsule as if it were measured. It is now a description of what is here, with the report and the command that re-measures each number, and a section that names what has not been run instead of leaving the reader to discover it. TRAINING_ARTIFACTS.md keeps its artifact tables and loses the scores that were never run: MMMU-Pro and OCRBench each read one sample, and the HashHop gate collected proofs until it hit its target. What stays is annotated with what it actually measures — citation accuracy there means the cited id existed in the context that was handed over, not that the passage supports the answer. Gaps.md gains a re-audit section: ten of the twelve gaps closed, two partial, each row naming the command or report behind it, and the remaining work stated plainly — the 0.90 recall bar is unmet and the deficit is entirely in the visual and audio legs. AGENTS.md stops being ignored. It records the constraints this repository paid for: the measurement rules, the environment traps, and the review protocol, so the next agent does not pay them again. The CLI could filter by holder but could not look through a holder's eyes, so search gains --as-holder: three states of eyes on every face, not two.
This commit is contained in:
8 files changed
+349
-125
No files matched your search
@@ -1,133 +1,150 @@
|
||||
# Ely-Eye
|
||||
|
||||
Ely-Eye is a local multimodal context operating system built on Qwen3.5-9B. It
|
||||
runs on a single consumer GPU and gives a small model three tiers of context: a
|
||||
262K native window, a 1.01M extended window, and a 100M-token memory capsule it
|
||||
can retrieve from, cite, and recall across sessions.
|
||||
Ely-Eye is a local memory store for AI. Every memory it keeps carries who it
|
||||
belongs to, where it came from, when it was true, in what situation, and who may
|
||||
read it. Retrieval answers as the person asking, so a store shared by several
|
||||
people and several agents does not blend them into one voice. A Context
|
||||
Cartridge is a scope you mount, not only a bundle you export. Coding agents
|
||||
reach the same store over MCP.
|
||||
|
||||
Under the hood it ingests PDFs, screenshots, video, and code into Evidence
|
||||
Atoms, packs them into a Context Cartridge, retrieves with sparse and Qwen
|
||||
embedding search, runs Qwen3.5-9B with trained LoRA adapters, and checks every
|
||||
answer with a verifier. A fifteen-check proof suite exercises the whole thing
|
||||
against the live model, so a real run can be told apart from a fabricated one.
|
||||
The difference between this and a vector database is the referral: "I don't know
|
||||
where the keys are, I know my wife knows, and I can find my wife." When the
|
||||
store cannot answer, it says nobody knows and names who does — instead of
|
||||
inventing a plausible answer.
|
||||
|
||||
The landing site lives at https://model.elydora.com/
|
||||
(its source is under `ely-eye-web/`, which this repository does not track).
|
||||
Everything below is either measured in this repository, with the report and the
|
||||
command that re-measures it, or it is marked as not measured.
|
||||
|
||||
## Requirements
|
||||
## What is measured
|
||||
|
||||
- Windows with an NVIDIA GPU and CUDA. Training and the proof suite were built
|
||||
on an RTX 4090; deployment targets a 16GB card such as an RTX 4080.
|
||||
- The extended-context runtimes (SGLang, vLLM, KTransformers, llama.cpp) are
|
||||
optional and run under WSL. The default Windows runtime uses Transformers and
|
||||
needs none of them.
|
||||
| question | answer | report |
|
||||
| --- | --- | --- |
|
||||
| can it find the memory that answers a question? | recall@10 **0.863**, MRR 0.682, over 300 answerable probes drawn from six third-party corpora, each asked as the holder that corpus assigns it | `bench/reports/LEGS.md` |
|
||||
| is that any good? | a Mem0-style counterpart on the same corpus scores **0.866** inside one holder's memory and **0.412** across all of them; chance is 0.0002 whole-corpus and 0.369 inside one holder | `bench/reports/BASELINE_EXTERNAL.md` |
|
||||
| does the holder boundary matter? | **+0.454 recall@10** with an identical ranker and identical items — the gain is scoping, not ranking | same report |
|
||||
| how fast at corpus size? | p50 **0.072 s**, p95 0.218 s over 165,973 atoms; the search this replaced took 20.16 s on the same corpus | `bench/reports/SCALE.md` |
|
||||
| how much does it hold? | 165,973 atoms and 153,857 vectors in one SQLite file; the vector build runs 106 memories/s. One million is an extrapolation, not a measurement | `bench/reports/DENSE.md` |
|
||||
| can a wordless picture be found? | yes for icon crops, weakly for whole screenshots — the report shows every arm, including the ones that hurt | `bench/reports/VISUAL.md` |
|
||||
| what does speech cost? | WER **0.242** over 533 reference words; 258.18 s of audio transcribed in 3.12 s (68.4–111.2× realtime per video, CPU int8), byte-identical across two passes | `bench/reports/ASR.md` |
|
||||
| is the acceptance bar met? | **no.** The bar is recall@10 ≥ 0.90; shipped is 0.863 | `bench/reports/LEGS.md` |
|
||||
|
||||
## Run it locally
|
||||
Where the deficit lives, by modality (recall@10, same run): code 0.945,
|
||||
conversation 1.000, pdf 0.885, ui_screenshot 0.818, image 0.720, video 0.455.
|
||||
Video is the worst because the frozen stores hold no audio atoms — a video is
|
||||
sampled frames plus OCR, so a question answered by speech has nothing to match.
|
||||
Images and screenshots are limited because CLIP cannot localize an icon inside a
|
||||
screenshot. `bench/reports/LEGS.md` names the arms; nothing here rounds those
|
||||
numbers up.
|
||||
|
||||
```powershell
|
||||
.\scripts\install.ps1 # virtual env, PyTorch + CUDA, Transformers, PEFT, bitsandbytes
|
||||
.\scripts\download-models.ps1 # Qwen3.5-9B, the 0.8B planner, Qwen3-VL-Embedding-8B
|
||||
.\scripts\serve-backend.ps1 # API on http://127.0.0.1:8765
|
||||
.\scripts\serve-frontend.ps1 # dashboard on http://127.0.0.1:5173
|
||||
Re-measure any of them:
|
||||
|
||||
```bash
|
||||
python -m bench.validate_sets # the frozen corpus and its gold
|
||||
python -m bench.leg_check --probes 300 # the shipped retrieval path
|
||||
python -m bench.run_external # the counterparts and the chance floor
|
||||
python -m bench.dense_scale # the vector leg at corpus size
|
||||
```
|
||||
|
||||
When `ELY_EYE_ADAPTER_DIR` is empty, the runtime picks the latest trained
|
||||
HME-Core-LoRA adapter for `Qwen/Qwen3.5-9B` on its own; `ely-eye status` prints
|
||||
which adapter it resolved.
|
||||
## What is not measured here
|
||||
|
||||
For the 262K and 1.01M extended-context profiles, set up the Linux runtimes once
|
||||
and launch one:
|
||||
The generator side is written and runnable but has not been run in this
|
||||
repository's reports, so no score from it is quoted anywhere in this README:
|
||||
|
||||
```powershell
|
||||
.\scripts\setup-linux-runtimes.ps1
|
||||
.\scripts\launch-sglang-live.ps1 # 262K
|
||||
.\scripts\launch-sglang-extreme.ps1 # 1.01M via YaRN
|
||||
- generation with Qwen3.5-9B and the trained LoRA adapters — `ely-eye runtime-generation-proof`
|
||||
- Visual HashHop multi-hop visual addressing — `ely-eye hashhop --kind visual`
|
||||
- the Visual Contradiction Lens — `ely-eye visual-contradiction-proof`
|
||||
- standard benchmarks (MMMU-Pro, OCRBench, and the rest) — `ely-eye run-benchmark`
|
||||
- the fifteen-check proof suite as a whole — `ely-eye proof-suite`
|
||||
|
||||
Each needs a CUDA machine and a model download. They are marked DEFERRED in
|
||||
`CHECKLIST.md`, and a run that has not happened is not allowed to look like a
|
||||
result: `ely-eye proof-suite` writes what it observed, and the checks that
|
||||
depend on a model report what they could not do rather than passing quietly.
|
||||
|
||||
`TRAINING_ARTIFACTS.md` keeps the six-adapter training table, which is
|
||||
artifact-backed on the machine that trained them, and no longer quotes benchmark
|
||||
scores that were never run.
|
||||
|
||||
## What a memory carries
|
||||
|
||||
`holder_id`, `origin` (self / observed / artifact / third_party), `scenario`,
|
||||
`event_time` with its precision and how it was learned, `valid_from` /
|
||||
`valid_to`, `supersedes`, `visibility` (holder / visible_to / everyone),
|
||||
`retention_until`, plus ingestion time and trust scores. Ingestion stamps
|
||||
provenance; a client cannot claim to be somebody else, because the server
|
||||
decides who a caller is from the token it presents.
|
||||
|
||||
Four verbs, one writer: `remember`, `supersede`, `forget`, `search`. They exist
|
||||
over HTTP, the CLI, and MCP, and they all go through the same Python writer, so
|
||||
the rules cannot drift apart. `forget` cascades to everything derived — vectors,
|
||||
FTS rows, edges, cache entries — and reports which cartridges contained the
|
||||
memory rather than silently rewriting an export whose digest somebody already
|
||||
quoted.
|
||||
|
||||
On top of that: a working tier (what one session recorded, in order),
|
||||
consolidation into a standing summary, an archive tier for memories that are
|
||||
neither used nor valued, and a conflict list for dated memories that disagree
|
||||
and that nobody has settled. A third-party memory that no other memory vouches
|
||||
for is quarantined: it stays in the store, and it is not citable.
|
||||
|
||||
## Run it
|
||||
|
||||
Linux and macOS:
|
||||
|
||||
```bash
|
||||
bash scripts/install.sh
|
||||
bash scripts/serve-backend.sh # API on http://127.0.0.1:8765
|
||||
bash scripts/serve-frontend.sh # dashboard on http://127.0.0.1:5173
|
||||
```
|
||||
|
||||
## Core commands
|
||||
Windows: the same names as `.ps1`. There is also a container image
|
||||
(`docker build -t ely-eye .`), and CI runs the whole suite on Linux.
|
||||
|
||||
```powershell
|
||||
.\.venv\Scripts\ely-eye.exe status
|
||||
.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project
|
||||
.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144
|
||||
.\.venv\Scripts\ely-eye.exe proof-suite
|
||||
```bash
|
||||
ely-eye ingest ~/some/package --cartridge-name project
|
||||
ely-eye search "where did we put the archive key" --as-holder wife
|
||||
ely-eye remember "the archive key is in the second drawer" --holder wife --scenario home
|
||||
ely-eye who-knows "archive key"
|
||||
ely-eye ask "where is the archive key" --as-holder husband
|
||||
ely-eye card
|
||||
ely-eye mounts && ely-eye mount project && ely-eye unmount
|
||||
ely-eye backup ./backup && ely-eye doctor
|
||||
bash scripts/mcp_write_smoke.sh # two processes, one store, one SQLite
|
||||
```
|
||||
|
||||
The console is served by the API itself once `ELY_EYE_UI_DIR` points at
|
||||
`frontend/dist`. It shows the state the CLI prints — runtime, GPU, library,
|
||||
cartridges, senses, adapters, proof suite — and it is the only place to search
|
||||
the store as somebody, page through atoms, forget one, and mount a cartridge.
|
||||
`scripts/ui_smoke.sh` drives that console in a real browser against a fresh
|
||||
store and fails if any of it stops working.
|
||||
The dashboard is served by the API itself once `ELY_EYE_UI_DIR` points at
|
||||
`frontend/dist`. `scripts/ui_smoke.sh` drives it in a real browser against a
|
||||
fresh store and fails if any of it stops working. The CLI does everything the
|
||||
console does, so nothing requires a browser.
|
||||
|
||||
## Fine-tuning
|
||||
## Retrieval
|
||||
|
||||
Training runs real BF16 LoRA updates on the GPU through Unsloth-patched
|
||||
Transformers, PEFT, and bitsandbytes with paged 8-bit AdamW. Each run writes the
|
||||
PEFT adapter alongside a manifest, a per-step training trace, a summary, and a
|
||||
proof file that records dataset and weight hashes, per-step loss, optimizer-update
|
||||
norms, and the CUDA/BF16 environment.
|
||||
|
||||
The adapter set:
|
||||
|
||||
```text
|
||||
HME-Core-LoRA Qwen3.5-9B attention/MLP
|
||||
HME-Vision-LoRA Qwen3.5-9B visual projector and selected vision layers
|
||||
HME-TTT-VL Qwen3.5-9B mutable MLP and visual adapter
|
||||
HME-Visual-MTP Qwen3.5-9B draft-head LoRA
|
||||
HME-Router Qwen3.5-0.8B context planner
|
||||
HME-Retrieval Qwen3-VL-Embedding-8B contrastive retrieval adapter
|
||||
```
|
||||
|
||||
## Proof suite
|
||||
|
||||
`ely-eye proof-suite` runs fifteen checks against the live model and writes the
|
||||
evidence under `.ely_eye/data/eval_proofs/`. Among them:
|
||||
|
||||
- real CUDA BF16 LoRA training — changed weight hashes, nonzero optimizer
|
||||
updates, per-step loss
|
||||
- the runtime loading the model and adapter and generating a cited answer
|
||||
- Visual HashHop: multi-hop visual addressing the model has to actually solve,
|
||||
with no OCR shortcut
|
||||
- the 100M-token memory capsule committing to a reproducible Memory DNA
|
||||
- the Visual Contradiction Lens localizing cross-version drift (design-token,
|
||||
layout, typography, copy, visual-code, temporal) at >=80% recall with no false
|
||||
positives
|
||||
- standard benchmark samples (MMMU-Pro, OCRBench, and others) scored against
|
||||
ground truth
|
||||
|
||||
```powershell
|
||||
.\.venv\Scripts\ely-eye.exe proof-suite
|
||||
.\.venv\Scripts\ely-eye.exe latest-proof-suite
|
||||
.\.venv\Scripts\ely-eye.exe runtime-generation-proof
|
||||
.\.venv\Scripts\ely-eye.exe visual-contradiction-proof
|
||||
```
|
||||
Three legs, fused by rank. The words leg always runs. The dense leg runs only
|
||||
when `ELY_EYE_DENSE_EMBEDDINGS_ENABLED=1` and an embedding model is configured.
|
||||
On the shipped page that leg moves recall from 0.863 to 0.860 while p50 goes
|
||||
from 122 ms to 713 ms; on a wider page both reach 0.873 while p50 goes from
|
||||
84 ms to 579 ms. It buys the memory questions (multi-session 0.857 → 0.920,
|
||||
preference 0.800 → 0.960) and costs the picture ones (image 0.720 → 0.680),
|
||||
which is why it is opt-in rather than the default. The picture leg runs only
|
||||
when the words page comes back thin and `ELY_EYE_CLIP_MODEL_DIR` is set.
|
||||
Vectors live in the database beside the atoms, written in the same transaction
|
||||
as the memory itself, so a memory is findable the moment it is stored and there
|
||||
is no index to rebuild.
|
||||
|
||||
## MCP server for coding agents
|
||||
|
||||
`ely-eye-mcp/` is a local Model Context Protocol server that exposes the
|
||||
`.ely_eye` memory store to coding agents such as Codex and Claude Code. One
|
||||
command builds a native Go binary (or downloads a prebuilt one), installs it
|
||||
under `.ely_eye/bin/`, and registers it with both clients. No Docker.
|
||||
|
||||
Windows:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\ely-eye-mcp\install.ps1 -Client both
|
||||
```
|
||||
|
||||
Linux and macOS:
|
||||
`ely-eye-mcp/` is a Go MCP server. One command builds a native binary and
|
||||
registers it with Codex and Claude Code. Reads mirror the store; writes go
|
||||
through the API, so the write rules live in one place. A read with no
|
||||
`ELY_EYE_AUTH_TOKEN` sees only memories nobody owns and memories marked for
|
||||
everyone; `ely_eye_card` answers "can this store answer my kind of question"
|
||||
before anything sensitive is sent, and it carries no memory text.
|
||||
|
||||
```bash
|
||||
bash ./ely-eye-mcp/install.sh --client both
|
||||
```
|
||||
|
||||
The agent then has nine read-only tools over the live memory — status,
|
||||
cartridge listing and detail, Evidence Atom search and fetch, evidence-pack
|
||||
compilation, proof suites, and HashHop proofs. The server opens the database
|
||||
read-only and never writes to it. The same installer works against any
|
||||
`.ely_eye` store, not only this repository. See `ely-eye-mcp/README.md` for the
|
||||
full tool list and the release pipeline.
|
||||
See `ely-eye-mcp/README.md` for the full tool list.
|
||||
|
||||
## Data layout
|
||||
|
||||
@@ -135,27 +152,31 @@ Runtime state lives under `.ely_eye/` (git-ignored):
|
||||
|
||||
```text
|
||||
.ely_eye/
|
||||
├─ adapters/ trained LoRA adapters
|
||||
├─ data/ sqlite database and eval proofs
|
||||
├─ training/ datasets and training traces
|
||||
├─ data/ the SQLite store: atoms, FTS, vectors (vec0), edges, mounts
|
||||
├─ objects/ ingested source files
|
||||
├─ cache/ cache fabric
|
||||
└─ cartridges/ materialized Context Cartridges
|
||||
├─ cache/ compiled-context cache and the picture index
|
||||
├─ cartridges/ materialized Context Cartridges
|
||||
├─ adapters/ trained LoRA adapters
|
||||
└─ training/ datasets and training traces
|
||||
```
|
||||
|
||||
A Context Cartridge is a self-contained memory package: the Evidence Atoms, a
|
||||
sparse index and vector indexes, a temporal graph, KV snapshots, the memory
|
||||
capsule, any attached adapters, and the eval proofs.
|
||||
A Context Cartridge is a self-contained memory package: the Evidence Atoms, the
|
||||
sparse and vector indexes, the temporal graph, and a DNA digest of the files.
|
||||
Mounting one scopes every subsequent search to it; switching is one command.
|
||||
|
||||
## Project layout
|
||||
|
||||
```text
|
||||
backend/ Python: ingestion, retrieval, runtime, training, proof suite, FastAPI
|
||||
frontend/ the local dashboard (Vite + React)
|
||||
scripts/ install, model download, serving, runtimes, fine-tuning (.ps1 on Windows, .sh on Linux)
|
||||
ely-eye-mcp/ Go: local MCP server exposing the memory store to coding agents
|
||||
backend/ Python: ingestion, retrieval, memory, runtime, training, FastAPI
|
||||
bench/ the benchmark suite: frozen third-party corpora, harnesses, reports
|
||||
frontend/ the local dashboard (Vite + React)
|
||||
scripts/ install, serving, runtimes, smoke (.sh on Linux/macOS, .ps1 on Windows)
|
||||
ely-eye-mcp/ Go: the MCP server
|
||||
```
|
||||
|
||||
`CHECKLIST.md` is the ordered work list with the acceptance line for every item;
|
||||
`Gaps.md` is the audit this work started from.
|
||||
|
||||
## License
|
||||
|
||||
AGPL-3.0-only. Copyright © 2026 雷电芽衣 <zhangyanghaha0407@outlook.com>.
|
||||
|
||||
Reference in new issue
Block a user