ZacharyZhang-NY
9f91ff7ed7
The store had a write path, a mount path and a search path, and the console
still showed a dashboard of counters. Nothing in the UI could ask what someone
knows, page through atoms, look at a wordless picture, forget one memory, or
mount a cartridge — so the mechanisms built in P0 were invisible.
Backend:
- senses.py: one function decides what this machine can perceive. /api/status
and doctor each computed it before, which is two truths. Picture atoms are now
counted by image_ref rather than by modality, so a screenshot crop from a PDF
counts and a picture whose bytes were deleted stops claiming it can be encoded.
- browse.py: GET /api/atoms pages newest-first through v_atoms_current with
modality/holder/source filters; GET /api/atoms/{id}/image serves the stored
bytes so a picture with no OCR text can be looked at.
- Deleted 8 HTTP routes nobody calls (4 benchmark, research modules, dense index,
visual-contradiction-proof, memory-map). Their functions stay: the CLI and the
proof checks use them. The surface is 30 routes, pinned as a set by a test so a
new unused route fails the suite.
Frontend: App.tsx (517 lines) became a 131-line shell plus seven panels that own
their own fetches, and lib/format.ts. types.ts now carries the memory dimensions
the backend has been sending since schema v2.
Two invented fields were rendering lies: AdapterPanel showed gpu_available and
probe_status, which AdapterLoadReport does not have — every machine looked GPU-less
including this one. The panel now shows only what the server says.
Proof: scripts/ui_smoke.sh starts the API on a fresh temp home, serves the built
UI from the same origin, and drives five interactions in a real browser — search
as somebody in a scenario, page and view a wordless picture, forget, mount, read
adapter states with reasons. It prints the mount refusal verbatim rather than
pretending the mount succeeded.
Gates: 263 pytest passed; ruff and mypy clean on the 8 touched backend files
(repo-wide mypy unchanged at 24 errors / 6 files); pyright 0 errors there;
go build/vet/test/gofmt clean; npm build + eslint clean; ui_smoke and
mcp_write_smoke both PASS.
166 lines
6.4 KiB
Markdown
166 lines
6.4 KiB
Markdown
# Ely-Eye
|
|
|
|
Ely-Eye is a local multimodal context operating system built on Qwen3.5-9B. It
|
|
runs on a single consumer GPU and gives a small model three tiers of context: a
|
|
262K native window, a 1.01M extended window, and a 100M-token memory capsule it
|
|
can retrieve from, cite, and recall across sessions.
|
|
|
|
Under the hood it ingests PDFs, screenshots, video, and code into Evidence
|
|
Atoms, packs them into a Context Cartridge, retrieves with sparse and Qwen
|
|
embedding search, runs Qwen3.5-9B with trained LoRA adapters, and checks every
|
|
answer with a verifier. A fifteen-check proof suite exercises the whole thing
|
|
against the live model, so a real run can be told apart from a fabricated one.
|
|
|
|
The landing site lives at https://model.elydora.com/
|
|
(its source is under `ely-eye-web/`, which this repository does not track).
|
|
|
|
## Requirements
|
|
|
|
- Windows with an NVIDIA GPU and CUDA. Training and the proof suite were built
|
|
on an RTX 4090; deployment targets a 16GB card such as an RTX 4080.
|
|
- The extended-context runtimes (SGLang, vLLM, KTransformers, llama.cpp) are
|
|
optional and run under WSL. The default Windows runtime uses Transformers and
|
|
needs none of them.
|
|
|
|
## Run it locally
|
|
|
|
```powershell
|
|
.\scripts\install.ps1 # virtual env, PyTorch + CUDA, Transformers, PEFT, bitsandbytes
|
|
.\scripts\download-models.ps1 # Qwen3.5-9B, the 0.8B planner, Qwen3-VL-Embedding-8B
|
|
.\scripts\serve-backend.ps1 # API on http://127.0.0.1:8765
|
|
.\scripts\serve-frontend.ps1 # dashboard on http://127.0.0.1:5173
|
|
```
|
|
|
|
When `ELY_EYE_ADAPTER_DIR` is empty, the runtime picks the latest trained
|
|
HME-Core-LoRA adapter for `Qwen/Qwen3.5-9B` on its own; `ely-eye status` prints
|
|
which adapter it resolved.
|
|
|
|
For the 262K and 1.01M extended-context profiles, set up the Linux runtimes once
|
|
and launch one:
|
|
|
|
```powershell
|
|
.\scripts\setup-linux-runtimes.ps1
|
|
.\scripts\launch-sglang-live.ps1 # 262K
|
|
.\scripts\launch-sglang-extreme.ps1 # 1.01M via YaRN
|
|
```
|
|
|
|
## Core commands
|
|
|
|
```powershell
|
|
.\.venv\Scripts\ely-eye.exe status
|
|
.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project
|
|
.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144
|
|
.\.venv\Scripts\ely-eye.exe proof-suite
|
|
```
|
|
|
|
The console is served by the API itself once `ELY_EYE_UI_DIR` points at
|
|
`frontend/dist`. It shows the state the CLI prints — runtime, GPU, library,
|
|
cartridges, senses, adapters, proof suite — and it is the only place to search
|
|
the store as somebody, page through atoms, forget one, and mount a cartridge.
|
|
`scripts/ui_smoke.sh` drives that console in a real browser against a fresh
|
|
store and fails if any of it stops working.
|
|
|
|
## Fine-tuning
|
|
|
|
Training runs real BF16 LoRA updates on the GPU through Unsloth-patched
|
|
Transformers, PEFT, and bitsandbytes with paged 8-bit AdamW. Each run writes the
|
|
PEFT adapter alongside a manifest, a per-step training trace, a summary, and a
|
|
proof file that records dataset and weight hashes, per-step loss, optimizer-update
|
|
norms, and the CUDA/BF16 environment.
|
|
|
|
The adapter set:
|
|
|
|
```text
|
|
HME-Core-LoRA Qwen3.5-9B attention/MLP
|
|
HME-Vision-LoRA Qwen3.5-9B visual projector and selected vision layers
|
|
HME-TTT-VL Qwen3.5-9B mutable MLP and visual adapter
|
|
HME-Visual-MTP Qwen3.5-9B draft-head LoRA
|
|
HME-Router Qwen3.5-0.8B context planner
|
|
HME-Retrieval Qwen3-VL-Embedding-8B contrastive retrieval adapter
|
|
```
|
|
|
|
## Proof suite
|
|
|
|
`ely-eye proof-suite` runs fifteen checks against the live model and writes the
|
|
evidence under `.ely_eye/data/eval_proofs/`. Among them:
|
|
|
|
- real CUDA BF16 LoRA training — changed weight hashes, nonzero optimizer
|
|
updates, per-step loss
|
|
- the runtime loading the model and adapter and generating a cited answer
|
|
- Visual HashHop: multi-hop visual addressing the model has to actually solve,
|
|
with no OCR shortcut
|
|
- the 100M-token memory capsule committing to a reproducible Memory DNA
|
|
- the Visual Contradiction Lens localizing cross-version drift (design-token,
|
|
layout, typography, copy, visual-code, temporal) at >=80% recall with no false
|
|
positives
|
|
- standard benchmark samples (MMMU-Pro, OCRBench, and others) scored against
|
|
ground truth
|
|
|
|
```powershell
|
|
.\.venv\Scripts\ely-eye.exe proof-suite
|
|
.\.venv\Scripts\ely-eye.exe latest-proof-suite
|
|
.\.venv\Scripts\ely-eye.exe runtime-generation-proof
|
|
.\.venv\Scripts\ely-eye.exe visual-contradiction-proof
|
|
```
|
|
|
|
## MCP server for coding agents
|
|
|
|
`ely-eye-mcp/` is a local Model Context Protocol server that exposes the
|
|
`.ely_eye` memory store to coding agents such as Codex and Claude Code. One
|
|
command builds a native Go binary (or downloads a prebuilt one), installs it
|
|
under `.ely_eye/bin/`, and registers it with both clients. No Docker.
|
|
|
|
Windows:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File .\ely-eye-mcp\install.ps1 -Client both
|
|
```
|
|
|
|
Linux and macOS:
|
|
|
|
```bash
|
|
bash ./ely-eye-mcp/install.sh --client both
|
|
```
|
|
|
|
The agent then has nine read-only tools over the live memory — status,
|
|
cartridge listing and detail, Evidence Atom search and fetch, evidence-pack
|
|
compilation, proof suites, and HashHop proofs. The server opens the database
|
|
read-only and never writes to it. The same installer works against any
|
|
`.ely_eye` store, not only this repository. See `ely-eye-mcp/README.md` for the
|
|
full tool list and the release pipeline.
|
|
|
|
## Data layout
|
|
|
|
Runtime state lives under `.ely_eye/` (git-ignored):
|
|
|
|
```text
|
|
.ely_eye/
|
|
├─ adapters/ trained LoRA adapters
|
|
├─ data/ sqlite database and eval proofs
|
|
├─ training/ datasets and training traces
|
|
├─ objects/ ingested source files
|
|
├─ cache/ cache fabric
|
|
└─ cartridges/ materialized Context Cartridges
|
|
```
|
|
|
|
A Context Cartridge is a self-contained memory package: the Evidence Atoms, a
|
|
sparse index and vector indexes, a temporal graph, KV snapshots, the memory
|
|
capsule, any attached adapters, and the eval proofs.
|
|
|
|
## Project layout
|
|
|
|
```text
|
|
backend/ Python: ingestion, retrieval, runtime, training, proof suite, FastAPI
|
|
frontend/ the local dashboard (Vite + React)
|
|
scripts/ install, model download, serving, runtimes, fine-tuning (.ps1 on Windows, .sh on Linux)
|
|
ely-eye-mcp/ Go: local MCP server exposing the memory store to coding agents
|
|
```
|
|
|
|
## License
|
|
|
|
AGPL-3.0-only. Copyright © 2026 雷电芽衣 <zhangyanghaha0407@outlook.com>.
|
|
|
|
The PDF reader (PyMuPDF) is dual-licensed under AGPL-3.0 or an Artifex
|
|
Commercial license, so this project ships under AGPL while that dependency is
|
|
in use. Buying the Artifex license is what would allow a permissive one.
|