ZacharyZhang-NY
07825913fe
CI was a workflow that could not run its own tests: pytest is not a dependency of anything, so a fresh clone fails with "No module named pytest". Test tooling now lives in the backend's dev extra, so CI and a new contributor install the same tools, and ruff is pinned there instead of drifting as a bare version in a workflow step. backup, restore and doctor move the store. Backup uses SQLite's own online API so a running server is safe, and the manifest records the schema version, row counts, sha256 and size read back from the copy. Restore verifies the digest, integrity_check, required tables and schema age before it touches the live file, and keeps what it replaces as .before-restore. Doctor separates broken from degraded. On the 906 MB baseline store (165,973 atoms) it reports no problems and one true warning: 21,362 picture atoms with no CLIP snapshot configured, so those atoms are findable only by the text already on them. The first version called that store healthy because it only complained about paths that were set and missing, never about a sense the store needs and never had. AGPL-3.0-only, because PyMuPDF is dual-licensed AGPL-or-commercial and the repository is public. A permissive license needs the Artifex commercial license bought first; the README says so rather than hiding it. Gates: 253 tests pass; ruff clean repo-wide (17 errors at HEAD); mypy unchanged from HEAD on touched files; pyright 0 errors on the new files; backup 4.0s and restore 2.8s on real data with counts verified equal.
162 lines
6.1 KiB
Markdown
162 lines
6.1 KiB
Markdown
# Ely-Eye
|
|
|
|
Ely-Eye is a local multimodal context operating system built on Qwen3.5-9B. It
|
|
runs on a single consumer GPU and gives a small model three tiers of context: a
|
|
262K native window, a 1.01M extended window, and a 100M-token memory capsule it
|
|
can retrieve from, cite, and recall across sessions.
|
|
|
|
Under the hood it ingests PDFs, screenshots, video, and code into Evidence
|
|
Atoms, packs them into a Context Cartridge, retrieves with sparse and Qwen
|
|
embedding search, runs Qwen3.5-9B with trained LoRA adapters, and checks every
|
|
answer with a verifier. A fifteen-check proof suite exercises the whole thing
|
|
against the live model, so a real run can be told apart from a fabricated one.
|
|
|
|
The landing site lives at https://model.elydora.com/
|
|
(its source is under `ely-eye-web/`, which this repository does not track).
|
|
|
|
## Requirements
|
|
|
|
- Windows with an NVIDIA GPU and CUDA. Training and the proof suite were built
|
|
on an RTX 4090; deployment targets a 16GB card such as an RTX 4080.
|
|
- The extended-context runtimes (SGLang, vLLM, KTransformers, llama.cpp) are
|
|
optional and run under WSL. The default Windows runtime uses Transformers and
|
|
needs none of them.
|
|
|
|
## Run it locally
|
|
|
|
```powershell
|
|
.\scripts\install.ps1 # virtual env, PyTorch + CUDA, Transformers, PEFT, bitsandbytes
|
|
.\scripts\download-models.ps1 # Qwen3.5-9B, the 0.8B planner, Qwen3-VL-Embedding-8B
|
|
.\scripts\serve-backend.ps1 # API on http://127.0.0.1:8765
|
|
.\scripts\serve-frontend.ps1 # dashboard on http://127.0.0.1:5173
|
|
```
|
|
|
|
When `ELY_EYE_ADAPTER_DIR` is empty, the runtime picks the latest trained
|
|
HME-Core-LoRA adapter for `Qwen/Qwen3.5-9B` on its own; `ely-eye status` prints
|
|
which adapter it resolved.
|
|
|
|
For the 262K and 1.01M extended-context profiles, set up the Linux runtimes once
|
|
and launch one:
|
|
|
|
```powershell
|
|
.\scripts\setup-linux-runtimes.ps1
|
|
.\scripts\launch-sglang-live.ps1 # 262K
|
|
.\scripts\launch-sglang-extreme.ps1 # 1.01M via YaRN
|
|
```
|
|
|
|
## Core commands
|
|
|
|
```powershell
|
|
.\.venv\Scripts\ely-eye.exe status
|
|
.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project
|
|
.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144
|
|
.\.venv\Scripts\ely-eye.exe proof-suite
|
|
```
|
|
|
|
Every command has an HTTP equivalent under `/api/...`, and the dashboard shows
|
|
the same state: Adapter Matrix, Proof Suite, Memory Map, Cache Trace.
|
|
|
|
## Fine-tuning
|
|
|
|
Training runs real BF16 LoRA updates on the GPU through Unsloth-patched
|
|
Transformers, PEFT, and bitsandbytes with paged 8-bit AdamW. Each run writes the
|
|
PEFT adapter alongside a manifest, a per-step training trace, a summary, and a
|
|
proof file that records dataset and weight hashes, per-step loss, optimizer-update
|
|
norms, and the CUDA/BF16 environment.
|
|
|
|
The adapter set:
|
|
|
|
```text
|
|
HME-Core-LoRA Qwen3.5-9B attention/MLP
|
|
HME-Vision-LoRA Qwen3.5-9B visual projector and selected vision layers
|
|
HME-TTT-VL Qwen3.5-9B mutable MLP and visual adapter
|
|
HME-Visual-MTP Qwen3.5-9B draft-head LoRA
|
|
HME-Router Qwen3.5-0.8B context planner
|
|
HME-Retrieval Qwen3-VL-Embedding-8B contrastive retrieval adapter
|
|
```
|
|
|
|
## Proof suite
|
|
|
|
`ely-eye proof-suite` runs fifteen checks against the live model and writes the
|
|
evidence under `.ely_eye/data/eval_proofs/`. Among them:
|
|
|
|
- real CUDA BF16 LoRA training — changed weight hashes, nonzero optimizer
|
|
updates, per-step loss
|
|
- the runtime loading the model and adapter and generating a cited answer
|
|
- Visual HashHop: multi-hop visual addressing the model has to actually solve,
|
|
with no OCR shortcut
|
|
- the 100M-token memory capsule committing to a reproducible Memory DNA
|
|
- the Visual Contradiction Lens localizing cross-version drift (design-token,
|
|
layout, typography, copy, visual-code, temporal) at >=80% recall with no false
|
|
positives
|
|
- standard benchmark samples (MMMU-Pro, OCRBench, and others) scored against
|
|
ground truth
|
|
|
|
```powershell
|
|
.\.venv\Scripts\ely-eye.exe proof-suite
|
|
.\.venv\Scripts\ely-eye.exe latest-proof-suite
|
|
.\.venv\Scripts\ely-eye.exe runtime-generation-proof
|
|
.\.venv\Scripts\ely-eye.exe visual-contradiction-proof
|
|
```
|
|
|
|
## MCP server for coding agents
|
|
|
|
`ely-eye-mcp/` is a local Model Context Protocol server that exposes the
|
|
`.ely_eye` memory store to coding agents such as Codex and Claude Code. One
|
|
command builds a native Go binary (or downloads a prebuilt one), installs it
|
|
under `.ely_eye/bin/`, and registers it with both clients. No Docker.
|
|
|
|
Windows:
|
|
|
|
```powershell
|
|
powershell -ExecutionPolicy Bypass -File .\ely-eye-mcp\install.ps1 -Client both
|
|
```
|
|
|
|
Linux and macOS:
|
|
|
|
```bash
|
|
bash ./ely-eye-mcp/install.sh --client both
|
|
```
|
|
|
|
The agent then has nine read-only tools over the live memory — status,
|
|
cartridge listing and detail, Evidence Atom search and fetch, evidence-pack
|
|
compilation, proof suites, and HashHop proofs. The server opens the database
|
|
read-only and never writes to it. The same installer works against any
|
|
`.ely_eye` store, not only this repository. See `ely-eye-mcp/README.md` for the
|
|
full tool list and the release pipeline.
|
|
|
|
## Data layout
|
|
|
|
Runtime state lives under `.ely_eye/` (git-ignored):
|
|
|
|
```text
|
|
.ely_eye/
|
|
├─ adapters/ trained LoRA adapters
|
|
├─ data/ sqlite database and eval proofs
|
|
├─ training/ datasets and training traces
|
|
├─ objects/ ingested source files
|
|
├─ cache/ cache fabric
|
|
└─ cartridges/ materialized Context Cartridges
|
|
```
|
|
|
|
A Context Cartridge is a self-contained memory package: the Evidence Atoms, a
|
|
sparse index and vector indexes, a temporal graph, KV snapshots, the memory
|
|
capsule, any attached adapters, and the eval proofs.
|
|
|
|
## Project layout
|
|
|
|
```text
|
|
backend/ Python: ingestion, retrieval, runtime, training, proof suite, FastAPI
|
|
frontend/ the local dashboard (Vite + React)
|
|
scripts/ install, model download, serving, runtimes, fine-tuning (.ps1 on Windows, .sh on Linux)
|
|
ely-eye-mcp/ Go: local MCP server exposing the memory store to coding agents
|
|
```
|
|
|
|
## License
|
|
|
|
AGPL-3.0-only. Copyright © 2026 雷电芽衣 <zhangyanghaha0407@outlook.com>.
|
|
|
|
The PDF reader (PyMuPDF) is dual-licensed under AGPL-3.0 or an Artifex
|
|
Commercial license, so this project ships under AGPL while that dependency is
|
|
in use. Buying the Artifex license is what would allow a permissive one.
|