ZacharyZhang-NY/ely-eye
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rewrite README — human-readable, drop PRD references

Browse Source
This commit is contained in:
ZacharyZhang-NY committed 2026-05-28 22:56:21 -04:00
1 parent 0ee99e271d
commit 0ca4022b21
1 file changed
+90 -114
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+90 -114
View File
@@ -1,60 +1,70 @@
# Ely-Eye
Ely-Eye is a local multimodal context operating system for Qwen3.5-9B. The implementation follows `PRD.md`: ingestion, Evidence Atoms, Context Cartridge, sparse and optional Qwen embedding retrieval, Cache Fabric, Qwen runtime adapters, verifier contracts, Visual HashHop proof generation, BF16 LoRA fine-tuning, adapter cartridge binding, and a local dashboard.
Ely-Eye is a local multimodal context operating system built on Qwen3.5-9B. It
runs on a single consumer GPU and gives a small model three tiers of context: a
262K native window, a 1.01M extended window, and a 100M-token memory capsule it
can retrieve from, cite, and recall across sessions.
## Local Run
Under the hood it ingests PDFs, screenshots, video, and code into Evidence
Atoms, packs them into a Context Cartridge, retrieves with sparse and Qwen
embedding search, runs Qwen3.5-9B with trained LoRA adapters, and checks every
answer with a verifier. A fifteen-check proof suite exercises the whole thing
against the live model, so a real run can be told apart from a fabricated one.
The landing site lives at https://ely-eye-web.zhangyanghaha0407.workers.dev
(its source is under `ely-eye-web/`, which this repository does not track).
## Requirements
- Windows with an NVIDIA GPU and CUDA. Training and the proof suite were built
on an RTX 4090; deployment targets a 16GB card such as an RTX 4080.
- The extended-context runtimes (SGLang, vLLM, KTransformers, llama.cpp) are
optional and run under WSL. The default Windows runtime uses Transformers and
needs none of them.
## Run it locally
```powershell
.\scripts\install.ps1 # virtual env, PyTorch + CUDA, Transformers, PEFT, bitsandbytes
.\scripts\download-models.ps1 # Qwen3.5-9B, the 0.8B planner, Qwen3-VL-Embedding-8B
.\scripts\serve-backend.ps1 # API on http://127.0.0.1:8765
.\scripts\serve-frontend.ps1 # dashboard on http://127.0.0.1:5173
```
When `ELY_EYE_ADAPTER_DIR` is empty, the runtime picks the latest trained
HME-Core-LoRA adapter for `Qwen/Qwen3.5-9B` on its own; `ely-eye status` prints
which adapter it resolved.
For the 262K and 1.01M extended-context profiles, set up the Linux runtimes once
and launch one:
```powershell
Set-Location C:\Users\zacharyzhang\Documents\Github\ELY-EYE
.\scripts\install.ps1
.\scripts\setup-linux-runtimes.ps1
.\scripts\serve-backend.ps1
.\scripts\serve-frontend.ps1
.\scripts\launch-sglang-live.ps1 # 262K
.\scripts\launch-sglang-extreme.ps1 # 1.01M via YaRN
```
Backend: `http://127.0.0.1:8765`
Frontend: `http://127.0.0.1:5173`
## Model Runtime
SGLang is supported as an OpenAI-compatible runtime target. The Windows local runtime uses Transformers.
## Core commands
```powershell
.\scripts\download-models.ps1
.\scripts\launch-sglang-live.ps1
```
The Transformers backend is available by setting:
```powershell
$env:ELY_EYE_RUNTIME_BACKEND = "transformers"
```
When `ELY_EYE_ADAPTER_DIR` is empty, the Transformers runtime auto-selects the
latest trained `HME-Core-LoRA` adapter whose base model matches
`Qwen/Qwen3.5-9B`. `ely-eye status` reports the resolved adapter path, kind,
adapter id, and adapter hash.
The runtime generation proof loads `Qwen/Qwen3.5-9B`, injects the trained
`HME-Core-LoRA` adapter, generates a PRD-cited answer, and writes the raw model
output plus verifier report:
```powershell
.\.venv\Scripts\ely-eye.exe runtime-generation-proof
```
## Fine-Tuning
The PRD-derived datasets are built from real `PRD.md` sections, the ingested PRD Evidence Atom, and local UI proof images. Training runs real BF16 LoRA optimizer updates on RTX 4090 through Unsloth-patched Transformers, PEFT, and bitsandbytes, then writes PEFT adapters, `adapter_manifest.json`, `training_trace.jsonl`, `training_summary.json`, `training_proof.json`, dataset hashes, adapter hashes, trainable parameter hashes, optimizer update norms, per-step loss, CUDA/BF16 environment metadata, target modules, adapter weight SHA-256, LoRA tensor statistics, and cartridge binding metadata. The training path requires CUDA BF16 and bitsandbytes `PagedAdamW8bit`; every proof records proof version, precision, optimizer family, checkpointing mode, max length, framework, and target-module coverage.
```powershell
.\scripts\run-prd-finetune.ps1 -MaxSteps 37 -CartridgeId ely_eye_prd_7e5cd27b8284
.\scripts\run-all-prd-finetunes.ps1 -CartridgeId ely_eye_prd_7e5cd27b8284
$env:ELY_EYE_ADAPTER_DIR = ".ely_eye\adapters\hme_core_lora_qwen35_9b"
.\.venv\Scripts\ely-eye.exe status
.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project
.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144
.\.venv\Scripts\ely-eye.exe proof-suite
```
The PRD adapter set is:
Every command has an HTTP equivalent under `/api/...`, and the dashboard shows
the same state: Adapter Matrix, Proof Suite, Memory Map, Cache Trace.
## Fine-tuning
Training runs real BF16 LoRA updates on the GPU through Unsloth-patched
Transformers, PEFT, and bitsandbytes with paged 8-bit AdamW. Each run writes the
PEFT adapter alongside a manifest, a per-step training trace, a summary, and a
proof file that records dataset and weight hashes, per-step loss, optimizer-update
norms, and the CUDA/BF16 environment.
The adapter set:
```text
HME-Core-LoRA Qwen3.5-9B attention/MLP
@@ -65,86 +75,52 @@ HME-Router Qwen3.5-0.8B context planner
HME-Retrieval Qwen3-VL-Embedding-8B contrastive retrieval adapter
```
## Core Commands
## Proof suite
`ely-eye proof-suite` runs fifteen checks against the live model and writes the
evidence under `.ely_eye/data/eval_proofs/`. Among them:
- real CUDA BF16 LoRA training — changed weight hashes, nonzero optimizer
updates, per-step loss
- the runtime loading the model and adapter and generating a cited answer
- Visual HashHop: multi-hop visual addressing the model has to actually solve,
with no OCR shortcut
- the 100M-token memory capsule committing to a reproducible Memory DNA
- the Visual Contradiction Lens localizing cross-version drift (design-token,
layout, typography, copy, visual-code, temporal) at >=80% recall with no false
positives
- standard benchmark samples (MMMU-Pro, OCRBench, and others) scored against
ground truth
```powershell
.\.venv\Scripts\ely-eye.exe status
.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project-cartridge
.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144
.\.venv\Scripts\ely-eye.exe validate-training-data C:\path\to\train.jsonl
.\.venv\Scripts\ely-eye.exe train-lora C:\path\to\train.jsonl C:\path\to\adapter
```
Adapter proof state is available through `GET /api/adapters` and the dashboard Adapter Matrix, including training method, precision, optimizer family, checkpointing state, optimizer update counts, and loss delta.
## PRD Proof Suite
The proof suite verifies the PRD adapter set with CUDA BF16 optimizer updates,
changed trainable parameter hashes, matching manifest/summary/proof hashes,
proof-versioned BF16 LoRA contracts, 8192-token max length, paged 8-bit AdamW,
checkpointing, target-module coverage, nonzero LoRA safetensors, runtime adapter binding, runtime adapter generation,
cartridge replay, Visual HashHop assets, model-solved text HashHop, model-solved
Visual HashHop 1-hop and 2-hop pass-rate gates, learned memory adapter binding,
100M memory capsule commitments, Memory DNA, cache trace evidence, and Visual
Contradiction Lens cross-version drift detection (>=80% recall with zero false
positives across the PRD 11.15 taxonomy: design-token, layout, typography, copy,
visual-code, and temporal drift).
It also verifies code repository input coverage through a local structure index,
call graph extraction, and git issue/PR history association.
It writes proof JSON, generated HashHop visual blocks, raw model HashHop
outputs, Visual HashHop arena images, and official benchmark sample predictions under `.ely_eye/data/eval_proofs/`, then
attaches the suite to the active Context Cartridge under `eval_proofs/`.
MMMU-Pro and OCRBench sample predictions must include finite local scores before
the standard benchmark check passes. Every standard benchmark sample must include
a sample SHA-256 and source dataset SHA. LongVideoBench uses the official
`longvideobench/LongVideoBench-Meta` public validation metadata source while the
canonical gated dataset remains tracked in the benchmark registry.
Latest local PRD proof suite: `prd_proof_suite_6bc3fa744c60a0e6` (15 checks passed).
```powershell
.\.venv\Scripts\ely-eye.exe proof-suite --cartridge-id ely_eye_prd_7e5cd27b8284
.\.venv\Scripts\ely-eye.exe proof-suite
.\.venv\Scripts\ely-eye.exe latest-proof-suite
.\.venv\Scripts\ely-eye.exe runtime-generation-proof
.\.venv\Scripts\ely-eye.exe visual-contradiction-proof
.\.venv\Scripts\ely-eye.exe refresh-cartridge ely_eye_prd_7e5cd27b8284
.\.venv\Scripts\ely-eye.exe finalize-cartridge ely_eye_prd_7e5cd27b8284
.\.venv\Scripts\ely-eye.exe benchmark-registry
.\.venv\Scripts\ely-eye.exe benchmark-sources
.\.venv\Scripts\ely-eye.exe benchmark-samples
.\.venv\Scripts\ely-eye.exe benchmark-predictions
.\.venv\Scripts\ely-eye.exe runtime-profiles
```
The same surface is available through `POST /api/proof-suite`, `GET /api/proof-suite/latest`, `POST /api/visual-contradiction-proof`, `POST /api/cartridges/{cartridge_id}/refresh`, `POST /api/cartridges/{cartridge_id}/finalize`, `GET /api/cartridges/{cartridge_id}/assets`, `GET /api/benchmarks/registry`, `GET /api/benchmarks/sources`, `GET /api/benchmarks/samples`, `GET /api/benchmarks/predictions`, `GET /api/runtime/profiles`, and the dashboard PRD Proof Suite panel.
## Data layout
Runtime launchers:
`ely-eye runtime-profiles` validates the WSL GPU, Linux runtime packages,
SGLang/vLLM launcher modules, KTransformers import path, and llama.cpp help
output before the proof suite marks runtime delivery ready.
```powershell
.\scripts\setup-linux-runtimes.ps1
.\scripts\launch-sglang-live.ps1
.\scripts\launch-sglang-extreme.ps1
.\scripts\launch-vllm-live.ps1
.\scripts\launch-vllm-extreme.ps1
.\scripts\launch-ktransformers-extreme.ps1
.\scripts\launch-llamacpp-offline.ps1
```
## Data Layout
Runtime state is stored under `.ely_eye/` by default.
Runtime state lives under `.ely_eye/` (git-ignored):
```text
.ely_eye/
├─ adapters/
├─ data/ely_eye.sqlite
├─ training/
├─ objects/
├─ cache/
└─ cartridges/
├─ adapters/ trained LoRA adapters
├─ data/ sqlite database and eval proofs
├─ training/ datasets and training traces
├─ objects/ ingested source files
├─ cache/ cache fabric
└─ cartridges/ materialized Context Cartridges
```
Context Cartridge artifacts are materialized as `manifest.json`, `atoms.parquet`, `sources.json`, `sparse_index.bm25.zst`, `temporal_graph.sqlite`, `text_vectors.fp16.zstd`, `visual_vectors.int8.zstd`, `kv_snapshots.kivi`, `memory_capsule.json.zst`, `memory_capsule_index.json`, `ttt_vl_adapter.safetensors`, `visual_mtp_head.safetensors`, `eval_proofs/`, and attached adapter folders.
A Context Cartridge is a self-contained memory package: the Evidence Atoms, a
sparse index and vector indexes, a temporal graph, KV snapshots, the memory
capsule, any attached adapters, and the eval proofs.
## Project layout
```text
backend/ Python: ingestion, retrieval, runtime, training, proof suite, FastAPI
frontend/ the local dashboard (Vite + React)
scripts/ PowerShell: install, model download, serving, runtimes, fine-tuning
```