From 0ca4022b21fa07f205265c6f5c85dfae2ed22f99 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=9B=B7=E7=94=B5=E8=8A=BD=E8=A1=A3?= Date: Thu, 28 May 2026 22:56:21 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20rewrite=20README=20=E2=80=94=20human-re?= =?UTF-8?q?adable,=20drop=20PRD=20references?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 204 ++++++++++++++++++++++++------------------------------ 1 file changed, 90 insertions(+), 114 deletions(-) diff --git a/README.md b/README.md index bbb72d5..776a5ac 100644 --- a/README.md +++ b/README.md @@ -1,60 +1,70 @@ # Ely-Eye -Ely-Eye is a local multimodal context operating system for Qwen3.5-9B. The implementation follows `PRD.md`: ingestion, Evidence Atoms, Context Cartridge, sparse and optional Qwen embedding retrieval, Cache Fabric, Qwen runtime adapters, verifier contracts, Visual HashHop proof generation, BF16 LoRA fine-tuning, adapter cartridge binding, and a local dashboard. +Ely-Eye is a local multimodal context operating system built on Qwen3.5-9B. It +runs on a single consumer GPU and gives a small model three tiers of context: a +262K native window, a 1.01M extended window, and a 100M-token memory capsule it +can retrieve from, cite, and recall across sessions. -## Local Run +Under the hood it ingests PDFs, screenshots, video, and code into Evidence +Atoms, packs them into a Context Cartridge, retrieves with sparse and Qwen +embedding search, runs Qwen3.5-9B with trained LoRA adapters, and checks every +answer with a verifier. A fifteen-check proof suite exercises the whole thing +against the live model, so a real run can be told apart from a fabricated one. + +The landing site lives at https://ely-eye-web.zhangyanghaha0407.workers.dev +(its source is under `ely-eye-web/`, which this repository does not track). + +## Requirements + +- Windows with an NVIDIA GPU and CUDA. Training and the proof suite were built + on an RTX 4090; deployment targets a 16GB card such as an RTX 4080. +- The extended-context runtimes (SGLang, vLLM, KTransformers, llama.cpp) are + optional and run under WSL. The default Windows runtime uses Transformers and + needs none of them. + +## Run it locally + +```powershell +.\scripts\install.ps1 # virtual env, PyTorch + CUDA, Transformers, PEFT, bitsandbytes +.\scripts\download-models.ps1 # Qwen3.5-9B, the 0.8B planner, Qwen3-VL-Embedding-8B +.\scripts\serve-backend.ps1 # API on http://127.0.0.1:8765 +.\scripts\serve-frontend.ps1 # dashboard on http://127.0.0.1:5173 +``` + +When `ELY_EYE_ADAPTER_DIR` is empty, the runtime picks the latest trained +HME-Core-LoRA adapter for `Qwen/Qwen3.5-9B` on its own; `ely-eye status` prints +which adapter it resolved. + +For the 262K and 1.01M extended-context profiles, set up the Linux runtimes once +and launch one: ```powershell -Set-Location C:\Users\zacharyzhang\Documents\Github\ELY-EYE -.\scripts\install.ps1 .\scripts\setup-linux-runtimes.ps1 -.\scripts\serve-backend.ps1 -.\scripts\serve-frontend.ps1 +.\scripts\launch-sglang-live.ps1 # 262K +.\scripts\launch-sglang-extreme.ps1 # 1.01M via YaRN ``` -Backend: `http://127.0.0.1:8765` -Frontend: `http://127.0.0.1:5173` - -## Model Runtime - -SGLang is supported as an OpenAI-compatible runtime target. The Windows local runtime uses Transformers. +## Core commands ```powershell -.\scripts\download-models.ps1 -.\scripts\launch-sglang-live.ps1 -``` - -The Transformers backend is available by setting: - -```powershell -$env:ELY_EYE_RUNTIME_BACKEND = "transformers" -``` - -When `ELY_EYE_ADAPTER_DIR` is empty, the Transformers runtime auto-selects the -latest trained `HME-Core-LoRA` adapter whose base model matches -`Qwen/Qwen3.5-9B`. `ely-eye status` reports the resolved adapter path, kind, -adapter id, and adapter hash. - -The runtime generation proof loads `Qwen/Qwen3.5-9B`, injects the trained -`HME-Core-LoRA` adapter, generates a PRD-cited answer, and writes the raw model -output plus verifier report: - -```powershell -.\.venv\Scripts\ely-eye.exe runtime-generation-proof -``` - -## Fine-Tuning - -The PRD-derived datasets are built from real `PRD.md` sections, the ingested PRD Evidence Atom, and local UI proof images. Training runs real BF16 LoRA optimizer updates on RTX 4090 through Unsloth-patched Transformers, PEFT, and bitsandbytes, then writes PEFT adapters, `adapter_manifest.json`, `training_trace.jsonl`, `training_summary.json`, `training_proof.json`, dataset hashes, adapter hashes, trainable parameter hashes, optimizer update norms, per-step loss, CUDA/BF16 environment metadata, target modules, adapter weight SHA-256, LoRA tensor statistics, and cartridge binding metadata. The training path requires CUDA BF16 and bitsandbytes `PagedAdamW8bit`; every proof records proof version, precision, optimizer family, checkpointing mode, max length, framework, and target-module coverage. - -```powershell -.\scripts\run-prd-finetune.ps1 -MaxSteps 37 -CartridgeId ely_eye_prd_7e5cd27b8284 -.\scripts\run-all-prd-finetunes.ps1 -CartridgeId ely_eye_prd_7e5cd27b8284 -$env:ELY_EYE_ADAPTER_DIR = ".ely_eye\adapters\hme_core_lora_qwen35_9b" .\.venv\Scripts\ely-eye.exe status +.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project +.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144 +.\.venv\Scripts\ely-eye.exe proof-suite ``` -The PRD adapter set is: +Every command has an HTTP equivalent under `/api/...`, and the dashboard shows +the same state: Adapter Matrix, Proof Suite, Memory Map, Cache Trace. + +## Fine-tuning + +Training runs real BF16 LoRA updates on the GPU through Unsloth-patched +Transformers, PEFT, and bitsandbytes with paged 8-bit AdamW. Each run writes the +PEFT adapter alongside a manifest, a per-step training trace, a summary, and a +proof file that records dataset and weight hashes, per-step loss, optimizer-update +norms, and the CUDA/BF16 environment. + +The adapter set: ```text HME-Core-LoRA Qwen3.5-9B attention/MLP @@ -65,86 +75,52 @@ HME-Router Qwen3.5-0.8B context planner HME-Retrieval Qwen3-VL-Embedding-8B contrastive retrieval adapter ``` -## Core Commands +## Proof suite + +`ely-eye proof-suite` runs fifteen checks against the live model and writes the +evidence under `.ely_eye/data/eval_proofs/`. Among them: + +- real CUDA BF16 LoRA training — changed weight hashes, nonzero optimizer + updates, per-step loss +- the runtime loading the model and adapter and generating a cited answer +- Visual HashHop: multi-hop visual addressing the model has to actually solve, + with no OCR shortcut +- the 100M-token memory capsule committing to a reproducible Memory DNA +- the Visual Contradiction Lens localizing cross-version drift (design-token, + layout, typography, copy, visual-code, temporal) at >=80% recall with no false + positives +- standard benchmark samples (MMMU-Pro, OCRBench, and others) scored against + ground truth ```powershell -.\.venv\Scripts\ely-eye.exe status -.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project-cartridge -.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144 -.\.venv\Scripts\ely-eye.exe validate-training-data C:\path\to\train.jsonl -.\.venv\Scripts\ely-eye.exe train-lora C:\path\to\train.jsonl C:\path\to\adapter -``` - -Adapter proof state is available through `GET /api/adapters` and the dashboard Adapter Matrix, including training method, precision, optimizer family, checkpointing state, optimizer update counts, and loss delta. - -## PRD Proof Suite - -The proof suite verifies the PRD adapter set with CUDA BF16 optimizer updates, -changed trainable parameter hashes, matching manifest/summary/proof hashes, -proof-versioned BF16 LoRA contracts, 8192-token max length, paged 8-bit AdamW, -checkpointing, target-module coverage, nonzero LoRA safetensors, runtime adapter binding, runtime adapter generation, -cartridge replay, Visual HashHop assets, model-solved text HashHop, model-solved -Visual HashHop 1-hop and 2-hop pass-rate gates, learned memory adapter binding, -100M memory capsule commitments, Memory DNA, cache trace evidence, and Visual -Contradiction Lens cross-version drift detection (>=80% recall with zero false -positives across the PRD 11.15 taxonomy: design-token, layout, typography, copy, -visual-code, and temporal drift). -It also verifies code repository input coverage through a local structure index, -call graph extraction, and git issue/PR history association. -It writes proof JSON, generated HashHop visual blocks, raw model HashHop -outputs, Visual HashHop arena images, and official benchmark sample predictions under `.ely_eye/data/eval_proofs/`, then -attaches the suite to the active Context Cartridge under `eval_proofs/`. -MMMU-Pro and OCRBench sample predictions must include finite local scores before -the standard benchmark check passes. Every standard benchmark sample must include -a sample SHA-256 and source dataset SHA. LongVideoBench uses the official -`longvideobench/LongVideoBench-Meta` public validation metadata source while the -canonical gated dataset remains tracked in the benchmark registry. - -Latest local PRD proof suite: `prd_proof_suite_6bc3fa744c60a0e6` (15 checks passed). - -```powershell -.\.venv\Scripts\ely-eye.exe proof-suite --cartridge-id ely_eye_prd_7e5cd27b8284 +.\.venv\Scripts\ely-eye.exe proof-suite .\.venv\Scripts\ely-eye.exe latest-proof-suite .\.venv\Scripts\ely-eye.exe runtime-generation-proof .\.venv\Scripts\ely-eye.exe visual-contradiction-proof -.\.venv\Scripts\ely-eye.exe refresh-cartridge ely_eye_prd_7e5cd27b8284 -.\.venv\Scripts\ely-eye.exe finalize-cartridge ely_eye_prd_7e5cd27b8284 -.\.venv\Scripts\ely-eye.exe benchmark-registry -.\.venv\Scripts\ely-eye.exe benchmark-sources -.\.venv\Scripts\ely-eye.exe benchmark-samples -.\.venv\Scripts\ely-eye.exe benchmark-predictions -.\.venv\Scripts\ely-eye.exe runtime-profiles ``` -The same surface is available through `POST /api/proof-suite`, `GET /api/proof-suite/latest`, `POST /api/visual-contradiction-proof`, `POST /api/cartridges/{cartridge_id}/refresh`, `POST /api/cartridges/{cartridge_id}/finalize`, `GET /api/cartridges/{cartridge_id}/assets`, `GET /api/benchmarks/registry`, `GET /api/benchmarks/sources`, `GET /api/benchmarks/samples`, `GET /api/benchmarks/predictions`, `GET /api/runtime/profiles`, and the dashboard PRD Proof Suite panel. +## Data layout -Runtime launchers: -`ely-eye runtime-profiles` validates the WSL GPU, Linux runtime packages, -SGLang/vLLM launcher modules, KTransformers import path, and llama.cpp help -output before the proof suite marks runtime delivery ready. - -```powershell -.\scripts\setup-linux-runtimes.ps1 -.\scripts\launch-sglang-live.ps1 -.\scripts\launch-sglang-extreme.ps1 -.\scripts\launch-vllm-live.ps1 -.\scripts\launch-vllm-extreme.ps1 -.\scripts\launch-ktransformers-extreme.ps1 -.\scripts\launch-llamacpp-offline.ps1 -``` - -## Data Layout - -Runtime state is stored under `.ely_eye/` by default. +Runtime state lives under `.ely_eye/` (git-ignored): ```text .ely_eye/ -├─ adapters/ -├─ data/ely_eye.sqlite -├─ training/ -├─ objects/ -├─ cache/ -└─ cartridges/ +├─ adapters/ trained LoRA adapters +├─ data/ sqlite database and eval proofs +├─ training/ datasets and training traces +├─ objects/ ingested source files +├─ cache/ cache fabric +└─ cartridges/ materialized Context Cartridges ``` -Context Cartridge artifacts are materialized as `manifest.json`, `atoms.parquet`, `sources.json`, `sparse_index.bm25.zst`, `temporal_graph.sqlite`, `text_vectors.fp16.zstd`, `visual_vectors.int8.zstd`, `kv_snapshots.kivi`, `memory_capsule.json.zst`, `memory_capsule_index.json`, `ttt_vl_adapter.safetensors`, `visual_mtp_head.safetensors`, `eval_proofs/`, and attached adapter folders. +A Context Cartridge is a self-contained memory package: the Evidence Atoms, a +sparse index and vector indexes, a temporal graph, KV snapshots, the memory +capsule, any attached adapters, and the eval proofs. + +## Project layout + +```text +backend/ Python: ingestion, retrieval, runtime, training, proof suite, FastAPI +frontend/ the local dashboard (Vite + React) +scripts/ PowerShell: install, model download, serving, runtimes, fine-tuning +```