Ely-Eye

Ely-Eye is a local multimodal context operating system for Qwen3.5-9B. The implementation follows PRD.md: ingestion, Evidence Atoms, Context Cartridge, sparse and optional Qwen embedding retrieval, Cache Fabric, Qwen runtime adapters, verifier contracts, Visual HashHop proof generation, BF16 LoRA fine-tuning, adapter cartridge binding, and a local dashboard.

Local Run

Set-Location C:\Users\zacharyzhang\Documents\Github\ELY-EYE
.\scripts\install.ps1
.\scripts\setup-linux-runtimes.ps1
.\scripts\serve-backend.ps1
.\scripts\serve-frontend.ps1

Backend: http://127.0.0.1:8765
Frontend: http://127.0.0.1:5173

Model Runtime

SGLang is supported as an OpenAI-compatible runtime target. The Windows local runtime uses Transformers.

.\scripts\download-models.ps1
.\scripts\launch-sglang-live.ps1

The Transformers backend is available by setting:

$env:ELY_EYE_RUNTIME_BACKEND = "transformers"

When ELY_EYE_ADAPTER_DIR is empty, the Transformers runtime auto-selects the latest trained HME-Core-LoRA adapter whose base model matches Qwen/Qwen3.5-9B. ely-eye status reports the resolved adapter path, kind, adapter id, and adapter hash.

The runtime generation proof loads Qwen/Qwen3.5-9B, injects the trained HME-Core-LoRA adapter, generates a PRD-cited answer, and writes the raw model output plus verifier report:

.\.venv\Scripts\ely-eye.exe runtime-generation-proof

Fine-Tuning

The PRD-derived datasets are built from real PRD.md sections, the ingested PRD Evidence Atom, and local UI proof images. Training runs real BF16 LoRA optimizer updates on RTX 4090 through Unsloth-patched Transformers, PEFT, and bitsandbytes, then writes PEFT adapters, adapter_manifest.json, training_trace.jsonl, training_summary.json, training_proof.json, dataset hashes, adapter hashes, trainable parameter hashes, optimizer update norms, per-step loss, CUDA/BF16 environment metadata, target modules, adapter weight SHA-256, LoRA tensor statistics, and cartridge binding metadata. The training path requires CUDA BF16 and bitsandbytes PagedAdamW8bit; every proof records proof version, precision, optimizer family, checkpointing mode, max length, framework, and target-module coverage.

.\scripts\run-prd-finetune.ps1 -MaxSteps 37 -CartridgeId ely_eye_prd_7e5cd27b8284
.\scripts\run-all-prd-finetunes.ps1 -CartridgeId ely_eye_prd_7e5cd27b8284
$env:ELY_EYE_ADAPTER_DIR = ".ely_eye\adapters\hme_core_lora_qwen35_9b"
.\.venv\Scripts\ely-eye.exe status

The PRD adapter set is:

HME-Core-LoRA      Qwen3.5-9B attention/MLP
HME-Vision-LoRA    Qwen3.5-9B visual projector and selected vision layers
HME-TTT-VL         Qwen3.5-9B mutable MLP and visual adapter
HME-Visual-MTP     Qwen3.5-9B draft-head LoRA
HME-Router         Qwen3.5-0.8B context planner
HME-Retrieval      Qwen3-VL-Embedding-8B contrastive retrieval adapter

Core Commands

.\.venv\Scripts\ely-eye.exe status
.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project-cartridge
.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144
.\.venv\Scripts\ely-eye.exe validate-training-data C:\path\to\train.jsonl
.\.venv\Scripts\ely-eye.exe train-lora C:\path\to\train.jsonl C:\path\to\adapter

Adapter proof state is available through GET /api/adapters and the dashboard Adapter Matrix, including training method, precision, optimizer family, checkpointing state, optimizer update counts, and loss delta.

PRD Proof Suite

The proof suite verifies the PRD adapter set with CUDA BF16 optimizer updates, changed trainable parameter hashes, matching manifest/summary/proof hashes, proof-versioned BF16 LoRA contracts, 8192-token max length, paged 8-bit AdamW, checkpointing, target-module coverage, nonzero LoRA safetensors, runtime adapter binding, runtime adapter generation, cartridge replay, Visual HashHop assets, model-solved text HashHop, model-solved Visual HashHop 1-hop and 2-hop pass-rate gates, learned memory adapter binding, 100M memory capsule commitments, Memory DNA, cache trace evidence, and Visual Contradiction Lens cross-version drift detection (>=80% recall with zero false positives across the PRD 11.15 taxonomy: design-token, layout, typography, copy, visual-code, and temporal drift). It also verifies code repository input coverage through a local structure index, call graph extraction, and git issue/PR history association. It writes proof JSON, generated HashHop visual blocks, raw model HashHop outputs, Visual HashHop arena images, and official benchmark sample predictions under .ely_eye/data/eval_proofs/, then attaches the suite to the active Context Cartridge under eval_proofs/. MMMU-Pro and OCRBench sample predictions must include finite local scores before the standard benchmark check passes. Every standard benchmark sample must include a sample SHA-256 and source dataset SHA. LongVideoBench uses the official longvideobench/LongVideoBench-Meta public validation metadata source while the canonical gated dataset remains tracked in the benchmark registry.

Latest local PRD proof suite: prd_proof_suite_6bc3fa744c60a0e6 (15 checks passed).

.\.venv\Scripts\ely-eye.exe proof-suite --cartridge-id ely_eye_prd_7e5cd27b8284
.\.venv\Scripts\ely-eye.exe latest-proof-suite
.\.venv\Scripts\ely-eye.exe runtime-generation-proof
.\.venv\Scripts\ely-eye.exe visual-contradiction-proof
.\.venv\Scripts\ely-eye.exe refresh-cartridge ely_eye_prd_7e5cd27b8284
.\.venv\Scripts\ely-eye.exe finalize-cartridge ely_eye_prd_7e5cd27b8284
.\.venv\Scripts\ely-eye.exe benchmark-registry
.\.venv\Scripts\ely-eye.exe benchmark-sources
.\.venv\Scripts\ely-eye.exe benchmark-samples
.\.venv\Scripts\ely-eye.exe benchmark-predictions
.\.venv\Scripts\ely-eye.exe runtime-profiles

The same surface is available through POST /api/proof-suite, GET /api/proof-suite/latest, POST /api/visual-contradiction-proof, POST /api/cartridges/{cartridge_id}/refresh, POST /api/cartridges/{cartridge_id}/finalize, GET /api/cartridges/{cartridge_id}/assets, GET /api/benchmarks/registry, GET /api/benchmarks/sources, GET /api/benchmarks/samples, GET /api/benchmarks/predictions, GET /api/runtime/profiles, and the dashboard PRD Proof Suite panel.

Runtime launchers: ely-eye runtime-profiles validates the WSL GPU, Linux runtime packages, SGLang/vLLM launcher modules, KTransformers import path, and llama.cpp help output before the proof suite marks runtime delivery ready.

.\scripts\setup-linux-runtimes.ps1
.\scripts\launch-sglang-live.ps1
.\scripts\launch-sglang-extreme.ps1
.\scripts\launch-vllm-live.ps1
.\scripts\launch-vllm-extreme.ps1
.\scripts\launch-ktransformers-extreme.ps1
.\scripts\launch-llamacpp-offline.ps1

Data Layout

Runtime state is stored under .ely_eye/ by default.

.ely_eye/
├─ adapters/
├─ data/ely_eye.sqlite
├─ training/
├─ objects/
├─ cache/
└─ cartridges/

Context Cartridge artifacts are materialized as manifest.json, atoms.parquet, sources.json, sparse_index.bm25.zst, temporal_graph.sqlite, text_vectors.fp16.zstd, visual_vectors.int8.zstd, kv_snapshots.kivi, memory_capsule.json.zst, memory_capsule_index.json, ttt_vl_adapter.safetensors, visual_mtp_head.safetensors, eval_proofs/, and attached adapter folders.