ely-eye/README.md

# Ely-Eye

Ely-Eye is a local multimodal context operating system for Qwen3.5-9B. The implementation follows `PRD.md`: ingestion, Evidence Atoms, Context Cartridge, sparse and optional Qwen embedding retrieval, Cache Fabric, Qwen runtime adapters, verifier contracts, Visual HashHop proof generation, BF16 LoRA fine-tuning, adapter cartridge binding, and a local dashboard.

## Local Run

```powershell
Set-Location C:\Users\zacharyzhang\Documents\Github\ELY-EYE
.\scripts\install.ps1
.\scripts\setup-linux-runtimes.ps1
.\scripts\serve-backend.ps1
.\scripts\serve-frontend.ps1
```

Backend: `http://127.0.0.1:8765`  
Frontend: `http://127.0.0.1:5173`

## Model Runtime

SGLang is supported as an OpenAI-compatible runtime target. The Windows local runtime uses Transformers.

```powershell
.\scripts\download-models.ps1
.\scripts\launch-sglang-live.ps1
```

The Transformers backend is available by setting:

```powershell
$env:ELY_EYE_RUNTIME_BACKEND = "transformers"
```

When `ELY_EYE_ADAPTER_DIR` is empty, the Transformers runtime auto-selects the
latest trained `HME-Core-LoRA` adapter whose base model matches
`Qwen/Qwen3.5-9B`. `ely-eye status` reports the resolved adapter path, kind,
adapter id, and adapter hash.

The runtime generation proof loads `Qwen/Qwen3.5-9B`, injects the trained
`HME-Core-LoRA` adapter, generates a PRD-cited answer, and writes the raw model
output plus verifier report:

```powershell
.\.venv\Scripts\ely-eye.exe runtime-generation-proof
```

## Fine-Tuning

The PRD-derived datasets are built from real `PRD.md` sections, the ingested PRD Evidence Atom, and local UI proof images. Training runs real BF16 LoRA optimizer updates on RTX 4090 through Unsloth-patched Transformers, PEFT, and bitsandbytes, then writes PEFT adapters, `adapter_manifest.json`, `training_trace.jsonl`, `training_summary.json`, `training_proof.json`, dataset hashes, adapter hashes, trainable parameter hashes, optimizer update norms, per-step loss, CUDA/BF16 environment metadata, target modules, adapter weight SHA-256, LoRA tensor statistics, and cartridge binding metadata. The training path requires CUDA BF16 and bitsandbytes `PagedAdamW8bit`; every proof records proof version, precision, optimizer family, checkpointing mode, max length, framework, and target-module coverage.

```powershell
.\scripts\run-prd-finetune.ps1 -MaxSteps 37 -CartridgeId ely_eye_prd_7e5cd27b8284
.\scripts\run-all-prd-finetunes.ps1 -CartridgeId ely_eye_prd_7e5cd27b8284
$env:ELY_EYE_ADAPTER_DIR = ".ely_eye\adapters\hme_core_lora_qwen35_9b"
.\.venv\Scripts\ely-eye.exe status
```

The PRD adapter set is:

```text
HME-Core-LoRA      Qwen3.5-9B attention/MLP
HME-Vision-LoRA    Qwen3.5-9B visual projector and selected vision layers
HME-TTT-VL         Qwen3.5-9B mutable MLP and visual adapter
HME-Visual-MTP     Qwen3.5-9B draft-head LoRA
HME-Router         Qwen3.5-0.8B context planner
HME-Retrieval      Qwen3-VL-Embedding-8B contrastive retrieval adapter
```

## Core Commands

```powershell
.\.venv\Scripts\ely-eye.exe status
.\.venv\Scripts\ely-eye.exe ingest C:\path\to\package --cartridge-name project-cartridge
.\.venv\Scripts\ely-eye.exe hashhop --kind visual --hops 2 --token-equivalent 262144
.\.venv\Scripts\ely-eye.exe validate-training-data C:\path\to\train.jsonl
.\.venv\Scripts\ely-eye.exe train-lora C:\path\to\train.jsonl C:\path\to\adapter
```

Adapter proof state is available through `GET /api/adapters` and the dashboard Adapter Matrix, including training method, precision, optimizer family, checkpointing state, optimizer update counts, and loss delta.

## PRD Proof Suite

The proof suite verifies the PRD adapter set with CUDA BF16 optimizer updates,
changed trainable parameter hashes, matching manifest/summary/proof hashes,
proof-versioned BF16 LoRA contracts, 8192-token max length, paged 8-bit AdamW,
checkpointing, target-module coverage, nonzero LoRA safetensors, runtime adapter binding, runtime adapter generation,
cartridge replay, Visual HashHop assets, model-solved text HashHop, model-solved
Visual HashHop 1-hop and 2-hop pass-rate gates, learned memory adapter binding,
100M memory capsule commitments, Memory DNA, cache trace evidence, and Visual
Contradiction Lens cross-version drift detection (>=80% recall with zero false
positives across the PRD 11.15 taxonomy: design-token, layout, typography, copy,
visual-code, and temporal drift).
It also verifies code repository input coverage through a local structure index,
call graph extraction, and git issue/PR history association.
It writes proof JSON, generated HashHop visual blocks, raw model HashHop
outputs, Visual HashHop arena images, and official benchmark sample predictions under `.ely_eye/data/eval_proofs/`, then
attaches the suite to the active Context Cartridge under `eval_proofs/`.
MMMU-Pro and OCRBench sample predictions must include finite local scores before
the standard benchmark check passes. Every standard benchmark sample must include
a sample SHA-256 and source dataset SHA. LongVideoBench uses the official
`longvideobench/LongVideoBench-Meta` public validation metadata source while the
canonical gated dataset remains tracked in the benchmark registry.

Latest local PRD proof suite: `prd_proof_suite_6bc3fa744c60a0e6` (15 checks passed).

```powershell
.\.venv\Scripts\ely-eye.exe proof-suite --cartridge-id ely_eye_prd_7e5cd27b8284
.\.venv\Scripts\ely-eye.exe latest-proof-suite
.\.venv\Scripts\ely-eye.exe runtime-generation-proof
.\.venv\Scripts\ely-eye.exe visual-contradiction-proof
.\.venv\Scripts\ely-eye.exe refresh-cartridge ely_eye_prd_7e5cd27b8284
.\.venv\Scripts\ely-eye.exe finalize-cartridge ely_eye_prd_7e5cd27b8284
.\.venv\Scripts\ely-eye.exe benchmark-registry
.\.venv\Scripts\ely-eye.exe benchmark-sources
.\.venv\Scripts\ely-eye.exe benchmark-samples
.\.venv\Scripts\ely-eye.exe benchmark-predictions
.\.venv\Scripts\ely-eye.exe runtime-profiles
```

The same surface is available through `POST /api/proof-suite`, `GET /api/proof-suite/latest`, `POST /api/visual-contradiction-proof`, `POST /api/cartridges/{cartridge_id}/refresh`, `POST /api/cartridges/{cartridge_id}/finalize`, `GET /api/cartridges/{cartridge_id}/assets`, `GET /api/benchmarks/registry`, `GET /api/benchmarks/sources`, `GET /api/benchmarks/samples`, `GET /api/benchmarks/predictions`, `GET /api/runtime/profiles`, and the dashboard PRD Proof Suite panel.

Runtime launchers:
`ely-eye runtime-profiles` validates the WSL GPU, Linux runtime packages,
SGLang/vLLM launcher modules, KTransformers import path, and llama.cpp help
output before the proof suite marks runtime delivery ready.

```powershell
.\scripts\setup-linux-runtimes.ps1
.\scripts\launch-sglang-live.ps1
.\scripts\launch-sglang-extreme.ps1
.\scripts\launch-vllm-live.ps1
.\scripts\launch-vllm-extreme.ps1
.\scripts\launch-ktransformers-extreme.ps1
.\scripts\launch-llamacpp-offline.ps1
```

## Data Layout

Runtime state is stored under `.ely_eye/` by default.

```text
.ely_eye/
├─ adapters/
├─ data/ely_eye.sqlite
├─ training/
├─ objects/
├─ cache/
└─ cartridges/
```

Context Cartridge artifacts are materialized as `manifest.json`, `atoms.parquet`, `sources.json`, `sparse_index.bm25.zst`, `temporal_graph.sqlite`, `text_vectors.fp16.zstd`, `visual_vectors.int8.zstd`, `kv_snapshots.kivi`, `memory_capsule.json.zst`, `memory_capsule_index.json`, `ttt_vl_adapter.safetensors`, `visual_mtp_head.safetensors`, `eval_proofs/`, and attached adapter folders.