ZacharyZhang-NY/ely-eye
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
ely-eye/TRAINING_ARTIFACTS.md
T
ZacharyZhang-NY 89c880fd36 Say what the store actually does, and put a receipt under each claim 
The README described a project that no longer exists: a Windows-only Qwen
runtime with nine read-only tools, a console as the only way to operate the
store, and a 100M-token capsule as if it were measured. It is now a description
of what is here, with the report and the command that re-measures each number,
and a section that names what has not been run instead of leaving the reader to
discover it.

TRAINING_ARTIFACTS.md keeps its artifact tables and loses the scores that were
never run: MMMU-Pro and OCRBench each read one sample, and the HashHop gate
collected proofs until it hit its target. What stays is annotated with what it
actually measures — citation accuracy there means the cited id existed in the
context that was handed over, not that the passage supports the answer.

Gaps.md gains a re-audit section: ten of the twelve gaps closed, two partial,
each row naming the command or report behind it, and the remaining work stated
plainly — the 0.90 recall bar is unmet and the deficit is entirely in the
visual and audio legs.

AGENTS.md stops being ignored. It records the constraints this repository paid
for: the measurement rules, the environment traps, and the review protocol, so
the next agent does not pay them again.

The CLI could filter by holder but could not look through a holder's eyes, so
search gains --as-holder: three states of eyes on every face, not two.
2026-10-09 16:11:53 -04:00

208 lines
11 KiB
Markdown
Raw Permalink Blame History

# Ely-Eye Training Artifacts
## What this file is, and what it is not
It is a record of what was produced on **2026-05-28** on the Windows + RTX 4090
machine that trained the adapters, from a `PRD.md` that no longer exists in this
repository. The adapters themselves, the cartridge, and every `.ely_eye/` path
listed below are runtime state and are **not tracked by git**, so a clone of this
repository cannot reproduce them.
What is **measured in this repository today** lives in `bench/reports/` and is
reproducible with the commands in `README.md`. Nothing in this file belongs to
that set. Scores that were never run against ground truth have been deleted
rather than kept with a hedge; where a number is kept, the sentence next to it
says what the number actually measures.
## Artifacts
Generated on 2026-05-28 from the local `PRD.md`, ingested Evidence Atom
`PRD_0b88b12114634a56:text`, visual Evidence Atom
`ely_eye_dashboard_0180927ffdd3f8b0:image`, local UI proof images, and Qwen
model snapshots. Every adapter has `training_trace.jsonl`, `training_summary.json`,
and `training_proof.json` with RTX 4090 CUDA BF16 execution metadata, proof
version 2, BF16 LoRA method, 8192-token max length, paged 8-bit AdamW,
Unsloth-patched checkpointing mode, per-step loss, trainable parameter hashes
before and after training, optimizer update norms, dataset hashes, environment
versions, adapter weight SHA-256, LoRA tensor statistics, runtime adapter binding
proof, runtime adapter generation proof, and proof-suite verified nonzero LoRA
safetensors.
| Adapter | Base model | Samples | Steps | Updates | Max length | Trainable params | Final loss | Loss delta | Max update L2 | Seconds | SHA-256 |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| HME-Core-LoRA | Qwen/Qwen3.5-9B | 37 | 37 | 10 | 8,192 | 29,097,984 | 0.0001596613001311198 | 0.22773139642958995 | 0.8183563251107077 | 104.4 | 8f97890e802f9adfa3bcb2c02b33d7b39dd8954654d7057d4058e1331e7b5df4 |
| HME-Vision-LoRA | Qwen/Qwen3.5-9B | 10 | 20 | 5 | 8,192 | 1,996,672 | 2.0484652519226074 | -0.11261439323425293 | 0.6595521616294707 | 49.3 | 593f7295f466c5fba07bdae26d1d8798c5affbad8f3863cda9f1881d06453dec |
| HME-TTT-VL | Qwen/Qwen3.5-9B | 10 | 20 | 5 | 8,192 | 8,288,128 | 1.5277884006500244 | 0.4080624580383301 | 1.2924649382409095 | 51.2 | 030b37480155a8eaa6ef4c2dfa819d5f2564838935654f0d826235eb66cc9de5 |
| HME-Visual-MTP | Qwen/Qwen3.5-9B | 10 | 20 | 5 | 8,192 | 2,019,328 | 2.066615343093872 | -0.13076448440551758 | 0.1419292917069751 | 75.8 | 489da9460f51693fa8775ad4975a18f7400dd3fc41eac401d00305617c627717 |
| HME-Router | Qwen/Qwen3.5-0.8B | 8 | 16 | 4 | 8,192 | 3,194,880 | 1.2781919240951538 | 1.0438536405563354 | 0.2871291982199057 | 28.8 | 9ef884a1078df71ba73376a29737a27e22e2ff25854352bfc2c7d960102d886e |
| HME-Retrieval | Qwen/Qwen3-VL-Embedding-8B | 32 | 32 | 32 | 8,192 | 21,823,488 | 0.5665192604064941 | 0.91921067237854 | 1.3181151117874825 | 55.3 | 6490f03e445eb70da7020be992c6eccd3ca2b0e4c48e73a80a819ddac5faa8f3 |
Adapter weight proof:
| Adapter | LoRA tensors | Nonzero elements | Weight SHA-256 |
|---|---:|---:|---|
| HME-Core-LoRA | 256 | 29,097,984 | 4673ac90abdd8ef952dad9c2292aecadba161669c3b1fc3d1af5c075cf0ae62d |
| HME-Vision-LoRA | 220 | 1,992,340 | dde2eb9051866634f9173366a6ffa3fbf9219e5b6d640eafb7f99d0258c52422 |
| HME-TTT-VL | 412 | 8,283,800 | 9a572ac51c6df08e38898c6c30acd8ed6530c7e852611552baaec6a67862e6c5 |
| HME-Visual-MTP | 4 | 1,028,316 | 58d058b862a542c042d27234b5b22344ba02ee6e016be6c8838b329ffea1c6f9 |
| HME-Router | 192 | 3,194,880 | 90079b8f2c244efb00c2285816b98f51ff03cf06a9ba2335ee5bbdf24a1aff8f |
| HME-Retrieval | 504 | 21,823,488 | bcb1381888ae84e82ab18256e152191e56a0892c46cdac81874a9c1fb1af14ee |
Local adapter roots:
```text
.ely_eye/adapters/hme_core_lora_qwen35_9b
.ely_eye/adapters/hme_vision_lora_qwen35_9b
.ely_eye/adapters/hme_ttt_vl_qwen35_9b
.ely_eye/adapters/hme_visual_mtp_qwen35_9b
.ely_eye/adapters/hme_router_qwen35_08b
.ely_eye/adapters/hme_retrieval_qwen3_vl_embedding_8b
```
Cartridge binding:
```text
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/adapters/hme_core_lora
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/adapters/hme_vision_lora
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/adapters/hme_ttt_vl
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/adapters/hme_visual_mtp
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/adapters/hme_router
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/adapters/hme_retrieval
```
PRD proof suite:
```text
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/proof_suite.json
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/proof_suite.json
```
Runtime generation proof:
```text
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/runtime_generation.json
transformers generated 218 tokens with hme_core_lora citation accuracy 1.00
```
"Citation accuracy 1.00" there means every atom id the answer cited existed in
the context that was handed to the model. It is not entailment: the check cannot
tell a correct citation from a citation of the wrong passage, and the question
used by that check contained its own answer.
100M memory capsule proof:
```text
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/memory_capsule.json.zst
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/memory_capsule_index.json
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/memory_capsule.json
The Context Cartridge materialized a 100,000,000 token-equivalent memory capsule with 4,096 committed segments and a verified segment Merkle root.
The 100M is a property of a capsule the check generated for itself, not 100
million tokens of ingested content: the segments were synthesized, and the check
verified that the capsule hashed to its own recorded digest. What the store
actually holds today is measured in `bench/reports/DENSE.md` (165,973 atoms,
153,857 vectors), and `Gaps.md` appendix B prices one million tokens of live KV
cache at 148.9 GB in fp16.
```
Model-solved HashHop proofs:
```text
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/0a10e6394ada0da7/model_text_hashhop.json
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/60c9d5f5f1648ea0/model_visual_hashhop.json
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/a29a310b31f9cef5/model_visual_hashhop.json
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/7f7b8b1f54ec3d75/model_visual_hashhop.json
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/4f24efd13342694c/model_visual_hashhop.json
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/a3f49402ef2710a3/model_visual_hashhop.json
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/hashhop/0bd17526ab51392b/model_visual_hashhop.json
Qwen/Qwen3.5-9B with HME-Core-LoRA solved a 1.01M token-equivalent 2-hop text HashHop and 6 Visual HashHop arenas at 262K token-equivalent.
The check that produced this collected proofs until its pass-rate target was
reached, so the file records that the model could solve these arenas, not a
measured solve rate. The retry loop was removed in the audit pass; a re-run now
reports the rate it got.
```
Benchmark source and sample proof:
```text
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/standard_benchmark_sources.json
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/standard_benchmark_samples.json
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/standard_benchmark_predictions.json
MMMU-Pro, OCRBench, MMLongBench-Doc, VideoMME, LongVideoBench, RULER, and LongBench v2 loaded official streaming samples or official public metadata samples with sample hashes and source dataset SHA values. LongVideoBench uses `longvideobench/LongVideoBench-Meta` validation metadata with canonical `longvideobench/LongVideoBench` registry tracking.
No benchmark score is quoted here. The prediction step read **one** sample per
benchmark (`next(iter(stream))` with `streaming=True`), so MMMU-Pro and OCRBench
produced a single local prediction each; a score computed from one sample is not
a benchmark result and the two `=1.00` lines that used to sit in this file have
been deleted. `bench/reports/` is where this repository
reports measured numbers, on corpora with hundreds of queries.
```
Repository input proof:
```text
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/repository_input.json
The repository input proof indexed 94 files, 49 code files, 11,977 call graph edges, and 26 local git commits for PRD 10.1 code repository coverage.
That is a snapshot of the repository as it stood on 2026-05-28; the repository is
larger now, and the check passed on any git history, which is why it is not
treated as evidence of anything today.
```
Runtime profile readiness:
```text
.ely_eye/data/eval_proofs/prd_proof_suite_b03b88d7cfe5ba96/runtime_profiles.json
SGLang 0.5.12.post1, vLLM 0.21.0, KTransformers 0.6.2.post4, WSL RTX 4090 GPU visibility, launcher module probes, and winget llama-server help output are verified.
Verified inside WSL on that machine, against the fixed distro name the launcher
used at the time. The launch profiles now read `ELY_EYE_WSL_DISTRO`, every
`.ps1` has a `.sh` twin for Linux and macOS, and CI runs the suite on Linux — so
this line is a record of one machine, not a portability claim.
```
Checks:
```text
Adapter Training Proof passed # proof-versioned BF16 LoRA + paged 8-bit AdamW + nonzero LoRA safetensors
Runtime Adapter Binding passed
Runtime Adapter Generation passed
Cartridge Physical Assets passed
100M Memory Capsule passed
Context Cartridge Replay passed
Visual HashHop passed # 7/7 model-solved proofs
Standard Benchmark Registry passed # one local sample per benchmark
Repository Input Coverage passed # 94 files + 11977 call graph edges + local git history
Runtime Profile Delivery passed
Research Module Delivery passed # executable module gates, evidence graph edge, probability sum
Learned Memory Recall passed
Memory DNA passed
Cache Trace passed
```
These are the rows that suite wrote on 2026-05-28, not a current verdict. `Gaps.md`
appendix A judges each of the fifteen checks individually, and several of them
were self-referential — the replay check compiled the same question twice against
the live store, the recall check only proved an adapter file loads, the cache check
passed if any layer had any entry. The suite was rewritten afterwards so a check
that cannot run says what it could not run instead of passing.
Cartridge physical assets:
```text
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/text_vectors.fp16.zstd
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/visual_vectors.int8.zstd # item_count=1
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/kv_snapshots.kivi
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/memory_capsule.json.zst # token_equivalent=100000000, segment_count=4096
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/ttt_vl_adapter.safetensors
.ely_eye/cartridges/ely_eye_prd_7e5cd27b8284/visual_mtp_head.safetensors
```
A cartridge written today is not this list. `cartridge.py` now writes
`manifest.json`, `atoms.parquet`, `sources.json`, `sparse_index.bm25.zst` and
`temporal_graph.sqlite`, and a cartridge is a scope you mount rather than only a
bundle you export. The files above are what that 2026-05-28 export contained;
`kv_snapshots.kivi` and `ttt_vl_adapter.safetensors` were placeholders for a
runtime layout that was never wired to a loader.
Reference in new issue View Git Blame Copy Permalink