The README described a project that no longer exists: a Windows-only Qwen
runtime with nine read-only tools, a console as the only way to operate the
store, and a 100M-token capsule as if it were measured. It is now a description
of what is here, with the report and the command that re-measures each number,
and a section that names what has not been run instead of leaving the reader to
discover it.
TRAINING_ARTIFACTS.md keeps its artifact tables and loses the scores that were
never run: MMMU-Pro and OCRBench each read one sample, and the HashHop gate
collected proofs until it hit its target. What stays is annotated with what it
actually measures — citation accuracy there means the cited id existed in the
context that was handed over, not that the passage supports the answer.
Gaps.md gains a re-audit section: ten of the twelve gaps closed, two partial,
each row naming the command or report behind it, and the remaining work stated
plainly — the 0.90 recall bar is unmet and the deficit is entirely in the
visual and audio legs.
AGENTS.md stops being ignored. It records the constraints this repository paid
for: the measurement rules, the environment traps, and the review protocol, so
the next agent does not pay them again.
The CLI could filter by holder but could not look through a holder's eyes, so
search gains --as-holder: three states of eyes on every face, not two.
- bench/ingest_baseline.py drives the unmodified product IngestionService over
all 50,789 corpus files in sharded SQLite DBs, then merges them. Thread and
memory ceilings are set explicitly: ONNX Runtime defaulted to every core
(95 threads/worker, load 490) and its CPU arena kept freed pages mapped,
which OOM-killed workers.
- bench/metrics.py: recall@k, MRR, nDCG@10, Wilson lower bound, chance
baseline. No model, no product import, so the scorer cannot grade itself.
- bench/systems.py + bench/run.py: the five ablation arms over the merged DB,
latency percentiles, scale curve, and a top-10 equivalence check against the
live RetrievalService.
- bench/counterpart.py + bench/run_external.py: in-repo Mem0-style flat store
and Zep-style temporal store as the external reference point.
- bench/audit.py: enforces single-attempt scoring, frozen n, a chance row
beside every score, and producer != verifier. It already flags the product's
retry-until-target loop at backend/src/ely_eye/evals.py:2351.
- backend/pyproject.toml: rapidocr-onnxruntime>=1.4.4 was unsatisfiable
(PyPI maxes at 1.2.3); the successor is the renamed rapidocr package.
Ground truth comes only from third-party annotations (MMLongBench-Doc
evidence_page_ids, VisRAG qrels, ScreenSpot-v2 bboxes, Video-MME QA,
LongMemEval answer_session_ids, CodeSearchNet func_path+func_name) and
from document structure; no model judgement is used anywhere.
- bench/corpus/manifest.jsonl: 28 sources with url, retrieved_at, sha256,
annotation_source, license, files[]; verified by bench.manifest_check
- bench/sets/{pdf,image,ui_screenshot,video,code,conversation}.{jsonl} plus
corpus indexes and a manifest that freezes every file's sha256
- 2,928 answerable + 2,380 unanswerable queries over 854 holders; every
query carries scope {holder, scenario, as_of} for transactive routing
- bench/validate_sets.py fails on missing gold, holder mismatch between
gold and query, gold observed after as_of, and manifest digest drift
Real-models implementation: multimodal ingestion, Context Cartridge memory,
sparse + Qwen embedding retrieval, Cache Fabric, Qwen runtime adapters, verifier
contracts, Visual HashHop proofs, BF16 LoRA fine-tuning, and a local dashboard.
Verified by a 15-check PRD proof suite on the live model.
ZacharyZhang-NY