0 Fork 0

Files

T

Say what the store actually does, and put a receipt under each claim

The README described a project that no longer exists: a Windows-only Qwen
runtime with nine read-only tools, a console as the only way to operate the
store, and a 100M-token capsule as if it were measured. It is now a description
of what is here, with the report and the command that re-measures each number,
and a section that names what has not been run instead of leaving the reader to
discover it.

TRAINING_ARTIFACTS.md keeps its artifact tables and loses the scores that were
never run: MMMU-Pro and OCRBench each read one sample, and the HashHop gate
collected proofs until it hit its target. What stays is annotated with what it
actually measures — citation accuracy there means the cited id existed in the
context that was handed over, not that the passage supports the answer.

Gaps.md gains a re-audit section: ten of the twelve gaps closed, two partial,
each row naming the command or report behind it, and the remaining work stated
plainly — the 0.90 recall bar is unmet and the deficit is entirely in the
visual and audio legs.

AGENTS.md stops being ignored. It records the constraints this repository paid
for: the measurement rules, the environment traps, and the review protocol, so
the next agent does not pay them again.

The CLI could filter by holder but could not look through a holder's eyes, so
search gains --as-holder: three states of eyes on every face, not two.

2026-10-09 16:11:53 -04:00

5.4 KiB

Raw Permalink Blame History

Agent notes for this repository

Read CHECKLIST.md for the ordered work list and Gaps.md for the audit the work started from. Measured results live in bench/reports/. If a sentence in this repository states a capability, there must be a command that demonstrates it — otherwise the sentence belongs in the "not measured" section.

Verify before you claim

Run the gates, and compare against the recorded baseline rather than "it built":

pytest -q                                       # both launch styles must work
ruff check backend/src/ely_eye backend/tests bench
.venv/bin/python -m bench.filesize_check         # fails on any tracked file > 500 lines
.venv/bin/python -m bench.audit --product        # the harness must not justify the product
.venv/bin/python -m mypy backend/src/ely_eye backend/tests   # cold cache; baseline is 24 errors in 9 files
./node_modules/.bin/pyright <the files you touched>          # must be 0 errors on touched files
cd ely-eye-mcp && go build ./... && go vet ./... && go test ./... && gofmt -l .
cd frontend && npm run build && npx eslint src --max-warnings 0
bash scripts/mcp_write_smoke.sh                  # two processes, one store

mypy carries a pre-existing error baseline. A change is clean when the count and the file set match the recorded baseline; measure the baseline in a worktree of the commit you branched from, do not assume it.

Rules this repo enforces

One writer per fact. Memory writes go through MemoryService; the Go MCP server forwards over HTTP instead of touching SQLite, so the rules cannot drift into two dialects.
Fail fast and name the thing. nothing to forget: no atom or source was given, database schema is version 99, this build understands 11. A silent default that hides a caller's mistake is a defect, not a convenience.
A file over 500 lines gets split before it gets commented.
No dead code, no v1/v2/v3 shadows of a thing that no longer exists.
A # comment is one line. A docstring is a contract and may be longer.
Absence is a state. v_atoms_current hides a closed, lapsed, or not-yet-valid memory; it does not delete it, and atom_count() still counts it.

Measurement rules

A number that moves with machine load is quoted as a range across runs, or it points at the report that holds it. Never quote a single load-dependent value as if it were stable.
A report's commit stamp must name the commit that carries the harness that produced it. That is why a harness change and its report land as two commits: a report cannot stamp its own commit.
Every recall number sits next to its chance floor and, where one exists, an external counterpart. bench/reports/BASELINE_EXTERNAL.md is the counterpart; the chance row is printed in the same table.
One sample is not a benchmark. next(iter(stream)) with streaming=True was how the old proof suite "scored" MMMU-Pro and OCRBench.
A probe sample drawn by file order can silently drop a whole modality. bench/query_set.py:scale_probes is modality-stratified for that reason, and a test pins that every modality appears.
A harness works on a copy of the frozen store (bench/scratch.py:copy_store). Never migrate bench/runs/baseline/data/ely_eye.sqlite in place.
When two correct engines disagree, suspect the metric before the engine: byte-identical atoms produce tied scores, so id-level recall can read 0.83 for no real reason. bench/reports/QUANT.md scores a hit as "any returned atom whose true cosine is at least the K-th best".

Environment traps already paid for

pi-lens runs pyright with 0 config files, so its Import could not be resolved findings for pydantic, fastapi, torch, sqlite_vec are an environment artifact. ./node_modules/.bin/pyright reads pyrightconfig.json and is the one that counts.
pi-lens auto-runs a ruff format-style formatter on files you touch. This repository is not ruff-format clean and its gate is ruff check, so revert that drift (git checkout --) before committing.
Run ruff check --fix on explicit paths, never on the package: it will reformat a dozen unrelated files and you will commit them by accident.
bench/tests import bench.*; the root pyproject.toml supplies pythonpath = ["."], which is why bare pytest works.
import fitz prints a deprecation warning to stdout, which corrupts every CLI JSON answer. Import it as import pymupdf as fitz.
SQLite statements must be literals at the call site: the repo's semgrep rule rejects connection.execute(variable, …) even when the variable is a module constant. Where a rule must be copied, a test compares the copies (backend/tests/test_visibility_rule.py).

Review protocol

Every task lands through an independent reviewer running in a herdr pane against the commit, not against a description of it. Findings are numbered D1…Dn; fix the mechanism, not the wording, and re-measure. A checklist item is not [x] before the reviewer returns PASS. Read the reviewer's verdict from its session JSONL (the last assistant text block), not from a pane scroll-back that may show an older round.

Cross-language contract

backend/src/ely_eye/schemas.py is the source of truth. ely-eye-mcp/internal/domain/types.go and frontend/src/types.ts mirror it; when a field changes there, it changes in all three, and the smoke script is what proves the Go path agrees.

5.4 KiB Raw Permalink Blame History