ely-eye

72 Commits 1 Branch 0 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
ZacharyZhang-NY	9145a23f18	Require a token for writes and quarantine what nobody vouches for Sixteen mutating routes now refuse a caller who cannot present the token the server started with; queries stay open, because a question is not a write. The environment variable is ELY_EYE_AUTH_TOKEN, following the repo's env_prefix convention — the shorter name I reached for first returned 403 where the smoke expected 401, and the server printed the reason. Words from a stranger are quarantined on the way in: an HTTP upload is third_party by definition, and a sentence carrying an instruction-shaped phrase is quarantined whoever wrote it. A quarantined atom stays findable and stays out of the compiled context until a second, live, unquarantined source agrees with its rarest words. Corroboration is computed at read time because it is a property of the corpus now, not of the moment a memory was written; it is term-level, and the docstring says so. forget now erases the source row and the compiled contexts beside the atoms, so a forgotten memory leaves nothing that can still speak. The semantic cache key carries a store revision, which fixes a real staleness: a context compiled before new evidence arrived kept its old quarantine decision forever. bench.audit --product still pointed at evals.py, which the p1-6 split moved into backend/src/ely_eye/proofs/. It failed fast, but the anti-self-justification rule had quietly stopped auditing the product; it now walks the package, and a test pins that the package exists and holds no re-roll. Gates: pytest 306 passed; ruff clean; mypy cold cache 24 errors in 9 files, the recorded baseline, net zero; pyright 0 errors on the 23 backend files and the 2 bench files; go build/vet/test and gofmt clean; npm run build 221.39 kB with eslint exit 0; bench/filesize_check clean; bench.audit --product clean over 31 harness files; scripts/mcp_write_smoke.sh PASS with the gate live across two servers and the MCP child.	2026-10-08 23:05:15 -04:00
ZacharyZhang-NY	cd9851a12c	Keep the vectors beside the atoms instead of rebuilding them per query The dense leg used to be a pair of files next to the database: a full .npy re-encoded from every atom, re-normalized on every query, and stale the moment anything was ingested. It is now a vec0 table inside the same SQLite file, written in the same transaction as the atom it describes, so a memory is queryable the instant it is remembered. - vector_index.py maintains the table; the writer inserts, the reader never rebuilds. A repair scan (ely-eye build-dense-index) catches what a crash hid, and it is not on the search path. - Ranking stays in the engine; every boundary (closed, retired, not yet true, another holder's private memory, the mounted cartridge) is checked once against the atoms table afterwards. A visibility subquery inside the KNN cost 48.9s on 153,857 vectors; the same check over 200 candidates costs 0.2ms. When the boundaries reject enough candidates the ask widens once, never twice. - The Go mirror and the web panel carry the new status fields. - bench/dense_scale.py and bench/quant_probe.py measure the leg at corpus size; bench/scale_check.py now works on a copy so the frozen baseline stays frozen. Measured on the frozen 165,973-atom corpus. bench/reports/DENSE.md, QUANT.md and SCALE.md carry the numbers and the machine they were taken on. Gates: 285 pytest, ruff clean, pyright 0 on the touched files, go build/vet/test and gofmt clean, frontend build and eslint clean. mypy is 25 findings against the parent's 24: the new one is "Skipping analyzing sqlite_vec: missing library stubs", the same class as the two existing qwen_vl_utils lines, and the repo keeps no type: ignore anywhere.	2026-10-08 14:05:27 -04:00
ZacharyZhang-NY	9fe13fffb3	Make a cartridge a scope you mount, not only a bundle you export A cartridge used to be an export: materialize it, hash it, and never let it change behaviour again. The transactive-memory promise needs the opposite — mounting the home cartridge must make the work cartridge's memories unfindable, and appending to a cartridge must make a new memory findable through it. Cartridge scope (schema v2 -> v3): - cartridge_atoms (which atoms are in which cartridge) and cartridge_mounts (which cartridges this machine has mounted now); both rebuilt on reopen, because a mount is a machine-local state, not a fact about the data. - mount / switch / unmount / append refuse a cartridge that was built for a different runtime, naming every differing field instead of a boolean. - append rewrites atoms.parquet and sparse_index.bm25.zst, recomputes DNA and the KV fingerprint, and keeps the cartridges row in one SQLite transaction; the files are written first so a failed export cannot leave a half-cartridge in the database. - RetrievalService.search() now ranks inside the mounted scope by default, and an explicit cartridge_ids filter overrides the mount. Nothing mounted means the whole library, so an empty store behaves as it did before. - forget() removes the forgotten atoms from every cartridge's membership: forgetting must not leave a memory retrievable through a cartridge. KV fingerprint (kv_fingerprint.py, stored in the manifest): - model / tokenizer / chat-template hashes, the RoPE block (factor, original_max_position_embeddings, mrope_interleaved, mrope_section, partial_rotary_factor), the adapter hash, the context length, the engine version, and the fingerprint format version. - A snapshot we cannot read is recorded as unverifiable and listed in unverifiable_fields; a blind machine refuses to mount rather than pretending the runtime matches, because a wrong RoPE factor silently corrupts retrieval. Two defects found by running the new tests, not by reading: - atoms.parquet could not export a memory at all: pyarrow refuses a struct with no child field, so metadata={} failed the whole table. The six JSON columns are now written as JSON text and decoded by the cartridge reader. - materialize() inserted cartridge membership before the cartridges row existed, so the foreign key rejected every cartridge. api.py crossed 500 lines, so the cartridge and mount routes moved to api_cartridges.py. That module must not use `from __future__ import annotations`: FastAPI then resolves Depends(settings_dep) against module globals, where the injected dependency is only a parameter of register(), and every route starts demanding settings as a query parameter. Verification: 138 pytest passed (26 new: 10 fingerprint, 16 scope), Go build/vet/gofmt/test clean, and scripts/mcp_write_smoke.sh now proves the whole loop across two processes - mount narrows search, a memory written after the upload is invisible until append_to_cartridge, unmount restores the library, and rewriting the RoPE factor makes mount_cartridge refuse with "rope: factor: cartridge 4.0 vs local 2.0". ruff clean on every touched file (the 20 remaining package errors are in files this change did not touch; HEAD had 25). mypy: one error, the pre-existing np.savez stub artifact at cartridge_assets.py:329 (HEAD:320). Repo pyright: 62 errors, unchanged from the p0-3 ledger.	2026-10-07 16:40:27 -04:00

Author

SHA1

Message

Date

ZacharyZhang-NY

9145a23f18

Require a token for writes and quarantine what nobody vouches for

Sixteen mutating routes now refuse a caller who cannot present the token the
server started with; queries stay open, because a question is not a write. The
environment variable is ELY_EYE_AUTH_TOKEN, following the repo's env_prefix
convention — the shorter name I reached for first returned 403 where the smoke
expected 401, and the server printed the reason.

Words from a stranger are quarantined on the way in: an HTTP upload is
third_party by definition, and a sentence carrying an instruction-shaped phrase
is quarantined whoever wrote it. A quarantined atom stays findable and stays out
of the compiled context until a second, live, unquarantined source agrees with
its rarest words. Corroboration is computed at read time because it is a
property of the corpus now, not of the moment a memory was written; it is
term-level, and the docstring says so.

forget now erases the source row and the compiled contexts beside the atoms, so
a forgotten memory leaves nothing that can still speak. The semantic cache key
carries a store revision, which fixes a real staleness: a context compiled
before new evidence arrived kept its old quarantine decision forever.

bench.audit --product still pointed at evals.py, which the p1-6 split moved into
backend/src/ely_eye/proofs/. It failed fast, but the anti-self-justification
rule had quietly stopped auditing the product; it now walks the package, and a
test pins that the package exists and holds no re-roll.

Gates: pytest 306 passed; ruff clean; mypy cold cache 24 errors in 9 files, the
recorded baseline, net zero; pyright 0 errors on the 23 backend files and the 2
bench files; go build/vet/test and gofmt clean; npm run build 221.39 kB with
eslint exit 0; bench/filesize_check clean; bench.audit --product clean over 31
harness files; scripts/mcp_write_smoke.sh PASS with the gate live across two
servers and the MCP child.

2026-10-08 23:05:15 -04:00

ZacharyZhang-NY

cd9851a12c

Keep the vectors beside the atoms instead of rebuilding them per query

The dense leg used to be a pair of files next to the database: a full .npy
re-encoded from every atom, re-normalized on every query, and stale the moment
anything was ingested. It is now a vec0 table inside the same SQLite file,
written in the same transaction as the atom it describes, so a memory is
queryable the instant it is remembered.

- vector_index.py maintains the table; the writer inserts, the reader never
  rebuilds. A repair scan (ely-eye build-dense-index) catches what a crash hid,
  and it is not on the search path.
- Ranking stays in the engine; every boundary (closed, retired, not yet true,
  another holder's private memory, the mounted cartridge) is checked once against
  the atoms table afterwards. A visibility subquery inside the KNN cost 48.9s on
  153,857 vectors; the same check over 200 candidates costs 0.2ms. When the
  boundaries reject enough candidates the ask widens once, never twice.
- The Go mirror and the web panel carry the new status fields.
- bench/dense_scale.py and bench/quant_probe.py measure the leg at corpus size;
  bench/scale_check.py now works on a copy so the frozen baseline stays frozen.

Measured on the frozen 165,973-atom corpus. bench/reports/DENSE.md, QUANT.md and
SCALE.md carry the numbers and the machine they were taken on.

Gates: 285 pytest, ruff clean, pyright 0 on the touched files, go build/vet/test
and gofmt clean, frontend build and eslint clean. mypy is 25 findings against the
parent's 24: the new one is "Skipping analyzing sqlite_vec: missing library stubs",
the same class as the two existing qwen_vl_utils lines, and the repo keeps no
type: ignore anywhere.

2026-10-08 14:05:27 -04:00

ZacharyZhang-NY

9fe13fffb3

Make a cartridge a scope you mount, not only a bundle you export

A cartridge used to be an export: materialize it, hash it, and never let it
change behaviour again. The transactive-memory promise needs the opposite —
mounting the home cartridge must make the work cartridge's memories
unfindable, and appending to a cartridge must make a new memory findable
through it.

Cartridge scope (schema v2 -> v3):
- cartridge_atoms (which atoms are in which cartridge) and cartridge_mounts
  (which cartridges this machine has mounted now); both rebuilt on reopen,
  because a mount is a machine-local state, not a fact about the data.
- mount / switch / unmount / append refuse a cartridge that was built for a
  different runtime, naming every differing field instead of a boolean.
- append rewrites atoms.parquet and sparse_index.bm25.zst, recomputes DNA and
  the KV fingerprint, and keeps the cartridges row in one SQLite transaction;
  the files are written first so a failed export cannot leave a half-cartridge
  in the database.
- RetrievalService.search() now ranks inside the mounted scope by default, and
  an explicit cartridge_ids filter overrides the mount. Nothing mounted means
  the whole library, so an empty store behaves as it did before.
- forget() removes the forgotten atoms from every cartridge's membership:
  forgetting must not leave a memory retrievable through a cartridge.

KV fingerprint (kv_fingerprint.py, stored in the manifest):
- model / tokenizer / chat-template hashes, the RoPE block (factor,
  original_max_position_embeddings, mrope_interleaved, mrope_section,
  partial_rotary_factor), the adapter hash, the context length, the engine
  version, and the fingerprint format version.
- A snapshot we cannot read is recorded as unverifiable and listed in
  unverifiable_fields; a blind machine refuses to mount rather than pretending
  the runtime matches, because a wrong RoPE factor silently corrupts retrieval.

Two defects found by running the new tests, not by reading:
- atoms.parquet could not export a memory at all: pyarrow refuses a struct with
  no child field, so metadata={} failed the whole table. The six JSON columns
  are now written as JSON text and decoded by the cartridge reader.
- materialize() inserted cartridge membership before the cartridges row existed,
  so the foreign key rejected every cartridge.

api.py crossed 500 lines, so the cartridge and mount routes moved to
api_cartridges.py. That module must not use `from __future__ import
annotations`: FastAPI then resolves Depends(settings_dep) against module
globals, where the injected dependency is only a parameter of register(), and
every route starts demanding settings as a query parameter.

Verification: 138 pytest passed (26 new: 10 fingerprint, 16 scope), Go
build/vet/gofmt/test clean, and scripts/mcp_write_smoke.sh now proves the whole
loop across two processes - mount narrows search, a memory written after the
upload is invisible until append_to_cartridge, unmount restores the library,
and rewriting the RoPE factor makes mount_cartridge refuse with "rope: factor:
cartridge 4.0 vs local 2.0". ruff clean on every touched file (the 20 remaining
package errors are in files this change did not touch; HEAD had 25). mypy: one
error, the pre-existing np.savez stub artifact at cartridge_assets.py:329
(HEAD:320). Repo pyright: 62 errors, unchanged from the p0-3 ledger.

2026-10-07 16:40:27 -04:00