The dense leg used to be a pair of files next to the database: a full .npy
re-encoded from every atom, re-normalized on every query, and stale the moment
anything was ingested. It is now a vec0 table inside the same SQLite file,
written in the same transaction as the atom it describes, so a memory is
queryable the instant it is remembered.
- vector_index.py maintains the table; the writer inserts, the reader never
rebuilds. A repair scan (ely-eye build-dense-index) catches what a crash hid,
and it is not on the search path.
- Ranking stays in the engine; every boundary (closed, retired, not yet true,
another holder's private memory, the mounted cartridge) is checked once against
the atoms table afterwards. A visibility subquery inside the KNN cost 48.9s on
153,857 vectors; the same check over 200 candidates costs 0.2ms. When the
boundaries reject enough candidates the ask widens once, never twice.
- The Go mirror and the web panel carry the new status fields.
- bench/dense_scale.py and bench/quant_probe.py measure the leg at corpus size;
bench/scale_check.py now works on a copy so the frozen baseline stays frozen.
Measured on the frozen 165,973-atom corpus. bench/reports/DENSE.md, QUANT.md and
SCALE.md carry the numbers and the machine they were taken on.
Gates: 285 pytest, ruff clean, pyright 0 on the touched files, go build/vet/test
and gofmt clean, frontend build and eslint clean. mypy is 25 findings against the
parent's 24: the new one is "Skipping analyzing sqlite_vec: missing library stubs",
the same class as the two existing qwen_vl_utils lines, and the repo keeps no
type: ignore anywhere.
The store had a write path, a mount path and a search path, and the console
still showed a dashboard of counters. Nothing in the UI could ask what someone
knows, page through atoms, look at a wordless picture, forget one memory, or
mount a cartridge — so the mechanisms built in P0 were invisible.
Backend:
- senses.py: one function decides what this machine can perceive. /api/status
and doctor each computed it before, which is two truths. Picture atoms are now
counted by image_ref rather than by modality, so a screenshot crop from a PDF
counts and a picture whose bytes were deleted stops claiming it can be encoded.
- browse.py: GET /api/atoms pages newest-first through v_atoms_current with
modality/holder/source filters; GET /api/atoms/{id}/image serves the stored
bytes so a picture with no OCR text can be looked at.
- Deleted 8 HTTP routes nobody calls (4 benchmark, research modules, dense index,
visual-contradiction-proof, memory-map). Their functions stay: the CLI and the
proof checks use them. The surface is 30 routes, pinned as a set by a test so a
new unused route fails the suite.
Frontend: App.tsx (517 lines) became a 131-line shell plus seven panels that own
their own fetches, and lib/format.ts. types.ts now carries the memory dimensions
the backend has been sending since schema v2.
Two invented fields were rendering lies: AdapterPanel showed gpu_available and
probe_status, which AdapterLoadReport does not have — every machine looked GPU-less
including this one. The panel now shows only what the server says.
Proof: scripts/ui_smoke.sh starts the API on a fresh temp home, serves the built
UI from the same origin, and drives five interactions in a real browser — search
as somebody in a scenario, page and view a wordless picture, forget, mount, read
adapter states with reasons. It prints the mount refusal verbatim rather than
pretending the mount succeeded.
Gates: 263 pytest passed; ruff and mypy clean on the 8 touched backend files
(repo-wide mypy unchanged at 24 errors / 6 files); pyright 0 errors there;
go build/vet/test/gofmt clean; npm build + eslint clean; ui_smoke and
mcp_write_smoke both PASS.
CI was a workflow that could not run its own tests: pytest is not a
dependency of anything, so a fresh clone fails with "No module named
pytest". Test tooling now lives in the backend's dev extra, so CI and a
new contributor install the same tools, and ruff is pinned there instead
of drifting as a bare version in a workflow step.
backup, restore and doctor move the store. Backup uses SQLite's own
online API so a running server is safe, and the manifest records the
schema version, row counts, sha256 and size read back from the copy.
Restore verifies the digest, integrity_check, required tables and schema
age before it touches the live file, and keeps what it replaces as
.before-restore.
Doctor separates broken from degraded. On the 906 MB baseline store
(165,973 atoms) it reports no problems and one true warning: 21,362
picture atoms with no CLIP snapshot configured, so those atoms are
findable only by the text already on them. The first version called that
store healthy because it only complained about paths that were set and
missing, never about a sense the store needs and never had.
AGPL-3.0-only, because PyMuPDF is dual-licensed AGPL-or-commercial and
the repository is public. A permissive license needs the Artifex
commercial license bought first; the README says so rather than hiding it.
Gates: 253 tests pass; ruff clean repo-wide (17 errors at HEAD); mypy
unchanged from HEAD on touched files; pyright 0 errors on the new files;
backup 4.0s and restore 2.8s on real data with counts verified equal.
ZacharyZhang-NY