Keep the vectors beside the atoms instead of rebuilding them per query

The dense leg used to be a pair of files next to the database: a full .npy
re-encoded from every atom, re-normalized on every query, and stale the moment
anything was ingested. It is now a vec0 table inside the same SQLite file,
written in the same transaction as the atom it describes, so a memory is
queryable the instant it is remembered.

- vector_index.py maintains the table; the writer inserts, the reader never
  rebuilds. A repair scan (ely-eye build-dense-index) catches what a crash hid,
  and it is not on the search path.
- Ranking stays in the engine; every boundary (closed, retired, not yet true,
  another holder's private memory, the mounted cartridge) is checked once against
  the atoms table afterwards. A visibility subquery inside the KNN cost 48.9s on
  153,857 vectors; the same check over 200 candidates costs 0.2ms. When the
  boundaries reject enough candidates the ask widens once, never twice.
- The Go mirror and the web panel carry the new status fields.
- bench/dense_scale.py and bench/quant_probe.py measure the leg at corpus size;
  bench/scale_check.py now works on a copy so the frozen baseline stays frozen.

Measured on the frozen 165,973-atom corpus. bench/reports/DENSE.md, QUANT.md and
SCALE.md carry the numbers and the machine they were taken on.

Gates: 285 pytest, ruff clean, pyright 0 on the touched files, go build/vet/test
and gofmt clean, frontend build and eslint clean. mypy is 25 findings against the
parent's 24: the new one is "Skipping analyzing sqlite_vec: missing library stubs",
the same class as the two existing qwen_vl_utils lines, and the repo keeps no
type: ignore anywhere.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-08 14:05:27 -04:00
1 parent be08e7bf9c
commit cd9851a12c
26 files changed
+1602 -118

No files matched your search

+26
View File
@@ -5,7 +5,9 @@ from __future__ import annotations
import hashlib
import json
import sqlite3
from collections.abc import Callable
from pathlib import Path
from typing import Any
import pytest
from typer.testing import CliRunner
@@ -20,6 +22,7 @@ from ely_eye.memory import MemoryService
from ely_eye.migrations import SCHEMA_VERSION
from ely_eye.schemas import EvidenceAtom, MemoryOrigin, Modality, SourceRecord, TrustScores, utc_now
from ely_eye.storage import ObjectStore
from ely_eye.vector_index import VectorIndex
KEYS = "the keys are on the shelf by the door"
PASSPORT = "the passport is in the blue box"
@@ -85,6 +88,29 @@ def test_a_restore_brings_the_memories_back(tmp_path: Path) -> None:
assert {atom.text for atom in database.list_atoms()} == {KEYS, PASSPORT}
def test_a_backup_carries_the_vector_index_and_a_restore_answers_from_it(
tmp_path: Path, fake_embeddings: Callable[[], Any]
) -> None:
source = Settings(home=tmp_path / "home", default_holder="wife", dense_embeddings_enabled=True)
database = Database(source)
database.init()
index = VectorIndex(source, database, fake_embeddings())
memory = MemoryService(source, database, vectors=index)
for statement in (KEYS, PASSPORT):
memory.remember(statement, holder_id="wife", origin=MemoryOrigin.self_)
assert index.stats()["vectors"] == 2
maintenance.backup(source, tmp_path / "backup")
destination = Settings(home=tmp_path / "elsewhere", default_holder="wife", dense_embeddings_enabled=True)
maintenance.restore(destination, tmp_path / "backup")
restored = VectorIndex(destination, Database(destination), fake_embeddings())
assert restored.stats()["vectors"] == 2, "the vectors are data, not a cache to redo"
hits = restored.search(restored.embeddings.embed_query(KEYS), 10)
best = next(iter(hits))
assert Database(destination).get_atoms([best])[0].text == KEYS
def test_a_restore_keeps_the_store_it_replaces(tmp_path: Path) -> None:
settings = _settings(tmp_path)
_store(settings)