Keep the vectors beside the atoms instead of rebuilding them per query

The dense leg used to be a pair of files next to the database: a full .npy
re-encoded from every atom, re-normalized on every query, and stale the moment
anything was ingested. It is now a vec0 table inside the same SQLite file,
written in the same transaction as the atom it describes, so a memory is
queryable the instant it is remembered.

- vector_index.py maintains the table; the writer inserts, the reader never
  rebuilds. A repair scan (ely-eye build-dense-index) catches what a crash hid,
  and it is not on the search path.
- Ranking stays in the engine; every boundary (closed, retired, not yet true,
  another holder's private memory, the mounted cartridge) is checked once against
  the atoms table afterwards. A visibility subquery inside the KNN cost 48.9s on
  153,857 vectors; the same check over 200 candidates costs 0.2ms. When the
  boundaries reject enough candidates the ask widens once, never twice.
- The Go mirror and the web panel carry the new status fields.
- bench/dense_scale.py and bench/quant_probe.py measure the leg at corpus size;
  bench/scale_check.py now works on a copy so the frozen baseline stays frozen.

Measured on the frozen 165,973-atom corpus. bench/reports/DENSE.md, QUANT.md and
SCALE.md carry the numbers and the machine they were taken on.

Gates: 285 pytest, ruff clean, pyright 0 on the touched files, go build/vet/test
and gofmt clean, frontend build and eslint clean. mypy is 25 findings against the
parent's 24: the new one is "Skipping analyzing sqlite_vec: missing library stubs",
the same class as the two existing qwen_vl_utils lines, and the repo keeps no
type: ignore anywhere.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-08 14:05:27 -04:00
1 parent be08e7bf9c
commit cd9851a12c
26 files changed
+1602 -118

No files matched your search

+41
View File
@@ -2,11 +2,13 @@
from __future__ import annotations
import hashlib
import json
from collections.abc import Callable
from pathlib import Path
from typing import Any
import numpy as np
import pytest
ROPE: dict[str, Any] = {
@@ -51,3 +53,42 @@ def write_model_fixture() -> Callable[..., Path]:
@pytest.fixture
def model_dir(tmp_path: Path) -> Path:
return write_model(tmp_path / "model")
class HashEmbeddings:
"""A stand-in for the embedding model: the same words give the same vector, no download.
Different words land in different directions, which is all a retrieval test needs; the
numbers mean nothing to a human, so no test should assert on a score's value.
"""
def __init__(self, dim: int = 16) -> None:
self.dim = dim
def enabled(self) -> bool:
return True
def encode_texts(self, texts: list[str]) -> np.ndarray:
if not texts:
raise ValueError("encode_texts was asked to encode nothing")
rows = [
np.frombuffer(hashlib.sha256(text.encode("utf-8")).digest()[: self.dim], dtype=np.int8).astype(np.float32)
for text in texts
]
matrix = np.stack(rows)
norms = np.linalg.norm(matrix, axis=1, keepdims=True)
return matrix / np.where(norms == 0, 1.0, norms)
def encode_atoms(self, atoms: list[Any]) -> np.ndarray:
return self.encode_texts([atom.text or "" for atom in atoms])
def embed_query(self, query: str) -> np.ndarray:
return self.encode_texts([query])[0]
def probe_dim(self) -> int:
return self.dim
@pytest.fixture
def fake_embeddings() -> Callable[[], HashEmbeddings]:
return HashEmbeddings