Rank memory search in SQL instead of rebuilding BM25 in Python

The product path pulled every atom into Python and rebuilt a BM25Okapi
index per query: p50 20.16s over 165,973 atoms. It now runs one SQLite
statement with LIMIT/OFFSET.

- text_index.py: one normalizer for both sides of the index. CJK runs are
  split per character (unicode61 cannot segment them), camelCase and
  separators are split so "search atoms" finds SearchAtoms, latin tokens
  get a prefix match. Rows written before this existed hold raw text in
  atom_fts, so schema v3 rebuilds the index on upgrade.
- retrieval.py: filters (holder / scenario / source / modality / since /
  until / min_trust), a base64 cursor, and reciprocal rank fusion in place
  of the 0.55/0.35/0.10 blend whose graph term was always zero.
- An empty result is {status: "no_match"}, not a bare array.
- Go: search_atoms and compile_context rank through POST /api/search. The
  second Go tokenizer is deleted; two implementations could drift, which
  is what the MCP/HTTP parity requirement forbids.
- db.py passed 500 lines, so the row codec moved to row_codec.py.
- Tooling: installed the repo's own pyright. pyrightconfig.json had no
  exclude and swept the downloaded corpus (1171 errors); with excludes the
  count is 62, all pre-existing (evals 48, profiles 4, training 4,
  ingestion 3, runtime 2, cartridge_assets 1). backend/pyproject.toml
  carried a second [tool.pyright] that shadowed the root config; deleted.
- bench/scale_check.py measures the product search at corpus size. The
  100k-atom p50 number is not written until it is measured on an idle
  machine.

Verification: 112 pytest passed; go build/vet/test clean, gofmt clean;
ruff clean on every touched file; mypy clean on the 14 touched Python
files; repo pyright 62 errors, unchanged per-file ledger.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 16:02:40 -04:00
1 parent 3d792417cf
commit 674c88e84e
29 files changed
+1097 -356

No files matched your search

+63
View File
@@ -0,0 +1,63 @@
"""One normalizer for both sides of the FTS index.
FTS5's unicode61 tokenizer splits on whitespace and punctuation only. It reads
「我们的钥匙放在玄关抽屉里」 as one token and `SearchAtoms` as one token, so a search for
钥匙 or atoms finds nothing and reports it as an empty result. Index and query must run the
same transform; keeping them in one file is what stops them from drifting apart.
"""
from __future__ import annotations
import re
from unicodedata import normalize
# Han, its extension A block, the compatibility ideographs, kana, and Hangul syllables:
# scripts written without spaces. One source of truth for both the tokenizer and the query.
_BLOCKS = ((0x3400, 0x4DBF), (0x4E00, 0x9FFF), (0xF900, 0xFAFF), (0x3040, 0x30FF), (0xAC00, 0xD7AF))
_CJK_CLASS = "".join(f"{chr(start)}-{chr(end)}" for start, end in _BLOCKS)
_CJK_CHARS = frozenset("".join("".join(chr(code) for code in range(start, end + 1)) for start, end in _BLOCKS))
_CAMEL_BOUNDARY = re.compile(r"(?<=[a-z0-9])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])|(?<=[A-Za-z])(?=\d)")
# Everything a reader would not type as a word: symbols, punctuation, and the underscore
# that joins identifiers. `[^[^\W\s]` looks equivalent but deletes CJK characters on 3.14.
_SEPARATORS = re.compile(r"[^\w\s]|_")
_CJK_RUN = re.compile(f"[{_CJK_CLASS}]+")
_SPACES = re.compile(r"\s+")
def index_text(text: str) -> str:
"""Rewrite text so every searchable unit is a separate FTS5 token."""
if not text:
return ""
return _SPACES.sub(" ", _CJK_RUN.sub(lambda match: " ".join(match.group()) + " ", _bare(text))).strip()
def fts_query(query: str) -> str:
"""Build the MATCH expression for a user query, using the same split as the index.
A CJK run becomes a quoted phrase so its characters stay adjacent: unphrased, a query for
钥匙 would match text that mentions 钥 and 匙 worlds apart. Latin tokens get a prefix match,
which is what lets `atom` reach `atoms`.
"""
terms = _bare(query).split()
if not terms:
raise ValueError(f"search needs at least one searchable word, got {query!r}")
return " ".join(_phrase(term) if _is_cjk(term) else f"{_quote(term)}*" for term in terms)
def _bare(text: str) -> str:
"""Fold width and case, split identifiers, and drop punctuation, leaving CJK runs whole."""
spaced = _CAMEL_BOUNDARY.sub(" ", normalize("NFKC", text))
return _SPACES.sub(" ", _SEPARATORS.sub(" ", spaced)).strip().lower()
def _is_cjk(term: str) -> bool:
return all(char in _CJK_CHARS for char in term)
def _phrase(run: str) -> str:
return '"' + " ".join(run) + '"'
def _quote(term: str) -> str:
return '"' + term.replace('"', '""') + '"'