Stop paying for words every document carries
The product's own search answered in p50 20.16s on 165,973 atoms. The SQL rewrite brought that to 0.68s; this brings it to 57ms and says why the rest was there. The cost was not SQL. An OR MATCH scores every row it touches, and a question carries words the whole library carries: `"parsing"* OR "error"* OR "in"* OR "the"* OR "pdf"* OR "converter"*` matched 127,145 atoms and took 305ms, while a CJK bigram phrase query with 32 matches took 1.0ms. Latency tracked match count. - text_index.rare_terms asks the index which of its own words are noise: count each term, drop the ones over 5% of the library, keep the rarest few when all of them are common. No stop-word list; a corpus of PDFs and a corpus of shell history get different answers. 0.05 is swept, not chosen — on n=173 recall@10 peaks there (0.899 / p50 58.6ms) against 0.893 / 112.9ms at 0.20 and 0.844 / 13.4ms at 0.01. - The bound is a cost, not a claim about rarity: `cap = max(int(total*fraction)+1, 2)`, so a word matching one atom in eleven is kept — scoring one candidate is free. A capped count cannot rank words that all saturate, so those few are counted exactly. - Schema v6 adds `term_df`: a word is counted once and the next question pays nothing. Every atom write clears it, so a stored count cannot outlive the library it described. Cold p50 73ms, warm 63ms, 1,324 terms cached. - The denominator is `count(*) FROM atoms` (0.01ms) rather than `count(*) FROM atom_fts` (37ms cold): FTS rows are written and deleted with their atoms, so the two counts move together and only one of them is free. SCALE.md, idle machine (load 1.76, 32 cores, 4 threads per query): 165,973 atoms — 1.66x the contract's 100k — 300 probes, p50 57ms, p95 138ms. Against the frozen baseline of 20.16s that is 354x. Holder-scoped (n=173): recall@10 0.870, MRR 0.740, p50 62ms. Verification: pytest backend/tests bench/tests -q -> 173 passed (5 new text_index tests: the common word is dropped and the rare one still answers, an all-common query keeps its rarest words in order, a one-word query is never filtered, an empty library keeps everything, and a write voids the counts); ruff clean; mypy fresh cache clean; pyright 0 errors on the touched files.
This commit is contained in:
1 parent
6de7439db6
commit
7e1fe649f6
7 files changed
+236
-13
No files matched your search
@@ -12,7 +12,7 @@ from .config import Settings, get_settings
|
||||
from .migrations import apply as apply_migrations
|
||||
from .row_codec import atom_params, from_json, row_to_atom, row_to_source, to_json
|
||||
from .schemas import EvidenceAtom, SourceRecord, utc_now
|
||||
from .text_index import fts_query, index_text
|
||||
from .text_index import fts_expression, fts_terms, index_text, rare_terms
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -122,6 +122,8 @@ class Database:
|
||||
"INSERT INTO atom_fts(atom_id, source, text) VALUES (?, ?, ?)",
|
||||
[(atom.atom_id, index_text(atom.source), index_text(atom.text)) for atom in atoms],
|
||||
)
|
||||
# New text changes which words are common, so every remembered count is void.
|
||||
connection.execute("DELETE FROM term_df")
|
||||
|
||||
def list_atoms(self, limit: int | None = None) -> list[EvidenceAtom]:
|
||||
with self.connect() as connection:
|
||||
@@ -202,6 +204,7 @@ class Database:
|
||||
asker the query runs over the whole library, which is what an operator debugging wants.
|
||||
"""
|
||||
with self.connect() as connection:
|
||||
expression = fts_expression(rare_terms(connection, fts_terms(query)))
|
||||
rows = connection.execute(
|
||||
"""
|
||||
SELECT f.atom_id, bm25(atom_fts) AS score
|
||||
@@ -238,7 +241,7 @@ class Database:
|
||||
LIMIT ? OFFSET ?
|
||||
""",
|
||||
(
|
||||
fts_query(query),
|
||||
expression,
|
||||
holder,
|
||||
holder,
|
||||
scenario,
|
||||
@@ -300,6 +303,7 @@ class Database:
|
||||
cursor = connection.execute(
|
||||
"DELETE FROM atoms WHERE atom_id IN (SELECT value FROM json_each(?))", (to_json(atom_ids),)
|
||||
)
|
||||
connection.execute("DELETE FROM term_df")
|
||||
return int(cursor.rowcount)
|
||||
|
||||
def delete_edges_touching(self, atom_ids: list[str]) -> int:
|
||||
|
||||
Reference in new issue
Block a user