Stop paying for words every document carries

The product's own search answered in p50 20.16s on 165,973 atoms. The SQL
rewrite brought that to 0.68s; this brings it to 57ms and says why the rest was
there.

The cost was not SQL. An OR MATCH scores every row it touches, and a question
carries words the whole library carries: `"parsing"* OR "error"* OR "in"* OR
"the"* OR "pdf"* OR "converter"*` matched 127,145 atoms and took 305ms, while a
CJK bigram phrase query with 32 matches took 1.0ms. Latency tracked match count.

- text_index.rare_terms asks the index which of its own words are noise: count
  each term, drop the ones over 5% of the library, keep the rarest few when all
  of them are common. No stop-word list; a corpus of PDFs and a corpus of shell
  history get different answers. 0.05 is swept, not chosen — on n=173
  recall@10 peaks there (0.899 / p50 58.6ms) against 0.893 / 112.9ms at 0.20
  and 0.844 / 13.4ms at 0.01.
- The bound is a cost, not a claim about rarity: `cap = max(int(total*fraction)+1, 2)`,
  so a word matching one atom in eleven is kept — scoring one candidate is free.
  A capped count cannot rank words that all saturate, so those few are counted
  exactly.
- Schema v6 adds `term_df`: a word is counted once and the next question pays
  nothing. Every atom write clears it, so a stored count cannot outlive the
  library it described. Cold p50 73ms, warm 63ms, 1,324 terms cached.
- The denominator is `count(*) FROM atoms` (0.01ms) rather than
  `count(*) FROM atom_fts` (37ms cold): FTS rows are written and deleted with
  their atoms, so the two counts move together and only one of them is free.

SCALE.md, idle machine (load 1.76, 32 cores, 4 threads per query):
165,973 atoms — 1.66x the contract's 100k — 300 probes, p50 57ms, p95 138ms.
Against the frozen baseline of 20.16s that is 354x. Holder-scoped (n=173):
recall@10 0.870, MRR 0.740, p50 62ms.

Verification: pytest backend/tests bench/tests -q -> 173 passed (5 new
text_index tests: the common word is dropped and the rare one still answers, an
all-common query keeps its rarest words in order, a one-word query is never
filtered, an empty library keeps everything, and a write voids the counts);
ruff clean; mypy fresh cache clean; pyright 0 errors on the touched files.

backend/tests/test_text_index.py

@@ -7,10 +7,22 @@ here inserts indexed text and asks the engine the question the user would ask.
 from __future__ import annotations
 
 import sqlite3
+from pathlib import Path
 
 import pytest
 
-from ely_eye.text_index import fts_query, index_text
+from ely_eye.config import Settings
+from ely_eye.db import Database
+from ely_eye.memory import MemoryService
+from ely_eye.schemas import MemoryOrigin
+from ely_eye.text_index import (
+    RAREST_KEEP,
+    fts_expression,
+    fts_query,
+    fts_terms,
+    index_text,
+    rare_terms,
+)
 
 
 def _fts(rows: list[str]) -> sqlite3.Connection:
@@ -108,3 +120,87 @@ def test_a_mixed_query_ranks_the_memory_holding_both_halves_first() -> None:
 
     assert ranked[0] == "钥匙放在 drawer 里"
     assert set(ranked) == {"钥匙放在 drawer 里", "钥匙在玄关", "the drawer is empty"}
+
+
+def _library(rows: list[str]) -> sqlite3.Connection:
+    """A store shaped like the product's: atoms, its FTS mirror, and the count cache."""
+    connection = sqlite3.connect(":memory:")
+    connection.row_factory = sqlite3.Row
+    connection.execute("CREATE TABLE atoms (atom_id TEXT PRIMARY KEY, text TEXT NOT NULL)")
+    connection.execute(
+        "CREATE VIRTUAL TABLE atom_fts USING fts5(atom_id UNINDEXED, source, text, tokenize = 'unicode61')"
+    )
+    connection.execute("CREATE TABLE term_df (term TEXT PRIMARY KEY, documents INTEGER NOT NULL)")
+    for index, row in enumerate(rows):
+        connection.execute("INSERT INTO atoms (atom_id, text) VALUES (?, ?)", (f"a{index}", row))
+        connection.execute(
+            "INSERT INTO atom_fts (atom_id, source, text) VALUES (?, ?, ?)",
+            (f"a{index}", "", index_text(row)),
+        )
+    return connection
+
+
+def _shelf(word: str) -> list[str]:
+    return [f"the {other} {word} converter" for other in ("csv", "xml", "json", "html", "yaml")]
+
+
+def test_a_word_the_library_uses_everywhere_is_dropped_from_the_query() -> None:
+    """bm25 scores every row an OR query touches, so a common word buys cost, not answers."""
+    rows = _shelf("pdf") + _shelf("docx") + ["the parser follows the hashhop chain"]
+    connection = _library(rows)
+
+    kept = rare_terms(connection, fts_terms("the hashhop"))
+
+    assert kept == ["hashhop"], "the word every row carries must not decide how many rows get scored"
+    hits = connection.execute(
+        "SELECT atoms.text FROM atoms JOIN atom_fts ON atom_fts.atom_id = atoms.atom_id WHERE atom_fts MATCH ?",
+        (fts_expression(kept),),
+    ).fetchall()
+    assert [row[0] for row in hits] == ["the parser follows the hashhop chain"]
+
+
+def test_a_query_whose_words_are_all_common_keeps_the_rarest_of_them() -> None:
+    """Dropping every word would search for nothing, so the least common words answer instead."""
+    rows = [f"alpha {other}" for other in ("one", "two", "three", "four", "five")] + [
+        f"beta {other}" for other in ("six", "seven")
+    ]
+    connection = _library(rows)
+
+    kept = rare_terms(connection, ["alpha", "beta"])
+
+    assert len(kept) == min(RAREST_KEEP, 2)
+    assert kept == ["beta", "alpha"], "the rarer word must lead"
+
+
+def test_a_query_of_one_word_is_never_filtered() -> None:
+    connection = _library(_shelf("pdf"))
+
+    assert rare_terms(connection, ["the"]) == ["the"], "dropping the only word would leave nothing to match"
+
+
+def test_an_empty_library_keeps_every_term() -> None:
+    connection = _library([])
+
+    assert rare_terms(connection, ["the", "hashhop"]) == ["the", "hashhop"]
+
+
+def test_the_remembered_match_counts_are_voided_by_a_write(tmp_path: Path) -> None:
+    """A count is only true of the library it was measured against, so any atom write clears it."""
+    settings = Settings(home=tmp_path)
+    memory = MemoryService(settings=settings, database=Database(settings))
+    for index in range(20):
+        memory.remember(f"the converter parses format {index} natively", holder_id="wife", origin=MemoryOrigin.self_)
+    memory.db.search_atoms("the converter", limit=5)
+
+    with memory.db.connect() as connection:
+        assert int(connection.execute("SELECT count(*) FROM term_df").fetchone()[0]) > 0
+
+    memory.remember("the parser follows the hashhop chain", holder_id="wife", origin=MemoryOrigin.self_)
+
+    with memory.db.connect() as connection:
+        assert int(connection.execute("SELECT count(*) FROM term_df").fetchone()[0]) == 0
+
+    page = memory.db.search_atoms("hashhop chain", limit=5)
+    assert "the parser follows the hashhop chain" in [
+        atom.text for atom in memory.db.get_atoms([row[0] for row in page.rows])
+    ], "a cleared cache must cost time, not answers"