Put the stop-word bound's evidence in the repo, not in a scratch script

COMMON_TERM_FRACTION = 0.05 is a magic number unless something prints the curve it
came from. bench/term_sweep.py asks the frozen suite as its holders would, once per
candidate bound, and writes recall and latency into the same row of
bench/reports/TERMS.md.

rare_terms now resolves its bound when the call runs instead of when the function was
defined, which is the only way a harness can move it. The default value is unchanged,
so the reports stamped at the parent commit still describe this code.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-08 14:36:59 -04:00
1 parent cd9851a12c
commit b8f83f422a
2 files changed
+165 -3

No files matched your search

+6 -3
View File
@@ -68,7 +68,7 @@ COMMON_TERM_FRACTION = 0.05
RAREST_KEEP = 3
def rare_terms(connection: sqlite3.Connection, terms: list[str], *, fraction: float = COMMON_TERM_FRACTION) -> list[str]:
def rare_terms(connection: sqlite3.Connection, terms: list[str], *, fraction: float | None = None) -> list[str]:
"""Drop the terms a library uses everywhere, and keep the ones that identify an answer.
bm25 scores every row an OR query touches, so a question carrying `the` pays for a hundred
@@ -85,14 +85,17 @@ def rare_terms(connection: sqlite3.Connection, terms: list[str], *, fraction: fl
counted exactly.
0.05 is measured, not chosen: on the frozen suite (n=173, holder-scoped) recall@10 peaks at
0.899 with p50 58.6 ms there, against 0.893 / 112.9 ms at 0.20 and 0.844 / 13.4 ms at 0.01.
0.899 with p50 58.6 ms there, against 0.893 / 112.9 ms at 0.20 and 0.844 / 13.4 ms at 0.01, and
`bench/term_sweep.py` is the artifact that prints that curve. The bound is read when the call
runs, not when the function was written, so that sweep can move it.
"""
bound = COMMON_TERM_FRACTION if fraction is None else fraction
if len(terms) == 1:
return terms
total = int(connection.execute("SELECT count(*) FROM atoms").fetchone()[0])
if total == 0:
return terms
cap = max(int(total * fraction) + 1, 2)
cap = max(int(total * bound) + 1, 2)
counts = [(term, _term_matches(connection, fts_expression([term]), cap)) for term in terms]
rare = [term for term, matches in counts if matches < cap]
if rare: