Make the proof suite say what it actually ran

The suite had fifteen checks that could not lose. The visual HashHop arena
painted a cyan line and a white ring on the answer slot, then asked the model
to find the answer by looking; the collector re-rolled puzzles until the
running rate crossed its target; the cache check passed if any layer had ever
recorded an entry; the repository check passed on a call graph built in memory
without touching the store; the research-module check asserted non-empty lists
of SHA256-derived placeholder vectors and so could not fail.

Each check now answers for one claim and reports a verdict the reader can act
on: passed, failed, deferred, or removed. A deferred row names what is missing
and the command that would settle it; a removed row says why the old claim was
withdrawn. The suite verdict is failed > deferred > passed, so a claim that
never ran holds the whole report back from saying "proven".

- evals.py (2427 lines) becomes the proofs/ package; largest module 445 lines.
- proofs/__main__.py deleted: it duplicated PRDProofSuite.latest() with a
  different glob. proofs/table.py renders; latest-proof-suite is the one face
  and exits 1 on a failed verdict.
- The arena paints nothing. The puzzle id is printed on its tile instead,
  because removing the ink without it removes the question.
- HashHop runs a fixed 8 puzzles per depth instead of re-rolling to target.
- The contradiction corpus moves from code to contradiction_scenarios.json;
  the check stays deferred while the corpus is the one this repo authored.
- The cache check compiles a nonce question over a real atom twice and reads
  the miss to hit movement; an empty store fails instead of agreeing by luck.
- The repository check requires rows in the store; the research-module check is
  removed.

Two product bugs surfaced on the way:
- text_index.fts_query joined Latin terms with a space, which FTS5 reads as
  AND, so a question longer than its answer retrieved nothing. Terms are ORed
  and bm25 rank orders the candidates.
- profiles.windows_path_to_wsl prefixed /mnt/ onto an already-POSIX path with
  an empty drive, producing /mnt//home/... and failing every launcher probe.

Verification: 168 passed (pytest backend/tests bench/tests); ruff clean on the
proofs package, cli.py, text_index.py, profiles.py, and both test files; mypy
clean on 16 files with a fresh cache; pyright reports 0 errors on proofs/ and
the test file. Ruff debt elsewhere is unchanged: 19 errors in seven files this
commit does not touch, measured against the same files at HEAD.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 18:39:46 -04:00
1 parent dfc6cd4c28
commit 4ab36cd5eb
27 files changed
+4284 -2453

No files matched your search

+38 -3
View File
@@ -24,6 +24,16 @@ def _hits(connection: sqlite3.Connection, query: str) -> list[str]:
return [row[0] for row in connection.execute("SELECT original FROM fts WHERE fts MATCH ?", (fts_query(query),))]
def _ranked(connection: sqlite3.Connection, query: str) -> list[str]:
"""Matches in bm25 order: the answer that covers more of the question should come first."""
return [
row[0]
for row in connection.execute(
"SELECT original FROM fts WHERE fts MATCH ? ORDER BY rank", (fts_query(query),)
)
]
def test_index_text_splits_a_chinese_sentence_into_characters() -> None:
assert index_text("我们的钥匙放在玄关抽屉里") == "我 们 的 钥 匙 放 在 玄 关 抽 屉 里"
@@ -40,7 +50,28 @@ def test_index_text_folds_full_width_letters_that_search_would_miss() -> None:
def test_fts_query_phrases_cjk_and_prefixes_latin() -> None:
assert fts_query("钥匙") == '"钥 匙"'
assert fts_query("search atoms") == '"search"* "atoms"*'
assert fts_query("search atoms") == '"search"* OR "atoms"*'
def test_a_long_question_reaches_the_answer_that_covers_most_of_its_words() -> None:
"""AND over every word answers a long question with nothing; bm25 picks the best candidate."""
connection = _fts(
[
"PRD requirements prove long context memory, HashHop proof, and adapter training.",
"The adapter was trained for 29 steps on 37 samples.",
"Unrelated notes about the weather.",
]
)
question = "Which Ely-Eye PRD requirements prove long context memory, HashHop, and adapters?"
hits = [
row[0]
for row in connection.execute(
"SELECT original FROM fts WHERE fts MATCH ? ORDER BY rank", (fts_query(question),)
)
]
assert hits[0].startswith("PRD requirements prove")
assert "Unrelated notes about the weather." not in hits
def test_fts_query_refuses_a_query_with_nothing_to_match() -> None:
@@ -70,6 +101,10 @@ def test_a_prefix_reaches_the_word_it_grew_into() -> None:
assert _hits(connection, "forg") == ["the atom was forgotten"]
def test_a_mixed_query_needs_both_halves_in_the_same_memory() -> None:
def test_a_mixed_query_ranks_the_memory_holding_both_halves_first() -> None:
"""A query with a Chinese half and an English half must prefer the memory holding both."""
connection = _fts(["钥匙放在 drawer 里", "钥匙在玄关", "the drawer is empty"])
assert _hits(connection, "钥匙 drawer") == ["钥匙放在 drawer 里"]
ranked = _ranked(connection, "钥匙 drawer")
assert ranked[0] == "钥匙放在 drawer 里"
assert set(ranked) == {"钥匙放在 drawer 里", "钥匙在玄关", "the drawer is empty"}