Make the proof suite say what it actually ran

The suite had fifteen checks that could not lose. The visual HashHop arena
painted a cyan line and a white ring on the answer slot, then asked the model
to find the answer by looking; the collector re-rolled puzzles until the
running rate crossed its target; the cache check passed if any layer had ever
recorded an entry; the repository check passed on a call graph built in memory
without touching the store; the research-module check asserted non-empty lists
of SHA256-derived placeholder vectors and so could not fail.

Each check now answers for one claim and reports a verdict the reader can act
on: passed, failed, deferred, or removed. A deferred row names what is missing
and the command that would settle it; a removed row says why the old claim was
withdrawn. The suite verdict is failed > deferred > passed, so a claim that
never ran holds the whole report back from saying "proven".

- evals.py (2427 lines) becomes the proofs/ package; largest module 445 lines.
- proofs/__main__.py deleted: it duplicated PRDProofSuite.latest() with a
  different glob. proofs/table.py renders; latest-proof-suite is the one face
  and exits 1 on a failed verdict.
- The arena paints nothing. The puzzle id is printed on its tile instead,
  because removing the ink without it removes the question.
- HashHop runs a fixed 8 puzzles per depth instead of re-rolling to target.
- The contradiction corpus moves from code to contradiction_scenarios.json;
  the check stays deferred while the corpus is the one this repo authored.
- The cache check compiles a nonce question over a real atom twice and reads
  the miss to hit movement; an empty store fails instead of agreeing by luck.
- The repository check requires rows in the store; the research-module check is
  removed.

Two product bugs surfaced on the way:
- text_index.fts_query joined Latin terms with a space, which FTS5 reads as
  AND, so a question longer than its answer retrieved nothing. Terms are ORed
  and bm25 rank orders the candidates.
- profiles.windows_path_to_wsl prefixed /mnt/ onto an already-POSIX path with
  an empty drive, producing /mnt//home/... and failing every launcher probe.

Verification: 168 passed (pytest backend/tests bench/tests); ruff clean on the
proofs package, cli.py, text_index.py, profiles.py, and both test files; mypy
clean on 16 files with a fresh cache; pyright reports 0 errors on proofs/ and
the test file. Ruff debt elsewhere is unchanged: 19 errors in seven files this
commit does not touch, measured against the same files at HEAD.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 18:39:46 -04:00
1 parent dfc6cd4c28
commit 4ab36cd5eb
27 files changed
+4284 -2453

No files matched your search

+10 -3
View File
@@ -3,7 +3,7 @@ from __future__ import annotations
import hashlib
from collections.abc import Iterable
from .cache_fabric import CacheFabric
from .cache_fabric import SEMANTIC_LAYER, CacheFabric
from .config import Settings, get_settings
from .db import Database
from .retrieval import RetrievalService
@@ -27,11 +27,18 @@ class ContextCompiler:
def compile(self, question: str, profile: RuntimeProfile) -> CompiledContext:
cache_key = self.cache.semantic_key(question, profile.value)
cached = self.cache.read_semantic_cache(cache_key)
if cached:
if cached is not None:
compiled = CompiledContext.model_validate(cached)
if compiled.hits:
self.cache.record_hit(SEMANTIC_LAYER, cache_key)
return compiled
# A cached context with no evidence in it is not an answer, so counting it as a hit
# would let an empty cache entry look like a warm cache.
self.cache.record_miss(
SEMANTIC_LAYER, cache_key, metadata={"reason": "cached context has no hits"}
)
else:
self.cache.record_miss(SEMANTIC_LAYER, cache_key)
plan = self.plan(question, profile)
hits = self.retrieval.search(plan.retrieval_query, plan.evidence_budget_atoms).hits
packed_text, token_total = self._pack_hits(hits, plan.token_budget, plan.retrieval_query)