Make the proof suite say what it actually ran

The suite had fifteen checks that could not lose. The visual HashHop arena
painted a cyan line and a white ring on the answer slot, then asked the model
to find the answer by looking; the collector re-rolled puzzles until the
running rate crossed its target; the cache check passed if any layer had ever
recorded an entry; the repository check passed on a call graph built in memory
without touching the store; the research-module check asserted non-empty lists
of SHA256-derived placeholder vectors and so could not fail.

Each check now answers for one claim and reports a verdict the reader can act
on: passed, failed, deferred, or removed. A deferred row names what is missing
and the command that would settle it; a removed row says why the old claim was
withdrawn. The suite verdict is failed > deferred > passed, so a claim that
never ran holds the whole report back from saying "proven".

- evals.py (2427 lines) becomes the proofs/ package; largest module 445 lines.
- proofs/__main__.py deleted: it duplicated PRDProofSuite.latest() with a
  different glob. proofs/table.py renders; latest-proof-suite is the one face
  and exits 1 on a failed verdict.
- The arena paints nothing. The puzzle id is printed on its tile instead,
  because removing the ink without it removes the question.
- HashHop runs a fixed 8 puzzles per depth instead of re-rolling to target.
- The contradiction corpus moves from code to contradiction_scenarios.json;
  the check stays deferred while the corpus is the one this repo authored.
- The cache check compiles a nonce question over a real atom twice and reads
  the miss to hit movement; an empty store fails instead of agreeing by luck.
- The repository check requires rows in the store; the research-module check is
  removed.

Two product bugs surfaced on the way:
- text_index.fts_query joined Latin terms with a space, which FTS5 reads as
  AND, so a question longer than its answer retrieved nothing. Terms are ORed
  and bm25 rank orders the candidates.
- profiles.windows_path_to_wsl prefixed /mnt/ onto an already-POSIX path with
  an empty drive, producing /mnt//home/... and failing every launcher probe.

Verification: 168 passed (pytest backend/tests bench/tests); ruff clean on the
proofs package, cli.py, text_index.py, profiles.py, and both test files; mypy
clean on 16 files with a fresh cache; pyright reports 0 errors on proofs/ and
the test file. Ruff debt elsewhere is unchanged: 19 errors in seven files this
commit does not touch, measured against the same files at HEAD.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 18:39:46 -04:00
1 parent dfc6cd4c28
commit 4ab36cd5eb
27 files changed
+4284 -2453

No files matched your search

+9 -5
View File
@@ -9,6 +9,8 @@ from .config import Settings, get_settings
from .db import Database
from .schemas import CacheLayerStatus
SEMANTIC_LAYER = "semantic"
class CacheFabric:
def __init__(self, settings: Settings | None = None, database: Database | None = None) -> None:
@@ -16,7 +18,7 @@ class CacheFabric:
self.db = database or Database(self.settings)
def semantic_key(self, question: str, profile: str) -> str:
return hashlib.sha256(f"compiler-v2\n{profile}\n{question}".encode("utf-8")).hexdigest()
return hashlib.sha256(f"compiler-v2\n{profile}\n{question}".encode()).hexdigest()
def record_hit(self, layer: str, cache_key: str, bytes_estimate: int = 0, metadata: dict[str, Any] | None = None) -> None:
self.db.log_cache_event(layer, cache_key, "hit", bytes_estimate, metadata)
@@ -29,15 +31,17 @@ class CacheFabric:
path.parent.mkdir(parents=True, exist_ok=True)
data = json.dumps(payload, ensure_ascii=False, indent=2).encode("utf-8")
path.write_bytes(data)
self.db.log_cache_event("semantic", cache_key, "write", len(data), {"path": str(path)})
self.db.log_cache_event(SEMANTIC_LAYER, cache_key, "write", len(data), {"path": str(path)})
def read_semantic_cache(self, cache_key: str) -> dict[str, Any] | None:
"""Return the cached payload, or None. The caller traces the event: one compile, one event."""
path = self.settings.cache_dir / "semantic" / f"{cache_key}.json"
if not path.exists():
self.record_miss("semantic", cache_key)
return None
self.record_hit("semantic", cache_key, path.stat().st_size)
return json.loads(path.read_text(encoding="utf-8"))
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as error:
raise ValueError(f"{path}: the semantic cache entry is unreadable ({error})") from error
def statuses(self) -> list[CacheLayerStatus]:
rows = self.db.cache_layer_rows()