Make the proof suite say what it actually ran

The suite had fifteen checks that could not lose. The visual HashHop arena
painted a cyan line and a white ring on the answer slot, then asked the model
to find the answer by looking; the collector re-rolled puzzles until the
running rate crossed its target; the cache check passed if any layer had ever
recorded an entry; the repository check passed on a call graph built in memory
without touching the store; the research-module check asserted non-empty lists
of SHA256-derived placeholder vectors and so could not fail.

Each check now answers for one claim and reports a verdict the reader can act
on: passed, failed, deferred, or removed. A deferred row names what is missing
and the command that would settle it; a removed row says why the old claim was
withdrawn. The suite verdict is failed > deferred > passed, so a claim that
never ran holds the whole report back from saying "proven".

- evals.py (2427 lines) becomes the proofs/ package; largest module 445 lines.
- proofs/__main__.py deleted: it duplicated PRDProofSuite.latest() with a
  different glob. proofs/table.py renders; latest-proof-suite is the one face
  and exits 1 on a failed verdict.
- The arena paints nothing. The puzzle id is printed on its tile instead,
  because removing the ink without it removes the question.
- HashHop runs a fixed 8 puzzles per depth instead of re-rolling to target.
- The contradiction corpus moves from code to contradiction_scenarios.json;
  the check stays deferred while the corpus is the one this repo authored.
- The cache check compiles a nonce question over a real atom twice and reads
  the miss to hit movement; an empty store fails instead of agreeing by luck.
- The repository check requires rows in the store; the research-module check is
  removed.

Two product bugs surfaced on the way:
- text_index.fts_query joined Latin terms with a space, which FTS5 reads as
  AND, so a question longer than its answer retrieved nothing. Terms are ORed
  and bm25 rank orders the candidates.
- profiles.windows_path_to_wsl prefixed /mnt/ onto an already-POSIX path with
  an empty drive, producing /mnt//home/... and failing every launcher probe.

Verification: 168 passed (pytest backend/tests bench/tests); ruff clean on the
proofs package, cli.py, text_index.py, profiles.py, and both test files; mypy
clean on 16 files with a fresh cache; pyright reports 0 errors on proofs/ and
the test file. Ruff debt elsewhere is unchanged: 19 errors in seven files this
commit does not touch, measured against the same files at HEAD.
This commit is contained in:
ZacharyZhang-NY committed 2026-10-07 18:39:46 -04:00
1 parent dfc6cd4c28
commit 4ab36cd5eb
27 files changed
+4284 -2453

No files matched your search

+21 -3
View File
@@ -20,10 +20,12 @@ from .cli_transactive import register as register_transactive_commands
from .config import get_settings
from .db import Database
from .embeddings import EmbeddingService
from .evals import HashHopEvaluator, PRDProofSuite
from .ingestion import IngestionService
from .memory import MemoryService
from .profiles import runtime_profile_report
from .proofs import HashHopEvaluator, PRDProofSuite
from .proofs.contradictions import CORPUS_NAME, build_contradiction_scenarios, write_corpus
from .proofs.table import render_table
from .research_modules import ResearchModuleService
from .retrieval import RetrievalService
from .runtime import QwenRuntime
@@ -232,11 +234,19 @@ def proof_suite(cartridge_id: str | None = None) -> None:
@app.command("latest-proof-suite")
def latest_proof_suite() -> None:
def latest_proof_suite(json_output: bool = typer.Option(False, "--json")) -> None:
"""Print what the last proof suite claimed and what it actually proved."""
report = PRDProofSuite(get_settings()).latest()
if report is None:
raise typer.BadParameter(
"no proof suite has been recorded yet; run: ely-eye proof-suite --cartridge-id <id>"
)
if json_output:
typer.echo(report.model_dump_json(indent=2))
else:
typer.echo(render_table(report))
if report.status == "failed":
raise typer.Exit(code=1)
typer.echo(report.model_dump_json(indent=2))
@app.command("runtime-generation-proof")
@@ -251,6 +261,14 @@ def visual_contradiction_proof() -> None:
typer.echo(check.model_dump_json(indent=2))
@app.command("contradiction-corpus")
def contradiction_corpus(data_dir: Path | None = None) -> None:
"""Write the corpus the contradiction lens is graded on, so it can be read and replaced."""
settings = get_settings()
target = write_corpus((data_dir or settings.data_dir) / CORPUS_NAME)
typer.echo(json_dump({"corpus": str(target), "scenarios": len(build_contradiction_scenarios())}))
@app.command("finalize-cartridge")
def finalize_cartridge(cartridge_id: str) -> None:
report = CartridgeAssetService(get_settings()).finalize(cartridge_id)