Make the proof suite say what it actually ran
The suite had fifteen checks that could not lose. The visual HashHop arena painted a cyan line and a white ring on the answer slot, then asked the model to find the answer by looking; the collector re-rolled puzzles until the running rate crossed its target; the cache check passed if any layer had ever recorded an entry; the repository check passed on a call graph built in memory without touching the store; the research-module check asserted non-empty lists of SHA256-derived placeholder vectors and so could not fail. Each check now answers for one claim and reports a verdict the reader can act on: passed, failed, deferred, or removed. A deferred row names what is missing and the command that would settle it; a removed row says why the old claim was withdrawn. The suite verdict is failed > deferred > passed, so a claim that never ran holds the whole report back from saying "proven". - evals.py (2427 lines) becomes the proofs/ package; largest module 445 lines. - proofs/__main__.py deleted: it duplicated PRDProofSuite.latest() with a different glob. proofs/table.py renders; latest-proof-suite is the one face and exits 1 on a failed verdict. - The arena paints nothing. The puzzle id is printed on its tile instead, because removing the ink without it removes the question. - HashHop runs a fixed 8 puzzles per depth instead of re-rolling to target. - The contradiction corpus moves from code to contradiction_scenarios.json; the check stays deferred while the corpus is the one this repo authored. - The cache check compiles a nonce question over a real atom twice and reads the miss to hit movement; an empty store fails instead of agreeing by luck. - The repository check requires rows in the store; the research-module check is removed. Two product bugs surfaced on the way: - text_index.fts_query joined Latin terms with a space, which FTS5 reads as AND, so a question longer than its answer retrieved nothing. Terms are ORed and bm25 rank orders the candidates. - profiles.windows_path_to_wsl prefixed /mnt/ onto an already-POSIX path with an empty drive, producing /mnt//home/... and failing every launcher probe. Verification: 168 passed (pytest backend/tests bench/tests); ruff clean on the proofs package, cli.py, text_index.py, profiles.py, and both test files; mypy clean on 16 files with a fresh cache; pyright reports 0 errors on proofs/ and the test file. Ruff debt elsewhere is unchanged: 19 errors in seven files this commit does not touch, measured against the same files at HEAD.
This commit is contained in:
1 parent
dfc6cd4c28
commit
4ab36cd5eb
27 files changed
+4284
-2453
No files matched your search
@@ -20,10 +20,12 @@ from .cli_transactive import register as register_transactive_commands
|
||||
from .config import get_settings
|
||||
from .db import Database
|
||||
from .embeddings import EmbeddingService
|
||||
from .evals import HashHopEvaluator, PRDProofSuite
|
||||
from .ingestion import IngestionService
|
||||
from .memory import MemoryService
|
||||
from .profiles import runtime_profile_report
|
||||
from .proofs import HashHopEvaluator, PRDProofSuite
|
||||
from .proofs.contradictions import CORPUS_NAME, build_contradiction_scenarios, write_corpus
|
||||
from .proofs.table import render_table
|
||||
from .research_modules import ResearchModuleService
|
||||
from .retrieval import RetrievalService
|
||||
from .runtime import QwenRuntime
|
||||
@@ -232,11 +234,19 @@ def proof_suite(cartridge_id: str | None = None) -> None:
|
||||
|
||||
|
||||
@app.command("latest-proof-suite")
|
||||
def latest_proof_suite() -> None:
|
||||
def latest_proof_suite(json_output: bool = typer.Option(False, "--json")) -> None:
|
||||
"""Print what the last proof suite claimed and what it actually proved."""
|
||||
report = PRDProofSuite(get_settings()).latest()
|
||||
if report is None:
|
||||
raise typer.BadParameter(
|
||||
"no proof suite has been recorded yet; run: ely-eye proof-suite --cartridge-id <id>"
|
||||
)
|
||||
if json_output:
|
||||
typer.echo(report.model_dump_json(indent=2))
|
||||
else:
|
||||
typer.echo(render_table(report))
|
||||
if report.status == "failed":
|
||||
raise typer.Exit(code=1)
|
||||
typer.echo(report.model_dump_json(indent=2))
|
||||
|
||||
|
||||
@app.command("runtime-generation-proof")
|
||||
@@ -251,6 +261,14 @@ def visual_contradiction_proof() -> None:
|
||||
typer.echo(check.model_dump_json(indent=2))
|
||||
|
||||
|
||||
@app.command("contradiction-corpus")
|
||||
def contradiction_corpus(data_dir: Path | None = None) -> None:
|
||||
"""Write the corpus the contradiction lens is graded on, so it can be read and replaced."""
|
||||
settings = get_settings()
|
||||
target = write_corpus((data_dir or settings.data_dir) / CORPUS_NAME)
|
||||
typer.echo(json_dump({"corpus": str(target), "scenarios": len(build_contradiction_scenarios())}))
|
||||
|
||||
|
||||
@app.command("finalize-cartridge")
|
||||
def finalize_cartridge(cartridge_id: str) -> None:
|
||||
report = CartridgeAssetService(get_settings()).finalize(cartridge_id)
|
||||
|
||||
Reference in new issue
Block a user