From bf79a9a22e28525641cfd3ed1426f20bca0a4a00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E9=9B=B7=E7=94=B5=E8=8A=BD=E8=A1=A3?= Date: Mon, 20 Apr 2026 11:41:25 -0400 Subject: [PATCH] Add B5.1 proposal for LLM-as-judge tier above assertion judge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document the follow-up tier on top of the B5 assertion harness: thin single-call rubric judge, additive RubricDimension/RubricVerdict types, cost cap ($2/run) with loud-skip behavior, per-fixture rubric cap (8 dimensions), baseline acceptance workflow, and explicit trigger conditions (10+ fixtures / 2 incidents / structural-miss / 50-run zero catch) — do not build speculatively. Roundtable outcome: 2-1 to defer tonight. Linus — don't bolt a second LLM to an empty harness. Jobs — single signal first. Karpathy — fixture accumulation is what will tell us which rubric dimensions matter; the schema stays extensible so we can add rubric-per-fixture without breaking B5. --- docs/proposals/B5.1-llm-as-judge-followup.md | 260 +++++++++++++++++++ 1 file changed, 260 insertions(+) create mode 100644 docs/proposals/B5.1-llm-as-judge-followup.md diff --git a/docs/proposals/B5.1-llm-as-judge-followup.md b/docs/proposals/B5.1-llm-as-judge-followup.md new file mode 100644 index 00000000..871fdcd4 --- /dev/null +++ b/docs/proposals/B5.1-llm-as-judge-followup.md @@ -0,0 +1,260 @@ +# B5.1 — LLM-as-judge tier above the B5 assertion harness + +> Status: proposal (deferred — triggers defined below) +> Depends on: B5 (assertion harness scaffold, landed 2026-04-20) +> Author note: roundtable 2-1 decision to defer this tonight; scaffold first, +> semantic judgment after N ≥ 10 fixtures exist. + +## TL;DR + +B5 today handles **structural** assertions against a recorded trace — tool +name, call count, call order, argument shape, turn cap, text substring. It +cannot judge whether a CEO's hiring plan is *reasonable*, whether a reply +tone is *appropriate*, or whether a response *addresses the user's intent*. +Semantic judgment belongs in a second tier — an LLM-as-judge that runs on +top of B5's fixtures, gated by a rubric, a cost budget, and a failure +escape hatch so it never becomes the bottleneck it wants to measure. + +This proposal defines the trigger conditions for building that tier, the +concrete rubric schema, the assertion-first fast path, the cost cap, and +the failure mode when the judge itself is wrong. + +## Why not tonight (roundtable reasoning) + +Three dissenting voices carried the delay decision 2-1: + +**Linus** — *"You don't bolt a second LLM onto an eval harness that has +zero fixtures. Build the bone first. Bolt the muscle when you have +something to measure."* Structural assertions work with 1 fixture. An +LLM-judge tier is pointless below ~10 fixtures because you can't +distinguish judge error from fixture sparseness. + +**Jobs** — *"Single signal source. Tonight the developer needs one +needle: 'did the trace match the recorded shape, yes or no.' A judge +introduces a second signal (confidence, rubric score) that dilutes +focus on the first bug we haven't found yet."* + +**Karpathy (me)** — agreed on the delay because the assertion tier *is* +the forcing function that reveals which behaviors actually matter. The +fixtures I record in the next two weeks will tell me exactly which +rubric dimensions the judge should grade. Building the judge rubric from +a blank page today is premature optimization. + +**Dissent (minority)** — the argument for shipping tonight was that +CEO-mode bootstrap behavior is qualitative enough that assertions alone +miss the point of B5. The counter: if a behavior can only be judged +semantically, that's a signal the behavior spec itself is underdetermined +— fix the spec before fixing the eval tier. + +## Trigger conditions (do this when any of the below holds) + +| Condition | Why it triggers B5.1 | +|---|---| +| `tests/eval/fixtures/` has 10+ recorded fixtures | Enough surface to distinguish rubric gaps from fixture gaps | +| Two separate production incidents trace back to a behavior that passed all assertions but was still wrong | Signal that structural checks miss semantic drift | +| A fixture exists where the desired assertion cannot be written as a structural match (example: "CEO should not sound defensive") | Direct evidence structural assertions are out of scope for this case | +| Quarterly review: B5 has caught zero regressions across 50+ fixture runs | Suggests structural assertions are over-fitting to already-green shapes — add semantic judgment before assuming coverage is adequate | + +Any one condition is sufficient. Do not build B5.1 speculatively on a +"maybe we'll need it" basis — build it the moment one concrete trigger +fires, and tie the PR to that trigger in the commit message. + +## Tier architecture + +### Tier 0 — fixture loader + structural judge (B5, landed) + +The cheap tier. `O(trace_length)` time, zero network calls. Runs in +`vitest` under `pnpm test:eval`. Every fixture must go through this +tier; failures here short-circuit Tier 1 because semantic judgment on a +structurally broken trace is nonsense. + +### Tier 1 — LLM rubric judge (this proposal) + +Runs only if Tier 0 passes. Consults an LLM with: + +1. The system prompt used during recording (retrievable via the + `systemPromptHash` in `meta`). +2. The recorded transcript (user / assistant / toolResult messages in + order). +3. A **rubric** — a structured list of yes/no grading dimensions unique + to this fixture (stored as part of the fixture's `assertions` block + under a new `rubric` field, see schema below). +4. A maximum token budget for the judge call. + +The judge returns a JSON verdict: `{ dimension, passed, reason }[]`. The +runner aggregates these into the `EvalResult` alongside structural +assertions. + +### Rubric schema (proposed, additive to B5 types) + +```ts +export interface RubricDimension { + /** Short kebab-case identifier stable across fixture revisions. */ + id: string; + /** Human-readable question the judge answers yes/no. */ + question: string; + /** Required: concrete phrasing of what a pass looks like. */ + passRubric: string; + /** Required: concrete phrasing of what a fail looks like. */ + failRubric: string; +} + +export interface RubricVerdict { + dimensionId: string; + passed: boolean; + reason: string; + judgeModel: string; + judgeTokensUsed: { input: number; output: number }; +} +``` + +The rubric lives inline in the fixture's assertions block as +`{ "rubric": [...] }`. The existing structural `assertions` array stays. +Backwards-compatible — fixtures without a rubric skip Tier 1 entirely. + +### Thin call shape (stay cheap) + +One call per fixture. Not one call per dimension. The prompt enumerates +all dimensions and asks for a single structured JSON response covering +them all. This keeps per-fixture cost to one round trip. + +``` +System: You are a rubric judge. For each dimension, respond yes or no +with a one-sentence reason. Never repeat the rubric. Never invent +dimensions. + +User: +[transcript verbatim] + +Dimensions to grade: +1. : + Pass if: + Fail if: + +Respond as JSON: [{"dimensionId": "...", "passed": true|false, "reason": "..."}] +``` + +### Cost cap + +Hard cap per `pnpm test:eval` run: **$2.00 of judge cost**. Implementation: + +1. Configure a fixed model (recommend `claude-sonnet` class — cheap + enough for high-volume judge calls, smart enough for rubric reasoning). +2. Runner tracks cumulative `judgeTokensUsed × model.cost` across + fixtures in a run. +3. Once the cap is hit, remaining fixtures run Tier 0 only and their + Tier 1 results record `skipped: true, reason: "cost cap exceeded"`. +4. The run exits non-zero if cost-cap skip would have masked a prior + Tier 0 pass — i.e. the run fails loud when skipping would hide a + genuine regression. + +Rationale: an eval tier that can silently cost $100 when a fixture +balloons is not an eval tier, it's a budget exfil vector. + +### Fast-path escape hatch + +Tier 1 runs only when at least one of the fixture's `rubric` dimensions +differs from the previous eval snapshot. The runner keeps a local +SHA-keyed cache `(fixtureId, rubric hash, transcript hash) → verdict`. +Identical inputs skip the LLM call and reuse the cached verdict. + +Rationale: if the CEO's trace hasn't changed and the rubric hasn't +changed, the verdict won't change either. Running it again is pure +waste. First-run of a new fixture or rubric still pays the call. + +## Failure modes (things that will actually go wrong) + +### The judge is wrong + +The judge's own failure mode is non-trivial: it can mark a good trace as +failing (false positive) or a bad trace as passing (false negative). +Mitigations: + +1. **Every Tier 1 failure requires a human sign-off to baseline.** The + first time a rubric dimension flips from pass to fail, the runner + does not treat the run as failed — it prints the verdict diff and + asks the developer to promote or reject it via a `pnpm eval:accept + --fixture --dimension ` command. After acceptance, the + baseline is updated and future matching verdicts pass silently. +2. **Judge disagreement logging.** When Tier 1 fails but Tier 0 passes, + the reason text is preserved in the fixture's `.accepted.jsonl` + sidecar for later review. + +### The rubric drifts out of date + +Fixtures grow rubric dimensions, rubrics get more verbose, the judge's +context window fills up. Mitigations: + +1. Per-fixture cap: **no more than 8 rubric dimensions**. Enforced at + load time — a 9th dimension is a load error, forcing the developer to + either split the fixture or prune rubrics. +2. **Rubric dimensions must be single-sentence pass/fail rules.** Long + prose rubrics are compressed or rejected at load. + +### Provider outage blocks CI + +Mitigation: if the judge provider is unreachable, Tier 1 verdicts record +`skipped: true, reason: "provider unreachable"` and the run exits with a +warning but **does not fail the CI build**. Tier 0 still gates the run. +The judge is a supplement, not a must-pass gate, for exactly this reason. + +## Explicit non-goals + +- **Not a teacher for the CEO agent.** Tier 1 does not produce training + data, does not feed back into prompts, does not auto-patch behavior. + It grades; humans act on grades. +- **Not a replacement for structural assertions.** Tier 0 runs always, + gates Tier 1, and handles 80%+ of regressions. If you find yourself + deleting Tier 0 assertions because Tier 1 covers them, stop — the + structural tier is cheaper and faster and reveals different bugs. +- **Not a judge that rewrites fixtures.** The judge reads; it does not + author rubrics, does not select which dimensions to grade, does not + modify the `.jsonl` on disk. + +## What this proposal does not cover + +- Multi-turn "soft" evaluation (did the agent converge on the right + answer across multiple turns?) — future tier, call it B5.2. +- Cross-fixture trend analysis (is the CEO getting more verbose over + time?) — future tier. +- Adversarial fixture generation (synthesizing fixtures that probe edge + cases) — explicitly out of scope; tonight's line in the sand is that + fixtures are *recorded*, never synthesized. + +## Concrete next steps once a trigger fires + +1. Land `RubricDimension` / `RubricVerdict` types in `tests/eval/types.ts` + as an additive extension (existing fixtures without a `rubric` field + continue to work). +2. Write `rubric-judge.ts` that takes a parsed fixture + rubric + judge + model and returns `RubricVerdict[]`. Keep under 250 lines. +3. Extend `eval-runner.ts` to call Tier 1 only after Tier 0 passes, + respecting cost cap and cache. +4. Add `pnpm eval:accept` CLI for baseline management. +5. Migrate one representative fixture to include a rubric block as the + first real-world exercise. Do not bulk-convert — one fixture, review + the judge's behavior, then scale. + +## Approval gate + +This proposal is not auto-approved. Before implementation, the roundtable +(or its equivalent reviewer set) must explicitly acknowledge: + +1. At least one trigger condition has fired. +2. The cost cap is acceptable for the current CI throughput. +3. Baseline acceptance workflow fits the team's dev loop. + +Without all three, stop at B5 structural judge. It's better to have one +tier that works than two tiers that lie. + +## Closing note + +The reason I (Karpathy) pushed for B5 to land tonight *without* this +tier is the same reason I'm drafting this proposal immediately: harness +scaffolds are forcing functions. Every line of the JSONL contract I +nailed down in B5 constrains what B5.1 can cheaply do. Writing the B5.1 +proposal the same night forces me to confirm the B5 schema is +extensible in the directions I'll actually need — and it is +(`assertions` can grow a `rubric` sibling without any breaking change). + +Ship the assertion judge. Record 10 fixtures. Then grade them.