Add B5.1 proposal for LLM-as-judge tier above assertion judge
Document the follow-up tier on top of the B5 assertion harness: thin single-call rubric judge, additive RubricDimension/RubricVerdict types, cost cap ($2/run) with loud-skip behavior, per-fixture rubric cap (8 dimensions), baseline acceptance workflow, and explicit trigger conditions (10+ fixtures / 2 incidents / structural-miss / 50-run zero catch) — do not build speculatively. Roundtable outcome: 2-1 to defer tonight. Linus — don't bolt a second LLM to an empty harness. Jobs — single signal first. Karpathy — fixture accumulation is what will tell us which rubric dimensions matter; the schema stays extensible so we can add rubric-per-fixture without breaking B5.
This commit is contained in:
ZacharyZhang-NY committed
1 parent
2c6dd8ed08
commit
bf79a9a22e
1 file changed
+260
@@ -0,0 +1,260 @@
|
||||
# B5.1 — LLM-as-judge tier above the B5 assertion harness
|
||||
|
||||
> Status: proposal (deferred — triggers defined below)
|
||||
> Depends on: B5 (assertion harness scaffold, landed 2026-04-20)
|
||||
> Author note: roundtable 2-1 decision to defer this tonight; scaffold first,
|
||||
> semantic judgment after N ≥ 10 fixtures exist.
|
||||
|
||||
## TL;DR
|
||||
|
||||
B5 today handles **structural** assertions against a recorded trace — tool
|
||||
name, call count, call order, argument shape, turn cap, text substring. It
|
||||
cannot judge whether a CEO's hiring plan is *reasonable*, whether a reply
|
||||
tone is *appropriate*, or whether a response *addresses the user's intent*.
|
||||
Semantic judgment belongs in a second tier — an LLM-as-judge that runs on
|
||||
top of B5's fixtures, gated by a rubric, a cost budget, and a failure
|
||||
escape hatch so it never becomes the bottleneck it wants to measure.
|
||||
|
||||
This proposal defines the trigger conditions for building that tier, the
|
||||
concrete rubric schema, the assertion-first fast path, the cost cap, and
|
||||
the failure mode when the judge itself is wrong.
|
||||
|
||||
## Why not tonight (roundtable reasoning)
|
||||
|
||||
Three dissenting voices carried the delay decision 2-1:
|
||||
|
||||
**Linus** — *"You don't bolt a second LLM onto an eval harness that has
|
||||
zero fixtures. Build the bone first. Bolt the muscle when you have
|
||||
something to measure."* Structural assertions work with 1 fixture. An
|
||||
LLM-judge tier is pointless below ~10 fixtures because you can't
|
||||
distinguish judge error from fixture sparseness.
|
||||
|
||||
**Jobs** — *"Single signal source. Tonight the developer needs one
|
||||
needle: 'did the trace match the recorded shape, yes or no.' A judge
|
||||
introduces a second signal (confidence, rubric score) that dilutes
|
||||
focus on the first bug we haven't found yet."*
|
||||
|
||||
**Karpathy (me)** — agreed on the delay because the assertion tier *is*
|
||||
the forcing function that reveals which behaviors actually matter. The
|
||||
fixtures I record in the next two weeks will tell me exactly which
|
||||
rubric dimensions the judge should grade. Building the judge rubric from
|
||||
a blank page today is premature optimization.
|
||||
|
||||
**Dissent (minority)** — the argument for shipping tonight was that
|
||||
CEO-mode bootstrap behavior is qualitative enough that assertions alone
|
||||
miss the point of B5. The counter: if a behavior can only be judged
|
||||
semantically, that's a signal the behavior spec itself is underdetermined
|
||||
— fix the spec before fixing the eval tier.
|
||||
|
||||
## Trigger conditions (do this when any of the below holds)
|
||||
|
||||
| Condition | Why it triggers B5.1 |
|
||||
|---|---|
|
||||
| `tests/eval/fixtures/` has 10+ recorded fixtures | Enough surface to distinguish rubric gaps from fixture gaps |
|
||||
| Two separate production incidents trace back to a behavior that passed all assertions but was still wrong | Signal that structural checks miss semantic drift |
|
||||
| A fixture exists where the desired assertion cannot be written as a structural match (example: "CEO should not sound defensive") | Direct evidence structural assertions are out of scope for this case |
|
||||
| Quarterly review: B5 has caught zero regressions across 50+ fixture runs | Suggests structural assertions are over-fitting to already-green shapes — add semantic judgment before assuming coverage is adequate |
|
||||
|
||||
Any one condition is sufficient. Do not build B5.1 speculatively on a
|
||||
"maybe we'll need it" basis — build it the moment one concrete trigger
|
||||
fires, and tie the PR to that trigger in the commit message.
|
||||
|
||||
## Tier architecture
|
||||
|
||||
### Tier 0 — fixture loader + structural judge (B5, landed)
|
||||
|
||||
The cheap tier. `O(trace_length)` time, zero network calls. Runs in
|
||||
`vitest` under `pnpm test:eval`. Every fixture must go through this
|
||||
tier; failures here short-circuit Tier 1 because semantic judgment on a
|
||||
structurally broken trace is nonsense.
|
||||
|
||||
### Tier 1 — LLM rubric judge (this proposal)
|
||||
|
||||
Runs only if Tier 0 passes. Consults an LLM with:
|
||||
|
||||
1. The system prompt used during recording (retrievable via the
|
||||
`systemPromptHash` in `meta`).
|
||||
2. The recorded transcript (user / assistant / toolResult messages in
|
||||
order).
|
||||
3. A **rubric** — a structured list of yes/no grading dimensions unique
|
||||
to this fixture (stored as part of the fixture's `assertions` block
|
||||
under a new `rubric` field, see schema below).
|
||||
4. A maximum token budget for the judge call.
|
||||
|
||||
The judge returns a JSON verdict: `{ dimension, passed, reason }[]`. The
|
||||
runner aggregates these into the `EvalResult` alongside structural
|
||||
assertions.
|
||||
|
||||
### Rubric schema (proposed, additive to B5 types)
|
||||
|
||||
```ts
|
||||
export interface RubricDimension {
|
||||
/** Short kebab-case identifier stable across fixture revisions. */
|
||||
id: string;
|
||||
/** Human-readable question the judge answers yes/no. */
|
||||
question: string;
|
||||
/** Required: concrete phrasing of what a pass looks like. */
|
||||
passRubric: string;
|
||||
/** Required: concrete phrasing of what a fail looks like. */
|
||||
failRubric: string;
|
||||
}
|
||||
|
||||
export interface RubricVerdict {
|
||||
dimensionId: string;
|
||||
passed: boolean;
|
||||
reason: string;
|
||||
judgeModel: string;
|
||||
judgeTokensUsed: { input: number; output: number };
|
||||
}
|
||||
```
|
||||
|
||||
The rubric lives inline in the fixture's assertions block as
|
||||
`{ "rubric": [...] }`. The existing structural `assertions` array stays.
|
||||
Backwards-compatible — fixtures without a rubric skip Tier 1 entirely.
|
||||
|
||||
### Thin call shape (stay cheap)
|
||||
|
||||
One call per fixture. Not one call per dimension. The prompt enumerates
|
||||
all dimensions and asks for a single structured JSON response covering
|
||||
them all. This keeps per-fixture cost to one round trip.
|
||||
|
||||
```
|
||||
System: You are a rubric judge. For each dimension, respond yes or no
|
||||
with a one-sentence reason. Never repeat the rubric. Never invent
|
||||
dimensions.
|
||||
|
||||
User:
|
||||
[transcript verbatim]
|
||||
|
||||
Dimensions to grade:
|
||||
1. <id>: <question>
|
||||
Pass if: <passRubric>
|
||||
Fail if: <failRubric>
|
||||
|
||||
Respond as JSON: [{"dimensionId": "...", "passed": true|false, "reason": "..."}]
|
||||
```
|
||||
|
||||
### Cost cap
|
||||
|
||||
Hard cap per `pnpm test:eval` run: **$2.00 of judge cost**. Implementation:
|
||||
|
||||
1. Configure a fixed model (recommend `claude-sonnet` class — cheap
|
||||
enough for high-volume judge calls, smart enough for rubric reasoning).
|
||||
2. Runner tracks cumulative `judgeTokensUsed × model.cost` across
|
||||
fixtures in a run.
|
||||
3. Once the cap is hit, remaining fixtures run Tier 0 only and their
|
||||
Tier 1 results record `skipped: true, reason: "cost cap exceeded"`.
|
||||
4. The run exits non-zero if cost-cap skip would have masked a prior
|
||||
Tier 0 pass — i.e. the run fails loud when skipping would hide a
|
||||
genuine regression.
|
||||
|
||||
Rationale: an eval tier that can silently cost $100 when a fixture
|
||||
balloons is not an eval tier, it's a budget exfil vector.
|
||||
|
||||
### Fast-path escape hatch
|
||||
|
||||
Tier 1 runs only when at least one of the fixture's `rubric` dimensions
|
||||
differs from the previous eval snapshot. The runner keeps a local
|
||||
SHA-keyed cache `(fixtureId, rubric hash, transcript hash) → verdict`.
|
||||
Identical inputs skip the LLM call and reuse the cached verdict.
|
||||
|
||||
Rationale: if the CEO's trace hasn't changed and the rubric hasn't
|
||||
changed, the verdict won't change either. Running it again is pure
|
||||
waste. First-run of a new fixture or rubric still pays the call.
|
||||
|
||||
## Failure modes (things that will actually go wrong)
|
||||
|
||||
### The judge is wrong
|
||||
|
||||
The judge's own failure mode is non-trivial: it can mark a good trace as
|
||||
failing (false positive) or a bad trace as passing (false negative).
|
||||
Mitigations:
|
||||
|
||||
1. **Every Tier 1 failure requires a human sign-off to baseline.** The
|
||||
first time a rubric dimension flips from pass to fail, the runner
|
||||
does not treat the run as failed — it prints the verdict diff and
|
||||
asks the developer to promote or reject it via a `pnpm eval:accept
|
||||
--fixture <id> --dimension <id>` command. After acceptance, the
|
||||
baseline is updated and future matching verdicts pass silently.
|
||||
2. **Judge disagreement logging.** When Tier 1 fails but Tier 0 passes,
|
||||
the reason text is preserved in the fixture's `.accepted.jsonl`
|
||||
sidecar for later review.
|
||||
|
||||
### The rubric drifts out of date
|
||||
|
||||
Fixtures grow rubric dimensions, rubrics get more verbose, the judge's
|
||||
context window fills up. Mitigations:
|
||||
|
||||
1. Per-fixture cap: **no more than 8 rubric dimensions**. Enforced at
|
||||
load time — a 9th dimension is a load error, forcing the developer to
|
||||
either split the fixture or prune rubrics.
|
||||
2. **Rubric dimensions must be single-sentence pass/fail rules.** Long
|
||||
prose rubrics are compressed or rejected at load.
|
||||
|
||||
### Provider outage blocks CI
|
||||
|
||||
Mitigation: if the judge provider is unreachable, Tier 1 verdicts record
|
||||
`skipped: true, reason: "provider unreachable"` and the run exits with a
|
||||
warning but **does not fail the CI build**. Tier 0 still gates the run.
|
||||
The judge is a supplement, not a must-pass gate, for exactly this reason.
|
||||
|
||||
## Explicit non-goals
|
||||
|
||||
- **Not a teacher for the CEO agent.** Tier 1 does not produce training
|
||||
data, does not feed back into prompts, does not auto-patch behavior.
|
||||
It grades; humans act on grades.
|
||||
- **Not a replacement for structural assertions.** Tier 0 runs always,
|
||||
gates Tier 1, and handles 80%+ of regressions. If you find yourself
|
||||
deleting Tier 0 assertions because Tier 1 covers them, stop — the
|
||||
structural tier is cheaper and faster and reveals different bugs.
|
||||
- **Not a judge that rewrites fixtures.** The judge reads; it does not
|
||||
author rubrics, does not select which dimensions to grade, does not
|
||||
modify the `.jsonl` on disk.
|
||||
|
||||
## What this proposal does not cover
|
||||
|
||||
- Multi-turn "soft" evaluation (did the agent converge on the right
|
||||
answer across multiple turns?) — future tier, call it B5.2.
|
||||
- Cross-fixture trend analysis (is the CEO getting more verbose over
|
||||
time?) — future tier.
|
||||
- Adversarial fixture generation (synthesizing fixtures that probe edge
|
||||
cases) — explicitly out of scope; tonight's line in the sand is that
|
||||
fixtures are *recorded*, never synthesized.
|
||||
|
||||
## Concrete next steps once a trigger fires
|
||||
|
||||
1. Land `RubricDimension` / `RubricVerdict` types in `tests/eval/types.ts`
|
||||
as an additive extension (existing fixtures without a `rubric` field
|
||||
continue to work).
|
||||
2. Write `rubric-judge.ts` that takes a parsed fixture + rubric + judge
|
||||
model and returns `RubricVerdict[]`. Keep under 250 lines.
|
||||
3. Extend `eval-runner.ts` to call Tier 1 only after Tier 0 passes,
|
||||
respecting cost cap and cache.
|
||||
4. Add `pnpm eval:accept` CLI for baseline management.
|
||||
5. Migrate one representative fixture to include a rubric block as the
|
||||
first real-world exercise. Do not bulk-convert — one fixture, review
|
||||
the judge's behavior, then scale.
|
||||
|
||||
## Approval gate
|
||||
|
||||
This proposal is not auto-approved. Before implementation, the roundtable
|
||||
(or its equivalent reviewer set) must explicitly acknowledge:
|
||||
|
||||
1. At least one trigger condition has fired.
|
||||
2. The cost cap is acceptable for the current CI throughput.
|
||||
3. Baseline acceptance workflow fits the team's dev loop.
|
||||
|
||||
Without all three, stop at B5 structural judge. It's better to have one
|
||||
tier that works than two tiers that lie.
|
||||
|
||||
## Closing note
|
||||
|
||||
The reason I (Karpathy) pushed for B5 to land tonight *without* this
|
||||
tier is the same reason I'm drafting this proposal immediately: harness
|
||||
scaffolds are forcing functions. Every line of the JSONL contract I
|
||||
nailed down in B5 constrains what B5.1 can cheaply do. Writing the B5.1
|
||||
proposal the same night forces me to confirm the B5 schema is
|
||||
extensible in the directions I'll actually need — and it is
|
||||
(`assertions` can grow a `rubric` sibling without any breaking change).
|
||||
|
||||
Ship the assertion judge. Record 10 fixtures. Then grade them.
|
||||
Reference in new issue
Block a user