278 lines
12 KiB
Markdown
278 lines
12 KiB
Markdown
# B5.1 — LLM-as-judge tier above the B5 assertion harness
|
||
|
||
> Status: phase 1 executed 2026-04-23; provider LLM judge remains proposal
|
||
> behind the triggers defined below.
|
||
> Depends on: B5 (assertion harness scaffold, landed 2026-04-20)
|
||
> Author note: roundtable 2-1 decision to defer this tonight; scaffold first,
|
||
> semantic judgment after N ≥ 10 fixtures exist.
|
||
|
||
## TL;DR
|
||
|
||
B5 handles **structural** assertions against a recorded trace — tool name,
|
||
call count, call order, argument shape, turn cap, text substring. The first
|
||
B5.1 layer now adds deterministic `semanticRubric` dimensions to every
|
||
fixture: each dimension states the semantic quality being protected and ties it
|
||
to required / forbidden assistant-transcript evidence. Provider LLM-as-judge is
|
||
the next tier on top of the same fixtures, gated by a rubric, a cost budget,
|
||
and a failure protocol.
|
||
|
||
This proposal defines the trigger conditions for building the provider tier, the
|
||
concrete rubric schema, the assertion-first fast path, the cost cap, and
|
||
the failure mode when the judge itself is wrong.
|
||
|
||
## 2026-04-23 executed layer
|
||
|
||
`tests/eval` now includes a semantic rubric gate:
|
||
|
||
- `types.ts` requires `AssertionsBlock.semanticRubric`.
|
||
- `fixture-loader.ts` rejects fixtures with missing or empty semantic rubrics.
|
||
- `semantic-rubric-judge.ts` evaluates each dimension against assistant
|
||
transcript evidence.
|
||
- `eval-runner.ts` folds semantic rubric failures into `EvalResult.passed`.
|
||
- Existing recorded fixtures now cover CEO bootstrap, Board reply, manager
|
||
handoff, peer reply, and IC task execution with semantic dimensions.
|
||
- The prompt matrix now pins the AI Co-Founder human-review mode contract.
|
||
|
||
This layer runs in `pnpm test:eval` and `pnpm verify`. It is deterministic,
|
||
provider-key-free, and tied to real recorded traces.
|
||
|
||
## Why not tonight (roundtable reasoning)
|
||
|
||
Three dissenting voices carried the delay decision 2-1:
|
||
|
||
**Linus** — *"You don't bolt a second LLM onto an eval harness that has
|
||
zero fixtures. Build the bone first. Bolt the muscle when you have
|
||
something to measure."* Structural assertions work with 1 fixture. An
|
||
LLM-judge tier is pointless below ~10 fixtures because you can't
|
||
distinguish judge error from fixture sparseness.
|
||
|
||
**Jobs** — *"Single signal source. Tonight the developer needs one
|
||
needle: 'did the trace match the recorded shape, yes or no.' A judge
|
||
introduces a second signal (confidence, rubric score) that dilutes
|
||
focus on the first bug we haven't found yet."*
|
||
|
||
**Karpathy (me)** — agreed on the delay because the assertion tier *is*
|
||
the forcing function that reveals which behaviors actually matter. The
|
||
fixtures I record in the next two weeks will tell me exactly which
|
||
rubric dimensions the judge should grade. Building the judge rubric from
|
||
a blank page today is premature optimization.
|
||
|
||
**Dissent (minority)** — the argument for shipping tonight was that
|
||
CEO-mode bootstrap behavior is qualitative enough that assertions alone
|
||
miss the point of B5. The counter: if a behavior can only be judged
|
||
semantically, that's a signal the behavior spec itself is underdetermined
|
||
— fix the spec before fixing the eval tier.
|
||
|
||
## Trigger conditions (do this when any of the below holds)
|
||
|
||
| Condition | Why it triggers B5.1 |
|
||
|---|---|
|
||
| `tests/eval/fixtures/` has 10+ recorded fixtures | Enough surface to distinguish rubric gaps from fixture gaps |
|
||
| Two separate production incidents trace back to a behavior that passed all assertions but was still wrong | Signal that structural checks miss semantic drift |
|
||
| A fixture exists where the desired assertion cannot be written as a structural match (example: "CEO should not sound defensive") | Direct evidence structural assertions are out of scope for this case |
|
||
| Quarterly review: B5 has caught zero regressions across 50+ fixture runs | Suggests structural assertions are over-fitting to already-green shapes — add semantic judgment before assuming coverage is adequate |
|
||
|
||
Any one condition is sufficient. Do not build B5.1 speculatively on a
|
||
"maybe we'll need it" basis — build it the moment one concrete trigger
|
||
fires, and tie the PR to that trigger in the commit message.
|
||
|
||
## Tier architecture
|
||
|
||
### Tier 0 — fixture loader + structural judge (B5, landed)
|
||
|
||
The cheap tier. `O(trace_length)` time, zero network calls. Runs in
|
||
`vitest` under `pnpm test:eval`. Every fixture must go through this
|
||
tier; failures here short-circuit Tier 1 because semantic judgment on a
|
||
structurally broken trace is nonsense.
|
||
|
||
### Tier 1 — LLM rubric judge (this proposal)
|
||
|
||
Runs only if Tier 0 passes. Consults an LLM with:
|
||
|
||
1. The system prompt used during recording (retrievable via the
|
||
`systemPromptHash` in `meta`).
|
||
2. The recorded transcript (user / assistant / toolResult messages in
|
||
order).
|
||
3. A **rubric** — a structured list of yes/no grading dimensions unique
|
||
to this fixture (stored as part of the fixture's `assertions` block
|
||
under a new `rubric` field, see schema below).
|
||
4. A maximum token budget for the judge call.
|
||
|
||
The judge returns a JSON verdict: `{ dimension, passed, reason }[]`. The
|
||
runner aggregates these into the `EvalResult` alongside structural
|
||
assertions.
|
||
|
||
### Rubric schema (proposed, additive to B5 types)
|
||
|
||
```ts
|
||
export interface RubricDimension {
|
||
/** Short kebab-case identifier stable across fixture revisions. */
|
||
id: string;
|
||
/** Human-readable question the judge answers yes/no. */
|
||
question: string;
|
||
/** Required: concrete phrasing of what a pass looks like. */
|
||
passRubric: string;
|
||
/** Required: concrete phrasing of what a fail looks like. */
|
||
failRubric: string;
|
||
}
|
||
|
||
export interface RubricVerdict {
|
||
dimensionId: string;
|
||
passed: boolean;
|
||
reason: string;
|
||
judgeModel: string;
|
||
judgeTokensUsed: { input: number; output: number };
|
||
}
|
||
```
|
||
|
||
The rubric lives inline in the fixture's assertions block as
|
||
`{ "rubric": [...] }`. The existing structural `assertions` array stays.
|
||
Backwards-compatible — fixtures without a rubric skip Tier 1 entirely.
|
||
|
||
### Thin call shape (stay cheap)
|
||
|
||
One call per fixture. Not one call per dimension. The prompt enumerates
|
||
all dimensions and asks for a single structured JSON response covering
|
||
them all. This keeps per-fixture cost to one round trip.
|
||
|
||
```
|
||
System: You are a rubric judge. For each dimension, respond yes or no
|
||
with a one-sentence reason. Never repeat the rubric. Never invent
|
||
dimensions.
|
||
|
||
User:
|
||
[transcript verbatim]
|
||
|
||
Dimensions to grade:
|
||
1. <id>: <question>
|
||
Pass if: <passRubric>
|
||
Fail if: <failRubric>
|
||
|
||
Respond as JSON: [{"dimensionId": "...", "passed": true|false, "reason": "..."}]
|
||
```
|
||
|
||
### Cost cap
|
||
|
||
Hard cap per `pnpm test:eval` run: **$2.00 of judge cost**. Implementation:
|
||
|
||
1. Configure a fixed model (recommend `claude-sonnet` class — cheap
|
||
enough for high-volume judge calls, smart enough for rubric reasoning).
|
||
2. Runner tracks cumulative `judgeTokensUsed × model.cost` across
|
||
fixtures in a run.
|
||
3. Once the cap is hit, remaining fixtures run Tier 0 only and their
|
||
Tier 1 results record `skipped: true, reason: "cost cap exceeded"`.
|
||
4. The run exits non-zero if cost-cap skip would have masked a prior
|
||
Tier 0 pass — i.e. the run fails loud when skipping would hide a
|
||
genuine regression.
|
||
|
||
Rationale: an eval tier that can silently cost $100 when a fixture
|
||
balloons is not an eval tier, it's a budget exfil vector.
|
||
|
||
### Fast-path escape hatch
|
||
|
||
Tier 1 runs only when at least one of the fixture's `rubric` dimensions
|
||
differs from the previous eval snapshot. The runner keeps a local
|
||
SHA-keyed cache `(fixtureId, rubric hash, transcript hash) → verdict`.
|
||
Identical inputs skip the LLM call and reuse the cached verdict.
|
||
|
||
Rationale: if the CEO's trace hasn't changed and the rubric hasn't
|
||
changed, the verdict won't change either. Running it again is pure
|
||
waste. First-run of a new fixture or rubric still pays the call.
|
||
|
||
## Failure modes (things that will actually go wrong)
|
||
|
||
### The judge is wrong
|
||
|
||
The judge's own failure mode is non-trivial: it can mark a good trace as
|
||
failing (false positive) or a bad trace as passing (false negative).
|
||
Mitigations:
|
||
|
||
1. **Every Tier 1 failure requires a human sign-off to baseline.** The
|
||
first time a rubric dimension flips from pass to fail, the runner
|
||
does not treat the run as failed — it prints the verdict diff and
|
||
asks the developer to promote or reject it via a `pnpm eval:accept
|
||
--fixture <id> --dimension <id>` command. After acceptance, the
|
||
baseline is updated and future matching verdicts pass silently.
|
||
2. **Judge disagreement logging.** When Tier 1 fails but Tier 0 passes,
|
||
the reason text is preserved in the fixture's `.accepted.jsonl`
|
||
sidecar for later review.
|
||
|
||
### The rubric drifts out of date
|
||
|
||
Fixtures grow rubric dimensions, rubrics get more verbose, the judge's
|
||
context window fills up. Mitigations:
|
||
|
||
1. Per-fixture cap: **no more than 8 rubric dimensions**. Enforced at
|
||
load time — a 9th dimension is a load error, forcing the developer to
|
||
either split the fixture or prune rubrics.
|
||
2. **Rubric dimensions must be single-sentence pass/fail rules.** Long
|
||
prose rubrics are compressed or rejected at load.
|
||
|
||
### Provider outage blocks CI
|
||
|
||
Mitigation: if the judge provider is unreachable, Tier 1 verdicts record
|
||
`skipped: true, reason: "provider unreachable"` and the run exits with a
|
||
warning but **does not fail the CI build**. Tier 0 still gates the run.
|
||
The judge is a supplement, not a must-pass gate, for exactly this reason.
|
||
|
||
## Explicit non-goals
|
||
|
||
- **Not a teacher for the CEO agent.** Tier 1 does not produce training
|
||
data, does not feed back into prompts, does not auto-patch behavior.
|
||
It grades; humans act on grades.
|
||
- **Not a replacement for structural assertions.** Tier 0 runs always,
|
||
gates Tier 1, and handles 80%+ of regressions. If you find yourself
|
||
deleting Tier 0 assertions because Tier 1 covers them, stop — the
|
||
structural tier is cheaper and faster and reveals different bugs.
|
||
- **Not a judge that rewrites fixtures.** The judge reads; it does not
|
||
author rubrics, does not select which dimensions to grade, does not
|
||
modify the `.jsonl` on disk.
|
||
|
||
## What this proposal does not cover
|
||
|
||
- Multi-turn "soft" evaluation (did the agent converge on the right
|
||
answer across multiple turns?) — future tier, call it B5.2.
|
||
- Cross-fixture trend analysis (is the CEO getting more verbose over
|
||
time?) — future tier.
|
||
- Adversarial fixture generation (synthesizing fixtures that probe edge
|
||
cases) — explicitly out of scope; tonight's line in the sand is that
|
||
fixtures are *recorded*, never synthesized.
|
||
|
||
## Concrete next steps once a trigger fires
|
||
|
||
1. Land `RubricDimension` / `RubricVerdict` types in `tests/eval/types.ts`
|
||
as an additive extension (existing fixtures without a `rubric` field
|
||
continue to work).
|
||
2. Write `rubric-judge.ts` that takes a parsed fixture + rubric + judge
|
||
model and returns `RubricVerdict[]`. Keep under 250 lines.
|
||
3. Extend `eval-runner.ts` to call Tier 1 only after Tier 0 passes,
|
||
respecting cost cap and cache.
|
||
4. Add `pnpm eval:accept` CLI for baseline management.
|
||
5. Migrate one representative fixture to include a rubric block as the
|
||
first real-world exercise. Do not bulk-convert — one fixture, review
|
||
the judge's behavior, then scale.
|
||
|
||
## Approval gate
|
||
|
||
This proposal is not auto-approved. Before implementation, the roundtable
|
||
(or its equivalent reviewer set) must explicitly acknowledge:
|
||
|
||
1. At least one trigger condition has fired.
|
||
2. The cost cap is acceptable for the current CI throughput.
|
||
3. Baseline acceptance workflow fits the team's dev loop.
|
||
|
||
Without all three, stop at B5 structural judge. It's better to have one
|
||
tier that works than two tiers that lie.
|
||
|
||
## Closing note
|
||
|
||
The reason I (Karpathy) pushed for B5 to land tonight *without* this
|
||
tier is the same reason I'm drafting this proposal immediately: harness
|
||
scaffolds are forcing functions. Every line of the JSONL contract I
|
||
nailed down in B5 constrains what B5.1 can cheaply do. Writing the B5.1
|
||
proposal the same night forces me to confirm the B5 schema is
|
||
extensible in the directions I'll actually need — and it is
|
||
(`assertions` can grow a `rubric` sibling without any breaking change).
|
||
|
||
Ship the assertion judge. Record 10 fixtures. Then grade them.
|