AgentCompany/docs/proposals/B5.1-llm-as-judge-followup.md

B5.1 — LLM-as-judge tier above the B5 assertion harness

Status: phase 1 executed 2026-04-23; provider LLM judge remains proposal behind the triggers defined below. Depends on: B5 (assertion harness scaffold, landed 2026-04-20) Author note: roundtable 2-1 decision to defer this tonight; scaffold first, semantic judgment after N ≥ 10 fixtures exist.

TL;DR

B5 handles structural assertions against a recorded trace — tool name, call count, call order, argument shape, turn cap, text substring. The first B5.1 layer now adds deterministic semanticRubric dimensions to every fixture: each dimension states the semantic quality being protected and ties it to required / forbidden assistant-transcript evidence. Provider LLM-as-judge is the next tier on top of the same fixtures, gated by a rubric, a cost budget, and a failure protocol.

This proposal defines the trigger conditions for building the provider tier, the concrete rubric schema, the assertion-first fast path, the cost cap, and the failure mode when the judge itself is wrong.

2026-04-23 executed layer

tests/eval now includes a semantic rubric gate:

• types.ts requires AssertionsBlock.semanticRubric.
• fixture-loader.ts rejects fixtures with missing or empty semantic rubrics.
• semantic-rubric-judge.ts evaluates each dimension against assistant transcript evidence.
• eval-runner.ts folds semantic rubric failures into EvalResult.passed.
• Existing recorded fixtures now cover CEO bootstrap, Board reply, manager handoff, peer reply, and IC task execution with semantic dimensions.
• The prompt matrix now pins the AI Co-Founder human-review mode contract.

This layer runs in pnpm test:eval and pnpm verify. It is deterministic, provider-key-free, and tied to real recorded traces.

Why not tonight (roundtable reasoning)

Three dissenting voices carried the delay decision 2-1:

Linus — "You don't bolt a second LLM onto an eval harness that has zero fixtures. Build the bone first. Bolt the muscle when you have something to measure." Structural assertions work with 1 fixture. An LLM-judge tier is pointless below ~10 fixtures because you can't distinguish judge error from fixture sparseness.

Jobs — "Single signal source. Tonight the developer needs one needle: 'did the trace match the recorded shape, yes or no.' A judge introduces a second signal (confidence, rubric score) that dilutes focus on the first bug we haven't found yet."

Karpathy (me) — agreed on the delay because the assertion tier is the forcing function that reveals which behaviors actually matter. The fixtures I record in the next two weeks will tell me exactly which rubric dimensions the judge should grade. Building the judge rubric from a blank page today is premature optimization.

Dissent (minority) — the argument for shipping tonight was that CEO-mode bootstrap behavior is qualitative enough that assertions alone miss the point of B5. The counter: if a behavior can only be judged semantically, that's a signal the behavior spec itself is underdetermined — fix the spec before fixing the eval tier.

Trigger conditions (do this when any of the below holds)

Condition	Why it triggers B5.1
tests/eval/fixtures/ has 10+ recorded fixtures	Enough surface to distinguish rubric gaps from fixture gaps
Two separate production incidents trace back to a behavior that passed all assertions but was still wrong	Signal that structural checks miss semantic drift
A fixture exists where the desired assertion cannot be written as a structural match (example: "CEO should not sound defensive")	Direct evidence structural assertions are out of scope for this case
Quarterly review: B5 has caught zero regressions across 50+ fixture runs	Suggests structural assertions are over-fitting to already-green shapes — add semantic judgment before assuming coverage is adequate

Any one condition is sufficient. Do not build B5.1 speculatively on a "maybe we'll need it" basis — build it the moment one concrete trigger fires, and tie the PR to that trigger in the commit message.

Tier architecture

Tier 0 — fixture loader + structural judge (B5, landed)

The cheap tier. O(trace_length) time, zero network calls. Runs in vitest under pnpm test:eval. Every fixture must go through this tier; failures here short-circuit Tier 1 because semantic judgment on a structurally broken trace is nonsense.

Tier 1 — LLM rubric judge (this proposal)

Runs only if Tier 0 passes. Consults an LLM with:

1. The system prompt used during recording (retrievable via the systemPromptHash in meta).
2. The recorded transcript (user / assistant / toolResult messages in order).
3. A rubric — a structured list of yes/no grading dimensions unique to this fixture (stored as part of the fixture's assertions block under a new rubric field, see schema below).
4. A maximum token budget for the judge call.

The judge returns a JSON verdict: { dimension, passed, reason }[]. The runner aggregates these into the EvalResult alongside structural assertions.

Rubric schema (proposed, additive to B5 types)

export interface RubricDimension {
  /** Short kebab-case identifier stable across fixture revisions. */
  id: string;
  /** Human-readable question the judge answers yes/no. */
  question: string;
  /** Required: concrete phrasing of what a pass looks like. */
  passRubric: string;
  /** Required: concrete phrasing of what a fail looks like. */
  failRubric: string;
}

export interface RubricVerdict {
  dimensionId: string;
  passed: boolean;
  reason: string;
  judgeModel: string;
  judgeTokensUsed: { input: number; output: number };
}

The rubric lives inline in the fixture's assertions block as { "rubric": [...] }. The existing structural assertions array stays. Backwards-compatible — fixtures without a rubric skip Tier 1 entirely.

Thin call shape (stay cheap)

One call per fixture. Not one call per dimension. The prompt enumerates all dimensions and asks for a single structured JSON response covering them all. This keeps per-fixture cost to one round trip.

System: You are a rubric judge. For each dimension, respond yes or no
with a one-sentence reason. Never repeat the rubric. Never invent
dimensions.

User:
[transcript verbatim]

Dimensions to grade:
1. <id>: <question>
   Pass if: <passRubric>
   Fail if: <failRubric>

Respond as JSON: [{"dimensionId": "...", "passed": true|false, "reason": "..."}]

Cost cap

Hard cap per pnpm test:eval run: $2.00 of judge cost. Implementation:

1. Configure a fixed model (recommend claude-sonnet class — cheap enough for high-volume judge calls, smart enough for rubric reasoning).
2. Runner tracks cumulative judgeTokensUsed × model.cost across fixtures in a run.
3. Once the cap is hit, remaining fixtures run Tier 0 only and their Tier 1 results record skipped: true, reason: "cost cap exceeded".
4. The run exits non-zero if cost-cap skip would have masked a prior Tier 0 pass — i.e. the run fails loud when skipping would hide a genuine regression.

Rationale: an eval tier that can silently cost $100 when a fixture balloons is not an eval tier, it's a budget exfil vector.

Fast-path escape hatch

Tier 1 runs only when at least one of the fixture's rubric dimensions differs from the previous eval snapshot. The runner keeps a local SHA-keyed cache (fixtureId, rubric hash, transcript hash) → verdict. Identical inputs skip the LLM call and reuse the cached verdict.

Rationale: if the CEO's trace hasn't changed and the rubric hasn't changed, the verdict won't change either. Running it again is pure waste. First-run of a new fixture or rubric still pays the call.

Failure modes (things that will actually go wrong)

The judge is wrong

The judge's own failure mode is non-trivial: it can mark a good trace as failing (false positive) or a bad trace as passing (false negative). Mitigations:

1. Every Tier 1 failure requires a human sign-off to baseline. The first time a rubric dimension flips from pass to fail, the runner does not treat the run as failed — it prints the verdict diff and asks the developer to promote or reject it via a pnpm eval:accept --fixture <id> --dimension <id> command. After acceptance, the baseline is updated and future matching verdicts pass silently.
2. Judge disagreement logging. When Tier 1 fails but Tier 0 passes, the reason text is preserved in the fixture's .accepted.jsonl sidecar for later review.

The rubric drifts out of date

Fixtures grow rubric dimensions, rubrics get more verbose, the judge's context window fills up. Mitigations:

1. Per-fixture cap: no more than 8 rubric dimensions. Enforced at load time — a 9th dimension is a load error, forcing the developer to either split the fixture or prune rubrics.
2. Rubric dimensions must be single-sentence pass/fail rules. Long prose rubrics are compressed or rejected at load.

Provider outage blocks CI

Mitigation: if the judge provider is unreachable, Tier 1 verdicts record skipped: true, reason: "provider unreachable" and the run exits with a warning but does not fail the CI build. Tier 0 still gates the run. The judge is a supplement, not a must-pass gate, for exactly this reason.

Explicit non-goals

• Not a teacher for the CEO agent. Tier 1 does not produce training data, does not feed back into prompts, does not auto-patch behavior. It grades; humans act on grades.
• Not a replacement for structural assertions. Tier 0 runs always, gates Tier 1, and handles 80%+ of regressions. If you find yourself deleting Tier 0 assertions because Tier 1 covers them, stop — the structural tier is cheaper and faster and reveals different bugs.
• Not a judge that rewrites fixtures. The judge reads; it does not author rubrics, does not select which dimensions to grade, does not modify the .jsonl on disk.

What this proposal does not cover

• Multi-turn "soft" evaluation (did the agent converge on the right answer across multiple turns?) — future tier, call it B5.2.
• Cross-fixture trend analysis (is the CEO getting more verbose over time?) — future tier.
• Adversarial fixture generation (synthesizing fixtures that probe edge cases) — explicitly out of scope; tonight's line in the sand is that fixtures are recorded, never synthesized.

Concrete next steps once a trigger fires

1. Land RubricDimension / RubricVerdict types in tests/eval/types.ts as an additive extension (existing fixtures without a rubric field continue to work).
2. Write rubric-judge.ts that takes a parsed fixture + rubric + judge model and returns RubricVerdict[]. Keep under 250 lines.
3. Extend eval-runner.ts to call Tier 1 only after Tier 0 passes, respecting cost cap and cache.
4. Add pnpm eval:accept CLI for baseline management.
5. Migrate one representative fixture to include a rubric block as the first real-world exercise. Do not bulk-convert — one fixture, review the judge's behavior, then scale.

Approval gate

This proposal is not auto-approved. Before implementation, the roundtable (or its equivalent reviewer set) must explicitly acknowledge:

1. At least one trigger condition has fired.
2. The cost cap is acceptable for the current CI throughput.
3. Baseline acceptance workflow fits the team's dev loop.

Without all three, stop at B5 structural judge. It's better to have one tier that works than two tiers that lie.

Closing note

The reason I (Karpathy) pushed for B5 to land tonight without this tier is the same reason I'm drafting this proposal immediately: harness scaffolds are forcing functions. Every line of the JSONL contract I nailed down in B5 constrains what B5.1 can cheaply do. Writing the B5.1 proposal the same night forces me to confirm the B5 schema is extensible in the directions I'll actually need — and it is (assertions can grow a rubric sibling without any breaking change).

Ship the assertion judge. Record 10 fixtures. Then grade them.