ZacharyZhang-NY
3ff891c6ec
Closes the second and final ask from the C4 cross-review (Karpathy specific ask #1). The previous non-ceo-dm-reply recording (409bec2) pre-dated C4 part 3/5 by ~18 minutes, so it could not discriminate between "non-CEO path correctly does not receive hire_agent / file_approval" and "the tools simply had not been injected yet." Re-recorded against anthropic/claude-sonnet-4-5 on the current main. The VP Product agent (manager — canRequestHire=true by keyword match, but after F3 canFileCeoStrategy=false because reportsTo is not null AND role is not CEO) produces: - mark_read + /api/agent_message/send + m_77 preserved — 3 positive assertions intact. - Zero hire_agent invocation (not asked to hire). - Zero file_approval invocation — positive evidence the F3 split correctly excludes file_approval from non-CEO managers. Without F3 this call would have had file_approval in scope and a manager under pressure might have reached for it. Fixture assertions updated to pin the two tool-scope invariants using the new message_does_not_contain_substring kind from F2. C4-followup.md marked CLOSED with resolution table mapping each of the four items to its landing commit, plus the companion cleanup (43337e5— RpcMethod single-source-of-truth) that came out of the same session. Final: typecheck green, 537 unit tests, 10 eval tests.
Eval fixtures
Every file in this directory is a recorded trace of a real AgentCompany turn against a real LLM provider. Fixtures are recorded-truth, not synthesized mock data — the distinction is what keeps the B5 harness on the "fixture, not fallback" side of the roundtable line.
File format
Each fixture is a .jsonl file with one JSON object per line:
- First line: a
{ "type": "meta", ... }record (seeFixtureMetain../types.ts). - Middle lines: one
{ "type": "turn", "turnIndex": N, ... }record per conversation turn, in order.turnIndexis strictly increasing from0. - Last line: a
{ "type": "assertions", "assertions": [...] }record with the expected behavior contract.
The loader rejects files that do not match this shape. No silent skipping.
Recording workflow
pnpm eval:record -- \
--fixture-id ceo-team-building-empty \
--scenario "Empty company, CEO bootstrap" \
--out tests/eval/fixtures/ceo-team-building-empty.jsonl \
--system-prompt-preview "$(cat /tmp/current-system-prompt.txt)"
The current CLI writes a stub meta + empty assertions. The full recorder
(which drives pi-agent-core against a real provider using the
AgentCompany worker's system prompt and tool registry) lands in a
follow-up commit — see ../README.md for the B5 roadmap.
Committing fixtures
Fixtures belong in git. They are small, deterministic, and the whole point
of the eval tier is to diff behavior against recorded truth. Do NOT add
tests/eval/fixtures/ to .gitignore.
Fixtures are excluded from the electron-builder production bundle; see
electron-builder.yml (the files glob only picks out/**, so tests/
never enters the packaged .asar).