ZacharyZhang-NY
1d0cfa29d2
Unblocks the march-of-nines third nine. Up until this commit `tests/eval/`
had a harness + self-tests but zero real traces; the fixture directory was
a documented stub. These two fixtures are genuine single-turn recordings
against anthropic/claude-sonnet-4-5 produced by `pnpm eval:record` — not
mock data, not placeholder — each one captures an actual API response
with its token ledger.
Coverage:
- ceo-bootstrap-empty: empty company, CEO bootstrap trigger. Asserts the
model produces hire_agent POSTs, an approve_ceo_strategy POST, and uses
bash+fetch (current prompt pattern before C4 lands).
- ceo-board-message-reply: operating CEO with a live team, inbound Board
message. Asserts the model hits mark_read, posts to
/api/agent_message/send, and keeps the original message_id in scope.
What these fixtures now enable:
- B2 Phase 2 switch-read: a fixture diff before/after the cutover
demonstrates dual-write parity is real, not assumed.
- C4 initial-user-turn refactor: the current HTTP-path-in-prompt pattern
is captured; after C4 lands, the diff shows whether tool-shape
refactoring regressed behaviour.
- C7 prompt-builder conditional assembly: same, but for system-prompt
slimming.
- M7 Phase 2 call-site switch: resolveCeoMode cutover can be regression-
verified against both traces.
Scenarios live at tests/eval/scenarios/<id>/{system.txt,user.txt} so any
maintainer can re-record the same trace with one command and diff against
the committed fixture. Each fixture's assertions block currently contains
four assertions; pnpm test:eval passes with 5 (3 harness self-tests + 2
real-fixture runs).
Eval fixtures
Every file in this directory is a recorded trace of a real AgentCompany turn against a real LLM provider. Fixtures are recorded-truth, not synthesized mock data — the distinction is what keeps the B5 harness on the "fixture, not fallback" side of the roundtable line.
File format
Each fixture is a .jsonl file with one JSON object per line:
- First line: a
{ "type": "meta", ... }record (seeFixtureMetain../types.ts). - Middle lines: one
{ "type": "turn", "turnIndex": N, ... }record per conversation turn, in order.turnIndexis strictly increasing from0. - Last line: a
{ "type": "assertions", "assertions": [...] }record with the expected behavior contract.
The loader rejects files that do not match this shape. No silent skipping.
Recording workflow
pnpm eval:record -- \
--fixture-id ceo-team-building-empty \
--scenario "Empty company, CEO bootstrap" \
--out tests/eval/fixtures/ceo-team-building-empty.jsonl \
--system-prompt-preview "$(cat /tmp/current-system-prompt.txt)"
The current CLI writes a stub meta + empty assertions. The full recorder
(which drives pi-agent-core against a real provider using the
AgentCompany worker's system prompt and tool registry) lands in a
follow-up commit — see ../README.md for the B5 roadmap.
Committing fixtures
Fixtures belong in git. They are small, deterministic, and the whole point
of the eval tier is to diff behavior against recorded truth. Do NOT add
tests/eval/fixtures/ to .gitignore.
Fixtures are excluded from the electron-builder production bundle; see
electron-builder.yml (the files glob only picks out/**, so tests/
never enters the packaged .asar).