ZacharyZhang-NY
8e99e0119f
This is the verification artefact for the C4 refactor (e2c839bthrough03864f3). The old fixture (recorded at1d0cfa2, scenario asked the model to POST hire_agent requests via bash+node+fetch) captured the model emitting `node -e 'fetch(\`http://...\`)'` — exactly the antipattern Jobs called out. Post-C4 scenario was updated to reflect the new world: the CEO now has explicit hire_agent and file_approval tools. Re-recording against anthropic/claude-sonnet-4-5 produced 509 in / 731 out tokens and the assistant response: - Contains `hire_agent` (pseudo-XML tool invocations, one per hire) - Contains `file_approval` with type=approve_ceo_strategy - No `node -e` / bash+fetch / HTTP POST pattern anywhere - 3-5 hires as requested, single approval block Assertions updated to pin the post-C4 tool names as positive signals. Absence of bash/fetch is demonstrated by diff against1d0cfa2(see that commit's recorded JSONL for the before state). Eval harness now green at 8/8 (3 self-tests + 5 recorded fixtures) with this new baseline. Any future regression that reverts CEO bootstrap to HTTP-path prompts will fail all three hire_agent / approve_ceo_strategy / file_approval assertions.
Eval fixtures
Every file in this directory is a recorded trace of a real AgentCompany turn against a real LLM provider. Fixtures are recorded-truth, not synthesized mock data — the distinction is what keeps the B5 harness on the "fixture, not fallback" side of the roundtable line.
File format
Each fixture is a .jsonl file with one JSON object per line:
- First line: a
{ "type": "meta", ... }record (seeFixtureMetain../types.ts). - Middle lines: one
{ "type": "turn", "turnIndex": N, ... }record per conversation turn, in order.turnIndexis strictly increasing from0. - Last line: a
{ "type": "assertions", "assertions": [...] }record with the expected behavior contract.
The loader rejects files that do not match this shape. No silent skipping.
Recording workflow
pnpm eval:record -- \
--fixture-id ceo-team-building-empty \
--scenario "Empty company, CEO bootstrap" \
--out tests/eval/fixtures/ceo-team-building-empty.jsonl \
--system-prompt-preview "$(cat /tmp/current-system-prompt.txt)"
The current CLI writes a stub meta + empty assertions. The full recorder
(which drives pi-agent-core against a real provider using the
AgentCompany worker's system prompt and tool registry) lands in a
follow-up commit — see ../README.md for the B5 roadmap.
Committing fixtures
Fixtures belong in git. They are small, deterministic, and the whole point
of the eval tier is to diff behavior against recorded truth. Do NOT add
tests/eval/fixtures/ to .gitignore.
Fixtures are excluded from the electron-builder production bundle; see
electron-builder.yml (the files glob only picks out/**, so tests/
never enters the packaged .asar).