ZacharyZhang-NY
17ffec7ea5
Closes review asks from cross-review of C4: - Jobs ask #2: team-building.en.md still referenced the bash+fetch pattern for goal creation. Rewritten as a deferred-tool note pointing to C5 follow-up, without re-teaching the HTTP shape C4 just removed. - Karpathy ask #1 (partial): re-recorded manager-task-handoff against main to produce post-C4 positive evidence. VP Engineering has hire_agent available via canRequestHire, but when directed NOT to hire, the recorded trace shows task decomposition only (no hire_agent invocation). 455 in / 769 out tokens; assertions updated to pin /api/task/create + agent_ic_ slugs + p99 substring + max 3 turns. The `non-ceo-dm-reply` re-record is deferred to the follow-up ticket. - Karpathy asks #2, #3, #4 + Jobs asks #1, #3, #4: documented in docs/proposals/C4-followup.md with concrete completion criteria. Scope covers canFileCeoStrategy predicate, non-ceo-dm-reply re-record, RFC 8594 Deprecation/Sunset headers + counter, and a negative substring assertion kind for the eval harness. Eval still green at 8/8. C4 main batch (e2c839b..8e99e01) stands unchanged; this commit is pure follow-up groundwork.
Eval fixtures
Every file in this directory is a recorded trace of a real AgentCompany turn against a real LLM provider. Fixtures are recorded-truth, not synthesized mock data — the distinction is what keeps the B5 harness on the "fixture, not fallback" side of the roundtable line.
File format
Each fixture is a .jsonl file with one JSON object per line:
- First line: a
{ "type": "meta", ... }record (seeFixtureMetain../types.ts). - Middle lines: one
{ "type": "turn", "turnIndex": N, ... }record per conversation turn, in order.turnIndexis strictly increasing from0. - Last line: a
{ "type": "assertions", "assertions": [...] }record with the expected behavior contract.
The loader rejects files that do not match this shape. No silent skipping.
Recording workflow
pnpm eval:record -- \
--fixture-id ceo-team-building-empty \
--scenario "Empty company, CEO bootstrap" \
--out tests/eval/fixtures/ceo-team-building-empty.jsonl \
--system-prompt-preview "$(cat /tmp/current-system-prompt.txt)"
The current CLI writes a stub meta + empty assertions. The full recorder
(which drives pi-agent-core against a real provider using the
AgentCompany worker's system prompt and tool registry) lands in a
follow-up commit — see ../README.md for the B5 roadmap.
Committing fixtures
Fixtures belong in git. They are small, deterministic, and the whole point
of the eval tier is to diff behavior against recorded truth. Do NOT add
tests/eval/fixtures/ to .gitignore.
Fixtures are excluded from the electron-builder production bundle; see
electron-builder.yml (the files glob only picks out/**, so tests/
never enters the packaged .asar).