ZacharyZhang-NY
9a5823b242
Closes the fourth ask from the C4 cross-review (docs/proposals/C4-followup.md Item 4). The eval harness previously had no way to assert the absence of a pattern in a trace — only positive `message_contains_substring`. That meant the C4 "no more node -e / bash+fetch from CEO" invariant was only demonstrated by the diff between two commits, not by any running assertion that future regressions would trip. Three changes: 1. Extend the Assertion union in tests/eval/types.ts with `message_does_not_contain_substring` (role + substring). Symmetric to the positive kind; fails on first match with a specific reason. 2. Add the case in assertion-judge.ts. Same traversal as the positive branch but inverted exit condition. 3. Add two harness self-tests covering pass + fail, so the new kind is exercised even when the fixtures directory is empty. 4. Pin two C4-regression assertions onto the recorded ceo-bootstrap-empty fixture: the assistant response must NOT contain "node -e" OR "/api/companies". Any future regression that reintroduces the HTTP-path-in-prompt pattern or the bash+node -e shell fallback will now fail pnpm test:eval instead of living in the fixture silently. typecheck green, test:eval up from 8 → 10 tests (2 new self-tests), all passing.
Eval fixtures
Every file in this directory is a recorded trace of a real AgentCompany turn against a real LLM provider. Fixtures are recorded-truth, not synthesized mock data — the distinction is what keeps the B5 harness on the "fixture, not fallback" side of the roundtable line.
File format
Each fixture is a .jsonl file with one JSON object per line:
- First line: a
{ "type": "meta", ... }record (seeFixtureMetain../types.ts). - Middle lines: one
{ "type": "turn", "turnIndex": N, ... }record per conversation turn, in order.turnIndexis strictly increasing from0. - Last line: a
{ "type": "assertions", "assertions": [...] }record with the expected behavior contract.
The loader rejects files that do not match this shape. No silent skipping.
Recording workflow
pnpm eval:record -- \
--fixture-id ceo-team-building-empty \
--scenario "Empty company, CEO bootstrap" \
--out tests/eval/fixtures/ceo-team-building-empty.jsonl \
--system-prompt-preview "$(cat /tmp/current-system-prompt.txt)"
The current CLI writes a stub meta + empty assertions. The full recorder
(which drives pi-agent-core against a real provider using the
AgentCompany worker's system prompt and tool registry) lands in a
follow-up commit — see ../README.md for the B5 roadmap.
Committing fixtures
Fixtures belong in git. They are small, deterministic, and the whole point
of the eval tier is to diff behavior against recorded truth. Do NOT add
tests/eval/fixtures/ to .gitignore.
Fixtures are excluded from the electron-builder production bundle; see
electron-builder.yml (the files glob only picks out/**, so tests/
never enters the packaged .asar).