Closes the fourth ask from the C4 cross-review (docs/proposals/C4-followup.md
Item 4). The eval harness previously had no way to assert the absence of a
pattern in a trace — only positive `message_contains_substring`. That meant
the C4 "no more node -e / bash+fetch from CEO" invariant was only
demonstrated by the diff between two commits, not by any running assertion
that future regressions would trip.
Three changes:
1. Extend the Assertion union in tests/eval/types.ts with
`message_does_not_contain_substring` (role + substring). Symmetric to
the positive kind; fails on first match with a specific reason.
2. Add the case in assertion-judge.ts. Same traversal as the positive
branch but inverted exit condition.
3. Add two harness self-tests covering pass + fail, so the new kind is
exercised even when the fixtures directory is empty.
4. Pin two C4-regression assertions onto the recorded ceo-bootstrap-empty
fixture: the assistant response must NOT contain "node -e" OR
"/api/companies". Any future regression that reintroduces the
HTTP-path-in-prompt pattern or the bash+node -e shell fallback will
now fail pnpm test:eval instead of living in the fixture silently.
typecheck green, test:eval up from 8 → 10 tests (2 new self-tests), all
passing.
Land the B5 harness skeleton: fixture loader (strict JSONL contract with
meta header + ordered turn records + assertions tail), replay-client that
swaps in for pi-ai's streamSimple by replaying recorded assistant turns,
structural assertion judge (tool_called / no_tool_called /
message_contains_substring / tool_call_arg_matches / trace_max_turns), an
eval-runner that sums the token ledger and produces an EvalResult, a
vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl
fixtures, and a record-fixture CLI that writes fixture meta headers.
Hard contracts:
- No fallback: replay-client and fixture-loader throw on exhaustion /
malformed input rather than synthesizing defaults.
- No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall
verbatim; shadow types are forbidden.
- No fake fixture data: tests/eval/fixtures/ ships only README +
.gitkeep. Real traces are recorded by the user against live providers.
package.json adds test:eval (vitest --passWithNoTests so an empty
fixture dir is green) and eval:record (CLI entrypoint). electron-builder
already excludes tests/ via its files: glob — fixtures never enter the
shipped .asar.
ZacharyZhang-NY