ZacharyZhang-NY
e7f879bed0
Land the B5 harness skeleton: fixture loader (strict JSONL contract with meta header + ordered turn records + assertions tail), replay-client that swaps in for pi-ai's streamSimple by replaying recorded assistant turns, structural assertion judge (tool_called / no_tool_called / message_contains_substring / tool_call_arg_matches / trace_max_turns), an eval-runner that sums the token ledger and produces an EvalResult, a vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl fixtures, and a record-fixture CLI that writes fixture meta headers. Hard contracts: - No fallback: replay-client and fixture-loader throw on exhaustion / malformed input rather than synthesizing defaults. - No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall verbatim; shadow types are forbidden. - No fake fixture data: tests/eval/fixtures/ ships only README + .gitkeep. Real traces are recorded by the user against live providers. package.json adds test:eval (vitest --passWithNoTests so an empty fixture dir is green) and eval:record (CLI entrypoint). electron-builder already excludes tests/ via its files: glob — fixtures never enter the shipped .asar.
45 lines
1.7 KiB
Markdown
45 lines
1.7 KiB
Markdown
# Eval fixtures
|
|
|
|
Every file in this directory is a **recorded trace** of a real AgentCompany
|
|
turn against a real LLM provider. Fixtures are recorded-truth, not
|
|
synthesized mock data — the distinction is what keeps the B5 harness on the
|
|
"fixture, not fallback" side of the roundtable line.
|
|
|
|
## File format
|
|
|
|
Each fixture is a `.jsonl` file with one JSON object per line:
|
|
|
|
1. First line: a `{ "type": "meta", ... }` record (see `FixtureMeta` in
|
|
`../types.ts`).
|
|
2. Middle lines: one `{ "type": "turn", "turnIndex": N, ... }` record per
|
|
conversation turn, in order. `turnIndex` is strictly increasing from `0`.
|
|
3. Last line: a `{ "type": "assertions", "assertions": [...] }` record with
|
|
the expected behavior contract.
|
|
|
|
The loader rejects files that do not match this shape. No silent skipping.
|
|
|
|
## Recording workflow
|
|
|
|
```bash
|
|
pnpm eval:record -- \
|
|
--fixture-id ceo-team-building-empty \
|
|
--scenario "Empty company, CEO bootstrap" \
|
|
--out tests/eval/fixtures/ceo-team-building-empty.jsonl \
|
|
--system-prompt-preview "$(cat /tmp/current-system-prompt.txt)"
|
|
```
|
|
|
|
The current CLI writes a stub meta + empty assertions. The full recorder
|
|
(which drives `pi-agent-core` against a real provider using the
|
|
AgentCompany worker's system prompt and tool registry) lands in a
|
|
follow-up commit — see `../README.md` for the B5 roadmap.
|
|
|
|
## Committing fixtures
|
|
|
|
Fixtures belong in git. They are small, deterministic, and the whole point
|
|
of the eval tier is to diff behavior against recorded truth. Do NOT add
|
|
`tests/eval/fixtures/` to `.gitignore`.
|
|
|
|
Fixtures are excluded from the electron-builder production bundle; see
|
|
`electron-builder.yml` (the `files` glob only picks `out/**`, so `tests/`
|
|
never enters the packaged `.asar`).
|