Build minimal LLM eval harness scaffold (B5) — fixtures recorded separately
Land the B5 harness skeleton: fixture loader (strict JSONL contract with meta header + ordered turn records + assertions tail), replay-client that swaps in for pi-ai's streamSimple by replaying recorded assistant turns, structural assertion judge (tool_called / no_tool_called / message_contains_substring / tool_call_arg_matches / trace_max_turns), an eval-runner that sums the token ledger and produces an EvalResult, a vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl fixtures, and a record-fixture CLI that writes fixture meta headers. Hard contracts: - No fallback: replay-client and fixture-loader throw on exhaustion / malformed input rather than synthesizing defaults. - No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall verbatim; shadow types are forbidden. - No fake fixture data: tests/eval/fixtures/ ships only README + .gitkeep. Real traces are recorded by the user against live providers. package.json adds test:eval (vitest --passWithNoTests so an empty fixture dir is green) and eval:record (CLI entrypoint). electron-builder already excludes tests/ via its files: glob — fixtures never enter the shipped .asar.
This commit is contained in:
1 parent
76d9031a41
commit
e7f879bed0
12 files changed
+931
No files matched your search
Whitespace-only changes.
@@ -0,0 +1,44 @@
|
||||
# Eval fixtures
|
||||
|
||||
Every file in this directory is a **recorded trace** of a real AgentCompany
|
||||
turn against a real LLM provider. Fixtures are recorded-truth, not
|
||||
synthesized mock data — the distinction is what keeps the B5 harness on the
|
||||
"fixture, not fallback" side of the roundtable line.
|
||||
|
||||
## File format
|
||||
|
||||
Each fixture is a `.jsonl` file with one JSON object per line:
|
||||
|
||||
1. First line: a `{ "type": "meta", ... }` record (see `FixtureMeta` in
|
||||
`../types.ts`).
|
||||
2. Middle lines: one `{ "type": "turn", "turnIndex": N, ... }` record per
|
||||
conversation turn, in order. `turnIndex` is strictly increasing from `0`.
|
||||
3. Last line: a `{ "type": "assertions", "assertions": [...] }` record with
|
||||
the expected behavior contract.
|
||||
|
||||
The loader rejects files that do not match this shape. No silent skipping.
|
||||
|
||||
## Recording workflow
|
||||
|
||||
```bash
|
||||
pnpm eval:record -- \
|
||||
--fixture-id ceo-team-building-empty \
|
||||
--scenario "Empty company, CEO bootstrap" \
|
||||
--out tests/eval/fixtures/ceo-team-building-empty.jsonl \
|
||||
--system-prompt-preview "$(cat /tmp/current-system-prompt.txt)"
|
||||
```
|
||||
|
||||
The current CLI writes a stub meta + empty assertions. The full recorder
|
||||
(which drives `pi-agent-core` against a real provider using the
|
||||
AgentCompany worker's system prompt and tool registry) lands in a
|
||||
follow-up commit — see `../README.md` for the B5 roadmap.
|
||||
|
||||
## Committing fixtures
|
||||
|
||||
Fixtures belong in git. They are small, deterministic, and the whole point
|
||||
of the eval tier is to diff behavior against recorded truth. Do NOT add
|
||||
`tests/eval/fixtures/` to `.gitignore`.
|
||||
|
||||
Fixtures are excluded from the electron-builder production bundle; see
|
||||
`electron-builder.yml` (the `files` glob only picks `out/**`, so `tests/`
|
||||
never enters the packaged `.asar`).
|
||||
Reference in new issue
Block a user