0 Fork 0

Files

T

ZacharyZhang-NY e7f879bed0 Build minimal LLM eval harness scaffold (B5) — fixtures recorded separately

Land the B5 harness skeleton: fixture loader (strict JSONL contract with
meta header + ordered turn records + assertions tail), replay-client that
swaps in for pi-ai's streamSimple by replaying recorded assistant turns,
structural assertion judge (tool_called / no_tool_called /
message_contains_substring / tool_call_arg_matches / trace_max_turns), an
eval-runner that sums the token ledger and produces an EvalResult, a
vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl
fixtures, and a record-fixture CLI that writes fixture meta headers.

Hard contracts:
- No fallback: replay-client and fixture-loader throw on exhaustion /
  malformed input rather than synthesizing defaults.
- No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall
  verbatim; shadow types are forbidden.
- No fake fixture data: tests/eval/fixtures/ ships only README +
  .gitkeep. Real traces are recorded by the user against live providers.

package.json adds test:eval (vitest --passWithNoTests so an empty
fixture dir is green) and eval:record (CLI entrypoint). electron-builder
already excludes tests/ via its files: glob — fixtures never enter the
shipped .asar.

2026-04-20 11:39:53 -04:00

.gitkeep

Build minimal LLM eval harness scaffold (B5) — fixtures recorded separately

2026-04-20 11:39:53 -04:00

README.md

Build minimal LLM eval harness scaffold (B5) — fixtures recorded separately

2026-04-20 11:39:53 -04:00

README.md

Eval fixtures

Every file in this directory is a recorded trace of a real AgentCompany turn against a real LLM provider. Fixtures are recorded-truth, not synthesized mock data — the distinction is what keeps the B5 harness on the "fixture, not fallback" side of the roundtable line.

File format

Each fixture is a .jsonl file with one JSON object per line:

First line: a { "type": "meta", ... } record (see FixtureMeta in ../types.ts).
Middle lines: one { "type": "turn", "turnIndex": N, ... } record per conversation turn, in order. turnIndex is strictly increasing from 0.
Last line: a { "type": "assertions", "assertions": [...] } record with the expected behavior contract.

The loader rejects files that do not match this shape. No silent skipping.

Recording workflow

pnpm eval:record -- \
  --fixture-id ceo-team-building-empty \
  --scenario "Empty company, CEO bootstrap" \
  --out tests/eval/fixtures/ceo-team-building-empty.jsonl \
  --system-prompt-preview "$(cat /tmp/current-system-prompt.txt)"

The current CLI writes a stub meta + empty assertions. The full recorder (which drives pi-agent-core against a real provider using the AgentCompany worker's system prompt and tool registry) lands in a follow-up commit — see ../README.md for the B5 roadmap.

Committing fixtures

Fixtures belong in git. They are small, deterministic, and the whole point of the eval tier is to diff behavior against recorded truth. Do NOT add tests/eval/fixtures/ to .gitignore.

Fixtures are excluded from the electron-builder production bundle; see electron-builder.yml (the files glob only picks out/**, so tests/ never enters the packaged .asar).