Replaces the stub recorder with a real-provider CLI. The command now:
1. Reads system prompt + user message from files on disk (empty files
are rejected — a fixture with no prompt captures no behaviour).
2. Resolves the provider spec from `--provider anthropic|openai|google`
and pulls the API key from the matching environment variable.
Missing env → hard error with a copy-pasteable export hint; no
fallback, no mock.
3. Calls `streamSimple` from @mariozechner/pi-ai with the user-picked
model. On provider error (stopReason=error|aborted) the CLI exits
non-zero and writes no JSONL. Users re-run if they think the
failure was transient.
4. Serialises one user turn + one assistant turn + an empty
assertions block into the JSONL shape documented in
tests/eval/README.md. Token ledger comes straight from the
assistant message's usage field (renamed to snake_case for
provider-neutral on-disk format).
Scope is deliberately single-turn: tool use / multi-turn agent loops
are B5.1's job. Importing the AgentCompany worker's system prompt
builder would require the Electron runtime; the CLI stays node-only
so it can run standalone.
`--help` works and prints the argument matrix. All failure modes exit
non-zero with a specific, actionable message.
Also document the new recording workflow in tests/eval/README.md —
four-step procedure, example scenario files, assertion templates, and
explicit list of what the recorder still does not do.
Land the B5 harness skeleton: fixture loader (strict JSONL contract with
meta header + ordered turn records + assertions tail), replay-client that
swaps in for pi-ai's streamSimple by replaying recorded assistant turns,
structural assertion judge (tool_called / no_tool_called /
message_contains_substring / tool_call_arg_matches / trace_max_turns), an
eval-runner that sums the token ledger and produces an EvalResult, a
vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl
fixtures, and a record-fixture CLI that writes fixture meta headers.
Hard contracts:
- No fallback: replay-client and fixture-loader throw on exhaustion /
malformed input rather than synthesizing defaults.
- No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall
verbatim; shadow types are forbidden.
- No fake fixture data: tests/eval/fixtures/ ships only README +
.gitkeep. Real traces are recorded by the user against live providers.
package.json adds test:eval (vitest --passWithNoTests so an empty
fixture dir is green) and eval:record (CLI entrypoint). electron-builder
already excludes tests/ via its files: glob — fixtures never enter the
shipped .asar.
ZacharyZhang-NY