Harden record-fixture CLI to emit real LLM trace JSONL (B5 step 2)

Replaces the stub recorder with a real-provider CLI. The command now:

  1. Reads system prompt + user message from files on disk (empty files
     are rejected — a fixture with no prompt captures no behaviour).
  2. Resolves the provider spec from `--provider anthropic|openai|google`
     and pulls the API key from the matching environment variable.
     Missing env → hard error with a copy-pasteable export hint; no
     fallback, no mock.
  3. Calls `streamSimple` from @mariozechner/pi-ai with the user-picked
     model. On provider error (stopReason=error|aborted) the CLI exits
     non-zero and writes no JSONL. Users re-run if they think the
     failure was transient.
  4. Serialises one user turn + one assistant turn + an empty
     assertions block into the JSONL shape documented in
     tests/eval/README.md. Token ledger comes straight from the
     assistant message's usage field (renamed to snake_case for
     provider-neutral on-disk format).

Scope is deliberately single-turn: tool use / multi-turn agent loops
are B5.1's job. Importing the AgentCompany worker's system prompt
builder would require the Electron runtime; the CLI stays node-only
so it can run standalone.

`--help` works and prints the argument matrix. All failure modes exit
non-zero with a specific, actionable message.

Also document the new recording workflow in tests/eval/README.md —
four-step procedure, example scenario files, assertion templates, and
explicit list of what the recorder still does not do.
This commit is contained in:
ZacharyZhang-NY committed 2026-04-20 12:28:23 -04:00
1 parent 1c7bc71931
commit ec0c81b4b3
2 files changed
+353 -55

No files matched your search

+124 -10
View File
@@ -25,14 +25,125 @@ yet; that tier is proposed in
```bash
# Assertion-only eval (no fixtures needed; passes on empty fixture dir)
pnpm test:eval
# Record a stub fixture header (full recorder lands in B5.1)
pnpm eval:record -- \
--fixture-id ceo-team-building-empty \
--scenario "Empty company, CEO bootstrap" \
--out tests/eval/fixtures/ceo-team-building-empty.jsonl
```
## Recording your first fixture
The recorder drives the **real** `@mariozechner/pi-ai` client against a
live provider. No mocks, no fallbacks. A fixture that exists was
produced by a real API call, end of story.
### Step 1 — export an API key
Pick the provider you want to record against and export the matching
variable:
```bash
export ANTHROPIC_API_KEY=sk-ant-…
# or
export OPENAI_API_KEY=sk-…
# or
export GOOGLE_API_KEY=…
```
The recorder **fails hard** if the variable is absent. This is
deliberate: silently skipping the call would produce a fixture that
documents the wrong thing.
### Step 2 — write the system prompt and user message to disk
The recorder reads both from files so you can version your scenarios
without pasting long prompts into shell args. Put them anywhere; the
conventional location is under `tests/eval/scenarios/<id>/`:
```bash
mkdir -p tests/eval/scenarios/ceo-bootstrap-empty
cat > tests/eval/scenarios/ceo-bootstrap-empty/system.txt <<'EOF'
You are the CEO of AgentCompany. Your team is empty. Call the `hire_agent`
tool to file the first three hires for the engineering, design, and
research departments. After all three, call `approve_ceo_strategy` once.
EOF
cat > tests/eval/scenarios/ceo-bootstrap-empty/user.txt <<'EOF'
Build me a team.
EOF
```
Empty files are rejected. A fixture with no real prompt captures no
real behaviour.
### Step 3 — record
```bash
pnpm eval:record \
--scenario "Empty company, CEO bootstrap" \
--fixture-id ceo-bootstrap-empty \
--provider anthropic \
--model claude-sonnet-4-5-20250929 \
--system-prompt-file tests/eval/scenarios/ceo-bootstrap-empty/system.txt \
--user-message-file tests/eval/scenarios/ceo-bootstrap-empty/user.txt \
--output tests/eval/fixtures/ceo-bootstrap-empty.jsonl
```
On success you'll see the usage ledger printed to stdout and the JSONL
written at `--output`. The assertions block is written empty; the next
step is to fill it in.
### Step 4 — add assertions
Open the fixture you just wrote and replace the empty assertions block:
```diff
- {"type":"assertions","assertions":[]}
+ {"type":"assertions","assertions":[
+ {"kind":"tool_called","name":"hire_agent","min_count":3,"max_count":5},
+ {"kind":"message_contains_substring","role":"assistant","substring":"engineering"},
+ {"kind":"trace_max_turns","max":40}
+ ]}
```
Common assertion shapes:
```json
{"kind":"tool_called","name":"hire_agent","min_count":3}
{"kind":"no_tool_called","name":"delete_everything"}
{"kind":"message_contains_substring","role":"assistant","substring":"approve_ceo_strategy"}
{"kind":"tool_call_arg_matches","name":"hire_agent","arg_path":"department","matches_regex":"^(engineering|design|research)$"}
```
See `types.ts` for the exhaustive schema.
### Step 5 — verify
```bash
pnpm test:eval --run
```
A green run means: given the recorded assistant turn, every assertion
you wrote passes. A red run means either the assertion is wrong or the
assistant did not produce the behaviour you expected — inspect the
fixture and decide which.
### Re-recording
Fixtures drift when the system prompt changes (the `systemPromptHash`
header surfaces that). To re-record, delete the old fixture and run the
recorder again. Do **not** hand-edit the `turn` records; that defeats
the point of recording against a real provider.
### What the recorder does NOT do
- It does not loop tool calls back to the provider. Today's recorder
captures one turn: user message in, assistant reply out. Multi-turn
with tool execution is B5.1's scope.
- It does not retry on provider errors. A 529 or a 5xx surfaces as a
non-zero exit and nothing is written. Re-run manually if you think
the failure was transient.
- It does not reach into AgentCompany's `src/main/` to import the
worker's system prompt builder or tool catalogue. You bring your own
prompt and message text on the CLI. Importing the worker would
require the Electron runtime; the CLI is node-only on purpose.
## Fixture JSONL format
Every fixture begins with a `meta` header, contains one `turn` per line in
@@ -78,11 +189,14 @@ belongs in the LLM-as-judge tier. See
## What this scaffold does NOT do tonight
- **No synthesized fixtures.** `fixtures/` ships empty. The user records
real traces by running the recorder against a live provider.
- **No synthesized fixtures.** `fixtures/` ships empty. Fixtures arrive
by running the recorder against a live provider.
- **No multi-turn tool-use recording.** The recorder captures one
user→assistant turn. Multi-turn runs with real tool execution are
B5.1's scope.
- **No live-replay.** `replay-client.ts` is ready to swap into the agent
loop; wiring it into a real `pi-agent-core` Agent instance is the B5.1
task. Tonight the runner asserts against recorded turns directly.
loop; wiring it into a real `pi-agent-core` Agent instance is also
B5.1. Tonight the runner asserts against recorded turns directly.
- **No LLM-as-judge.** Assertions are structural only. Semantic judgment
tier is proposed separately.
- **No logging / telemetry.** Results are returned from `runEval` as