Files
AgentCompany/tests
ZacharyZhang-NY ec0c81b4b3 Harden record-fixture CLI to emit real LLM trace JSONL (B5 step 2)
Replaces the stub recorder with a real-provider CLI. The command now:

  1. Reads system prompt + user message from files on disk (empty files
     are rejected — a fixture with no prompt captures no behaviour).
  2. Resolves the provider spec from `--provider anthropic|openai|google`
     and pulls the API key from the matching environment variable.
     Missing env → hard error with a copy-pasteable export hint; no
     fallback, no mock.
  3. Calls `streamSimple` from @mariozechner/pi-ai with the user-picked
     model. On provider error (stopReason=error|aborted) the CLI exits
     non-zero and writes no JSONL. Users re-run if they think the
     failure was transient.
  4. Serialises one user turn + one assistant turn + an empty
     assertions block into the JSONL shape documented in
     tests/eval/README.md. Token ledger comes straight from the
     assistant message's usage field (renamed to snake_case for
     provider-neutral on-disk format).

Scope is deliberately single-turn: tool use / multi-turn agent loops
are B5.1's job. Importing the AgentCompany worker's system prompt
builder would require the Electron runtime; the CLI stays node-only
so it can run standalone.

`--help` works and prints the argument matrix. All failure modes exit
non-zero with a specific, actionable message.

Also document the new recording workflow in tests/eval/README.md —
four-step procedure, example scenario files, assertion templates, and
explicit list of what the recorder still does not do.
2026-04-20 12:28:23 -04:00
..
2026-04-17 14:03:56 -04:00