ZacharyZhang-NY e7f879bed0 Build minimal LLM eval harness scaffold (B5) — fixtures recorded separately ...

Land the B5 harness skeleton: fixture loader (strict JSONL contract with
meta header + ordered turn records + assertions tail), replay-client that
swaps in for pi-ai's streamSimple by replaying recorded assistant turns,
structural assertion judge (tool_called / no_tool_called /
message_contains_substring / tool_call_arg_matches / trace_max_turns), an
eval-runner that sums the token ledger and produces an EvalResult, a
vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl
fixtures, and a record-fixture CLI that writes fixture meta headers.

Hard contracts:
- No fallback: replay-client and fixture-loader throw on exhaustion /
  malformed input rather than synthesizing defaults.
- No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall
  verbatim; shadow types are forbidden.
- No fake fixture data: tests/eval/fixtures/ ships only README +
  .gitkeep. Real traces are recorded by the user against live providers.

package.json adds test:eval (vitest --passWithNoTests so an empty
fixture dir is green) and eval:record (CLI entrypoint). electron-builder
already excludes tests/ via its files: glob — fixtures never enter the
shipped .asar.

2026-04-20 11:39:53 -04:00

0 lines

0 B

Plaintext

Raw Blame History

The file is empty.