ZacharyZhang-NY
e7f879bed0
Land the B5 harness skeleton: fixture loader (strict JSONL contract with meta header + ordered turn records + assertions tail), replay-client that swaps in for pi-ai's streamSimple by replaying recorded assistant turns, structural assertion judge (tool_called / no_tool_called / message_contains_substring / tool_call_arg_matches / trace_max_turns), an eval-runner that sums the token ledger and produces an EvalResult, a vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl fixtures, and a record-fixture CLI that writes fixture meta headers. Hard contracts: - No fallback: replay-client and fixture-loader throw on exhaustion / malformed input rather than synthesizing defaults. - No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall verbatim; shadow types are forbidden. - No fake fixture data: tests/eval/fixtures/ ships only README + .gitkeep. Real traces are recorded by the user against live providers. package.json adds test:eval (vitest --passWithNoTests so an empty fixture dir is green) and eval:record (CLI entrypoint). electron-builder already excludes tests/ via its files: glob — fixtures never enter the shipped .asar.
101 lines
5.2 KiB
Markdown
101 lines
5.2 KiB
Markdown
# B5 eval harness (minimal scaffold)
|
|
|
|
This directory holds the minimal LLM behavior eval tier for AgentCompany.
|
|
It replays recorded LLM traces (fixtures) and checks structural assertions
|
|
against them — tool names, call counts, argument shapes, message
|
|
substrings, and turn caps. It does *not* perform semantic quality judgment
|
|
yet; that tier is proposed in
|
|
`docs/proposals/B5.1-llm-as-judge-followup.md`.
|
|
|
|
## File layout
|
|
|
|
| File | Role |
|
|
|---|---|
|
|
| `types.ts` | Wire types — fixtures, assertions, results. Reuses `@mariozechner/pi-ai` primitives. |
|
|
| `fixture-loader.ts` | JSONL → `FixtureFile`. Rejects malformed or missing meta/assertions blocks. |
|
|
| `replay-client.ts` | Swaps in for `pi-ai`'s `streamSimple`; replays recorded assistant turns. Used by the live-replay tier in B5.1. |
|
|
| `assertion-judge.ts` | Structural assertions over recorded trace. No semantic judgment. |
|
|
| `eval-runner.ts` | Parse fixture → judge assertions → produce `EvalResult`. |
|
|
| `record-fixture.ts` | CLI for writing fixture meta headers. Full record loop lands in B5.1. |
|
|
| `harness.test.ts` | vitest driver that walks `fixtures/` and asserts every fixture passes. |
|
|
| `fixtures/` | Recorded JSONL traces. Git-tracked. Not bundled into production (`electron-builder.yml` only ships `out/**`). |
|
|
|
|
## Running
|
|
|
|
```bash
|
|
# Assertion-only eval (no fixtures needed; passes on empty fixture dir)
|
|
pnpm test:eval
|
|
|
|
# Record a stub fixture header (full recorder lands in B5.1)
|
|
pnpm eval:record -- \
|
|
--fixture-id ceo-team-building-empty \
|
|
--scenario "Empty company, CEO bootstrap" \
|
|
--out tests/eval/fixtures/ceo-team-building-empty.jsonl
|
|
```
|
|
|
|
## Fixture JSONL format
|
|
|
|
Every fixture begins with a `meta` header, contains one `turn` per line in
|
|
ascending `turnIndex` order, and ends with an `assertions` block.
|
|
|
|
```jsonl
|
|
{"type":"meta","fixtureId":"ceo-team-building-empty","piAgentCoreVersion":"0.67.68","recordedAt":"2026-04-20T21:14:00.000Z","scenario":"Empty company, CEO bootstrap","systemPromptHash":"sha256:…"}
|
|
{"type":"turn","turnIndex":0,"role":"user","content":[{"type":"text","text":"build me a team"}]}
|
|
{"type":"turn","turnIndex":1,"role":"assistant","content":[{"type":"text","text":"Starting team build."},{"type":"toolCall","id":"call_1","name":"hire_agent","arguments":{"role":"engineer"}}],"tokenLedger":{"input":12345,"cache_read":11000,"cache_creation":345,"output":234}}
|
|
{"type":"turn","turnIndex":2,"role":"toolResult","toolUseId":"call_1","content":[{"type":"text","text":"hired agent-42"}]}
|
|
{"type":"assertions","assertions":[{"kind":"tool_called","name":"hire_agent","min_count":3,"max_count":5},{"kind":"trace_max_turns","max":40}]}
|
|
```
|
|
|
|
Field contracts:
|
|
|
|
- `meta.fixtureId` — stable kebab-case identifier used as the scenario name.
|
|
- `meta.piAgentCoreVersion` — the `@mariozechner/pi-agent-core` version
|
|
used when the fixture was recorded. Drift against the installed version
|
|
is allowed but future tooling will surface a warning.
|
|
- `meta.systemPromptHash` — sha256 of the system prompt text. Changes in
|
|
this hash are the canonical signal that a fixture is stale.
|
|
- `turn.turnIndex` — strictly increasing from 0. Gaps or duplicates are
|
|
hard errors.
|
|
- `turn.role = "assistant"` requires a `tokenLedger` in snake_case
|
|
(`input`, `cache_read`, `cache_creation`, `output`). This maps 1:1 to
|
|
pi-ai's `Usage.{input, cacheRead, cacheWrite, output}` — renamed to
|
|
provider-neutral snake_case on disk.
|
|
- `assertions.assertions` — see `types.ts` for the complete schema.
|
|
|
|
## Assertion kinds
|
|
|
|
| Kind | Semantics |
|
|
|---|---|
|
|
| `tool_called` | Matches tool calls by `name`. Supports `min_count` (default 1), `max_count`, and `order` (0-based ordinal across all tool calls in the trace). |
|
|
| `no_tool_called` | Fails if any tool call with the given name appears. |
|
|
| `message_contains_substring` | Any `role`-matching message whose text content contains `substring` passes. |
|
|
| `tool_call_arg_matches` | At least one call to `name` has the dotted `arg_path` resolving to a string matching `matches_regex`. |
|
|
| `trace_max_turns` | Trace must have at most `max` turns. |
|
|
|
|
Anything that requires semantic judgment (tone, strategy, plan quality)
|
|
belongs in the LLM-as-judge tier. See
|
|
`docs/proposals/B5.1-llm-as-judge-followup.md`.
|
|
|
|
## What this scaffold does NOT do tonight
|
|
|
|
- **No synthesized fixtures.** `fixtures/` ships empty. The user records
|
|
real traces by running the recorder against a live provider.
|
|
- **No live-replay.** `replay-client.ts` is ready to swap into the agent
|
|
loop; wiring it into a real `pi-agent-core` Agent instance is the B5.1
|
|
task. Tonight the runner asserts against recorded turns directly.
|
|
- **No LLM-as-judge.** Assertions are structural only. Semantic judgment
|
|
tier is proposed separately.
|
|
- **No logging / telemetry.** Results are returned from `runEval` as
|
|
plain values; the vitest reporter is the only output surface.
|
|
|
|
Each omission is deliberate and documented. Nothing here silently degrades
|
|
to a placeholder — missing fixture = hard error, malformed fixture = hard
|
|
error, exhausted fixture = hard error.
|
|
|
|
## Production bundle posture
|
|
|
|
`electron-builder.yml` ships only `out/**` and `package.json`. `tests/` —
|
|
including `tests/eval/fixtures/` — is never packaged into the shipped
|
|
`.asar`. Fixtures are dev-time / CI-time artifacts; they do not leak into
|
|
end-user installs.
|