AgentCompany/tests/eval

B5 eval harness (minimal scaffold)

This directory holds the minimal LLM behavior eval tier for AgentCompany. It replays recorded LLM traces (fixtures) and checks structural assertions against them — tool names, call counts, argument shapes, message substrings, and turn caps. It does not perform semantic quality judgment yet; that tier is proposed in docs/proposals/B5.1-llm-as-judge-followup.md.

File layout

File	Role
types.ts	Wire types — fixtures, assertions, results. Reuses @mariozechner/pi-ai primitives.
fixture-loader.ts	JSONL → FixtureFile. Rejects malformed or missing meta/assertions blocks.
replay-client.ts	Swaps in for pi-ai's streamSimple; replays recorded assistant turns. Used by the live-replay tier in B5.1.
assertion-judge.ts	Structural assertions over recorded trace. No semantic judgment.
eval-runner.ts	Parse fixture → judge assertions → produce EvalResult.
record-fixture.ts	CLI for writing fixture meta headers. Full record loop lands in B5.1.
harness.test.ts	vitest driver that walks fixtures/ and asserts every fixture passes.
fixtures/	Recorded JSONL traces. Git-tracked. Not bundled into production (electron-builder.yml only ships out/**).

Running

# Assertion-only eval (no fixtures needed; passes on empty fixture dir)
pnpm test:eval

# Record a stub fixture header (full recorder lands in B5.1)
pnpm eval:record -- \
  --fixture-id ceo-team-building-empty \
  --scenario "Empty company, CEO bootstrap" \
  --out tests/eval/fixtures/ceo-team-building-empty.jsonl

Fixture JSONL format

Every fixture begins with a meta header, contains one turn per line in ascending turnIndex order, and ends with an assertions block.

{"type":"meta","fixtureId":"ceo-team-building-empty","piAgentCoreVersion":"0.67.68","recordedAt":"2026-04-20T21:14:00.000Z","scenario":"Empty company, CEO bootstrap","systemPromptHash":"sha256:…"}
{"type":"turn","turnIndex":0,"role":"user","content":[{"type":"text","text":"build me a team"}]}
{"type":"turn","turnIndex":1,"role":"assistant","content":[{"type":"text","text":"Starting team build."},{"type":"toolCall","id":"call_1","name":"hire_agent","arguments":{"role":"engineer"}}],"tokenLedger":{"input":12345,"cache_read":11000,"cache_creation":345,"output":234}}
{"type":"turn","turnIndex":2,"role":"toolResult","toolUseId":"call_1","content":[{"type":"text","text":"hired agent-42"}]}
{"type":"assertions","assertions":[{"kind":"tool_called","name":"hire_agent","min_count":3,"max_count":5},{"kind":"trace_max_turns","max":40}]}

Field contracts:

• meta.fixtureId — stable kebab-case identifier used as the scenario name.
• meta.piAgentCoreVersion — the @mariozechner/pi-agent-core version used when the fixture was recorded. Drift against the installed version is allowed but future tooling will surface a warning.
• meta.systemPromptHash — sha256 of the system prompt text. Changes in this hash are the canonical signal that a fixture is stale.
• turn.turnIndex — strictly increasing from 0. Gaps or duplicates are hard errors.
• turn.role = "assistant" requires a tokenLedger in snake_case (input, cache_read, cache_creation, output). This maps 1:1 to pi-ai's Usage.{input, cacheRead, cacheWrite, output} — renamed to provider-neutral snake_case on disk.
• assertions.assertions — see types.ts for the complete schema.

Assertion kinds

Kind	Semantics
tool_called	Matches tool calls by name. Supports min_count (default 1), max_count, and order (0-based ordinal across all tool calls in the trace).
no_tool_called	Fails if any tool call with the given name appears.
message_contains_substring	Any role-matching message whose text content contains substring passes.
tool_call_arg_matches	At least one call to name has the dotted arg_path resolving to a string matching matches_regex.
trace_max_turns	Trace must have at most max turns.

Anything that requires semantic judgment (tone, strategy, plan quality) belongs in the LLM-as-judge tier. See docs/proposals/B5.1-llm-as-judge-followup.md.

What this scaffold does NOT do tonight

• No synthesized fixtures. fixtures/ ships empty. The user records real traces by running the recorder against a live provider.
• No live-replay. replay-client.ts is ready to swap into the agent loop; wiring it into a real pi-agent-core Agent instance is the B5.1 task. Tonight the runner asserts against recorded turns directly.
• No LLM-as-judge. Assertions are structural only. Semantic judgment tier is proposed separately.
• No logging / telemetry. Results are returned from runEval as plain values; the vitest reporter is the only output surface.

Each omission is deliberate and documented. Nothing here silently degrades to a placeholder — missing fixture = hard error, malformed fixture = hard error, exhausted fixture = hard error.

Production bundle posture

electron-builder.yml ships only out/** and package.json. tests/ — including tests/eval/fixtures/ — is never packaged into the shipped .asar. Fixtures are dev-time / CI-time artifacts; they do not leak into end-user installs.