ZacharyZhang-NY
e7f879bed0
Land the B5 harness skeleton: fixture loader (strict JSONL contract with meta header + ordered turn records + assertions tail), replay-client that swaps in for pi-ai's streamSimple by replaying recorded assistant turns, structural assertion judge (tool_called / no_tool_called / message_contains_substring / tool_call_arg_matches / trace_max_turns), an eval-runner that sums the token ledger and produces an EvalResult, a vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl fixtures, and a record-fixture CLI that writes fixture meta headers. Hard contracts: - No fallback: replay-client and fixture-loader throw on exhaustion / malformed input rather than synthesizing defaults. - No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall verbatim; shadow types are forbidden. - No fake fixture data: tests/eval/fixtures/ ships only README + .gitkeep. Real traces are recorded by the user against live providers. package.json adds test:eval (vitest --passWithNoTests so an empty fixture dir is green) and eval:record (CLI entrypoint). electron-builder already excludes tests/ via its files: glob — fixtures never enter the shipped .asar.
50 lines
1.8 KiB
TypeScript
50 lines
1.8 KiB
TypeScript
/**
|
|
* B5 eval harness — driver spec.
|
|
*
|
|
* This spec discovers every `*.jsonl` fixture under `tests/eval/fixtures/`
|
|
* and runs it through the assertion judge. If there are zero fixtures, the
|
|
* spec file has zero tests — vitest is invoked with `--passWithNoTests`
|
|
* from the `test:eval` script so this is a green no-op until the user
|
|
* records real fixtures.
|
|
*
|
|
* The harness does NOT synthesize placeholder fixtures under any
|
|
* circumstance. An empty `fixtures/` directory is a real state — the tier
|
|
* is simply not yet exercising anything — and the green run reflects that
|
|
* honestly.
|
|
*/
|
|
|
|
import { readdir } from "node:fs/promises";
|
|
import { join } from "node:path";
|
|
import { describe, expect, it } from "vitest";
|
|
import { loadFixture } from "./fixture-loader.js";
|
|
import { runEval } from "./eval-runner.js";
|
|
|
|
const FIXTURE_DIR = join(__dirname, "fixtures");
|
|
|
|
async function listFixtures(): Promise<string[]> {
|
|
const entries = await readdir(FIXTURE_DIR, { withFileTypes: true });
|
|
return entries
|
|
.filter((entry) => entry.isFile() && entry.name.endsWith(".jsonl"))
|
|
.map((entry) => join(FIXTURE_DIR, entry.name))
|
|
.sort();
|
|
}
|
|
|
|
const fixturePaths = await listFixtures();
|
|
|
|
describe.each(fixturePaths)("eval fixture %s", (fixturePath) => {
|
|
it("passes all assertions", async () => {
|
|
const fixture = await loadFixture(fixturePath);
|
|
const result = runEval(fixture);
|
|
if (!result.passed) {
|
|
const failures = result.assertions
|
|
.filter((assertion) => !assertion.passed)
|
|
.map((assertion) => `- ${assertion.assertion.kind}: ${assertion.reason ?? "<no reason>"}`)
|
|
.join("\n");
|
|
throw new Error(
|
|
`fixture ${result.fixtureId} failed:\n${failures}\ntoken ledger: ${JSON.stringify(result.tokenLedger)}`,
|
|
);
|
|
}
|
|
expect(result.passed).toBe(true);
|
|
});
|
|
});
|