Build minimal LLM eval harness scaffold (B5) — fixtures recorded separately
Land the B5 harness skeleton: fixture loader (strict JSONL contract with meta header + ordered turn records + assertions tail), replay-client that swaps in for pi-ai's streamSimple by replaying recorded assistant turns, structural assertion judge (tool_called / no_tool_called / message_contains_substring / tool_call_arg_matches / trace_max_turns), an eval-runner that sums the token ledger and produces an EvalResult, a vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl fixtures, and a record-fixture CLI that writes fixture meta headers. Hard contracts: - No fallback: replay-client and fixture-loader throw on exhaustion / malformed input rather than synthesizing defaults. - No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall verbatim; shadow types are forbidden. - No fake fixture data: tests/eval/fixtures/ ships only README + .gitkeep. Real traces are recorded by the user against live providers. package.json adds test:eval (vitest --passWithNoTests so an empty fixture dir is green) and eval:record (CLI entrypoint). electron-builder already excludes tests/ via its files: glob — fixtures never enter the shipped .asar.
This commit is contained in:
1 parent
76d9031a41
commit
e7f879bed0
12 files changed
+931
No files matched your search
@@ -64,6 +64,7 @@ oldreference/
|
||||
# the parent directory; losing SKILL.md to the blanket *.md ignore meant a
|
||||
# fresh checkout silently dropped every skill from the catalogue.
|
||||
!/builtin-skills/**/*.md
|
||||
!/tests/eval/**/*.md
|
||||
*.zip
|
||||
*.blockmap
|
||||
*.exe
|
||||
|
||||
Reference in new issue
Block a user