test(eval): enforce SOP prompt coverage

This commit is contained in:
ZacharyZhang-NY committed 2026-04-22 16:28:50 -04:00
1 parent 775598ef4c
commit 58a4be7fe7
3 files changed
+99 -9

No files matched your search

+13
View File
@@ -22,6 +22,17 @@ shapes, message substrings, and turn caps. Semantic quality judgment belongs in
| `harness.test.ts` | vitest driver that walks `fixtures/` and asserts every fixture passes. |
| `fixtures/` | Recorded JSONL traces. Git-tracked. Not bundled into production (`electron-builder.yml` only ships `out/**`). |
## SOP coverage gate
`scenario-contract.ts` owns the prompt SOP coverage matrix. Each row renders a
real `buildTaskPrompt` context for one audience and lists the SOP headings that
must be present or absent for that scenario.
`harness.test.ts` reads every source file under
`src/main/prompt-builder/sops/*.en.md` and requires each `SOP-###` heading to be
claimed by at least one matrix scenario. Adding, deleting, or renaming a SOP
therefore changes `pnpm test:eval` directly.
## Running
```bash
@@ -187,6 +198,8 @@ belongs in the LLM-as-judge tier. See
- Fixtures are recorded traces from live provider calls.
- `fixtures/` belongs in git and participates in `pnpm verify`.
- Every source SOP under `src/main/prompt-builder/sops/` belongs to at least
one `EVAL_SCENARIO_MATRIX.requiredSops` entry.
- `scenarios/<fixtureId>/system.txt` must hash to
`meta.systemPromptHash`.
- `scenarios/<fixtureId>/user.txt` must match the first user turn exactly.