C4 follow-up #4: add message_does_not_contain_substring assertion kind + pin C4 invariants
Closes the fourth ask from the C4 cross-review (docs/proposals/C4-followup.md Item 4). The eval harness previously had no way to assert the absence of a pattern in a trace — only positive `message_contains_substring`. That meant the C4 "no more node -e / bash+fetch from CEO" invariant was only demonstrated by the diff between two commits, not by any running assertion that future regressions would trip. Three changes: 1. Extend the Assertion union in tests/eval/types.ts with `message_does_not_contain_substring` (role + substring). Symmetric to the positive kind; fails on first match with a specific reason. 2. Add the case in assertion-judge.ts. Same traversal as the positive branch but inverted exit condition. 3. Add two harness self-tests covering pass + fail, so the new kind is exercised even when the fixtures directory is empty. 4. Pin two C4-regression assertions onto the recorded ceo-bootstrap-empty fixture: the assistant response must NOT contain "node -e" OR "/api/companies". Any future regression that reintroduces the HTTP-path-in-prompt pattern or the bash+node -e shell fallback will now fail pnpm test:eval instead of living in the fixture silently. typecheck green, test:eval up from 8 → 10 tests (2 new self-tests), all passing.
This commit is contained in:
1 parent
43337e535b
commit
9a5823b242
4 files changed
+63
-1
No files matched your search
@@ -131,6 +131,34 @@ describe("harness self-test (in-memory fixture)", () => {
|
||||
expect(reasons[2]).toContain("trace has 2 turns, max allowed is 1");
|
||||
});
|
||||
|
||||
it("message_does_not_contain_substring passes when substring is absent", () => {
|
||||
const fixture = buildInMemoryFixture([
|
||||
{
|
||||
kind: "message_does_not_contain_substring",
|
||||
role: "assistant",
|
||||
substring: "node -e",
|
||||
},
|
||||
]);
|
||||
const result = runEval(fixture);
|
||||
expect(result.passed).toBe(true);
|
||||
expect(result.assertions[0]?.passed).toBe(true);
|
||||
});
|
||||
|
||||
it("message_does_not_contain_substring fails when substring is present", () => {
|
||||
const fixture = buildInMemoryFixture([
|
||||
{
|
||||
kind: "message_does_not_contain_substring",
|
||||
role: "assistant",
|
||||
substring: "Hiring a writer",
|
||||
},
|
||||
]);
|
||||
const result = runEval(fixture);
|
||||
expect(result.passed).toBe(false);
|
||||
expect(result.assertions[0]?.reason).toContain(
|
||||
'a assistant message contained forbidden substring "Hiring a writer"',
|
||||
);
|
||||
});
|
||||
|
||||
it("judgeAssertions is a pure function of (assertions, turns)", () => {
|
||||
const fixture = buildInMemoryFixture([]);
|
||||
const first = judgeAssertions(
|
||||
|
||||
Reference in new issue
Block a user