C4 follow-up #4: add message_does_not_contain_substring assertion kind + pin C4 invariants

Closes the fourth ask from the C4 cross-review (docs/proposals/C4-followup.md
Item 4). The eval harness previously had no way to assert the absence of a
pattern in a trace — only positive `message_contains_substring`. That meant
the C4 "no more node -e / bash+fetch from CEO" invariant was only
demonstrated by the diff between two commits, not by any running assertion
that future regressions would trip.

Three changes:

1. Extend the Assertion union in tests/eval/types.ts with
   `message_does_not_contain_substring` (role + substring). Symmetric to
   the positive kind; fails on first match with a specific reason.

2. Add the case in assertion-judge.ts. Same traversal as the positive
   branch but inverted exit condition.

3. Add two harness self-tests covering pass + fail, so the new kind is
   exercised even when the fixtures directory is empty.

4. Pin two C4-regression assertions onto the recorded ceo-bootstrap-empty
   fixture: the assistant response must NOT contain "node -e" OR
   "/api/companies". Any future regression that reintroduces the
   HTTP-path-in-prompt pattern or the bash+node -e shell fallback will
   now fail pnpm test:eval instead of living in the fixture silently.

typecheck green, test:eval up from 8 → 10 tests (2 new self-tests), all
passing.
This commit is contained in:
ZacharyZhang-NY committed 2026-04-20 14:53:24 -04:00
1 parent 43337e535b
commit 9a5823b242
4 files changed
+63 -1

No files matched your search

+23
View File
@@ -134,6 +134,29 @@ function evaluateOne(
reason: `no ${assertion.role} message contained substring "${assertion.substring}"`,
};
}
case "message_does_not_contain_substring": {
for (const turn of turns) {
if (turn.role !== assertion.role) continue;
const content = turn.content;
if (!Array.isArray(content)) continue;
for (const block of content) {
if (
block &&
typeof block === "object" &&
(block as { type?: string }).type === "text" &&
typeof (block as { text?: unknown }).text === "string" &&
((block as { text: string }).text).includes(assertion.substring)
) {
return {
assertion,
passed: false,
reason: `a ${assertion.role} message contained forbidden substring "${assertion.substring}"`,
};
}
}
}
return { assertion, passed: true };
}
case "tool_call_arg_matches": {
const matches = toolCalls.filter(
(occ) => occ.toolCall.name === assertion.name,