This is the verification artefact for the C4 refactor (e2c839b through
03864f3). The old fixture (recorded at 1d0cfa2, scenario asked the model
to POST hire_agent requests via bash+node+fetch) captured the model
emitting `node -e 'fetch(\`http://...\`)'` — exactly the antipattern
Jobs called out.
Post-C4 scenario was updated to reflect the new world: the CEO now has
explicit hire_agent and file_approval tools. Re-recording against
anthropic/claude-sonnet-4-5 produced 509 in / 731 out tokens and the
assistant response:
- Contains `hire_agent` (pseudo-XML tool invocations, one per hire)
- Contains `file_approval` with type=approve_ceo_strategy
- No `node -e` / bash+fetch / HTTP POST pattern anywhere
- 3-5 hires as requested, single approval block
Assertions updated to pin the post-C4 tool names as positive signals.
Absence of bash/fetch is demonstrated by diff against 1d0cfa2 (see that
commit's recorded JSONL for the before state).
Eval harness now green at 8/8 (3 self-tests + 5 recorded fixtures) with
this new baseline. Any future regression that reverts CEO bootstrap to
HTTP-path prompts will fail all three hire_agent / approve_ceo_strategy /
file_approval assertions.
Unblocks the march-of-nines third nine. Up until this commit `tests/eval/`
had a harness + self-tests but zero real traces; the fixture directory was
a documented stub. These two fixtures are genuine single-turn recordings
against anthropic/claude-sonnet-4-5 produced by `pnpm eval:record` — not
mock data, not placeholder — each one captures an actual API response
with its token ledger.
Coverage:
- ceo-bootstrap-empty: empty company, CEO bootstrap trigger. Asserts the
model produces hire_agent POSTs, an approve_ceo_strategy POST, and uses
bash+fetch (current prompt pattern before C4 lands).
- ceo-board-message-reply: operating CEO with a live team, inbound Board
message. Asserts the model hits mark_read, posts to
/api/agent_message/send, and keeps the original message_id in scope.
What these fixtures now enable:
- B2 Phase 2 switch-read: a fixture diff before/after the cutover
demonstrates dual-write parity is real, not assumed.
- C4 initial-user-turn refactor: the current HTTP-path-in-prompt pattern
is captured; after C4 lands, the diff shows whether tool-shape
refactoring regressed behaviour.
- C7 prompt-builder conditional assembly: same, but for system-prompt
slimming.
- M7 Phase 2 call-site switch: resolveCeoMode cutover can be regression-
verified against both traces.
Scenarios live at tests/eval/scenarios/<id>/{system.txt,user.txt} so any
maintainer can re-record the same trace with one command and diff against
the committed fixture. Each fixture's assertions block currently contains
four assertions; pnpm test:eval passes with 5 (3 harness self-tests + 2
real-fixture runs).
ZacharyZhang-NY