ZacharyZhang-NY
1d0cfa29d2
Unblocks the march-of-nines third nine. Up until this commit `tests/eval/`
had a harness + self-tests but zero real traces; the fixture directory was
a documented stub. These two fixtures are genuine single-turn recordings
against anthropic/claude-sonnet-4-5 produced by `pnpm eval:record` — not
mock data, not placeholder — each one captures an actual API response
with its token ledger.
Coverage:
- ceo-bootstrap-empty: empty company, CEO bootstrap trigger. Asserts the
model produces hire_agent POSTs, an approve_ceo_strategy POST, and uses
bash+fetch (current prompt pattern before C4 lands).
- ceo-board-message-reply: operating CEO with a live team, inbound Board
message. Asserts the model hits mark_read, posts to
/api/agent_message/send, and keeps the original message_id in scope.
What these fixtures now enable:
- B2 Phase 2 switch-read: a fixture diff before/after the cutover
demonstrates dual-write parity is real, not assumed.
- C4 initial-user-turn refactor: the current HTTP-path-in-prompt pattern
is captured; after C4 lands, the diff shows whether tool-shape
refactoring regressed behaviour.
- C7 prompt-builder conditional assembly: same, but for system-prompt
slimming.
- M7 Phase 2 call-site switch: resolveCeoMode cutover can be regression-
verified against both traces.
Scenarios live at tests/eval/scenarios/<id>/{system.txt,user.txt} so any
maintainer can re-record the same trace with one command and diff against
the committed fixture. Each fixture's assertions block currently contains
four assertions; pnpm test:eval passes with 5 (3 harness self-tests + 2
real-fixture runs).
5 lines
3.5 KiB
JSON
5 lines
3.5 KiB
JSON
{"type":"meta","fixtureId":"ceo-bootstrap-empty","piAgentCoreVersion":"0.67.68","recordedAt":"2026-04-20T17:59:36.130Z","scenario":"Newly founded company, empty team, CEO bootstrap trigger. Expected: 3-5 hire_agent POSTs + 1 approve_ceo_strategy POST via bash/fetch.","systemPromptHash":"sha256:cea0fa9e4728e952db302b61ed82beb0ad5ff41fedf0ad1ad0d47e91520400b9"}
|
|
{"type":"turn","turnIndex":0,"role":"user","content":[{"type":"text","text":"Trigger: bootstrap wake. Zero active direct reports. Start Team Building now — file three to five hire_agent POSTs and one approve_ceo_strategy POST against the internal HTTP API. Do not write a plan document. Execute tool calls.\n"}]}
|
|
{"type":"turn","turnIndex":1,"role":"assistant","content":[{"type":"text","text":"I'll bootstrap the team now by filing hire requests and then submitting the CEO strategy approval.\n\n<function_calls>\n<invoke name=\"bash\">\n<parameter name=\"command\">node -e \"\nconst port = process.env.AGENTCOMPANY_API_PORT;\nconst companyId = process.env.AGENTCOMPANY_COMPANY_ID;\nconst token = process.env.AGENTCOMPANY_AGENT_TOKEN;\n\nconst hires = [\n {\n role: 'CTO',\n reportsTo: 'CEO',\n mission: 'Build and scale technical infrastructure for AI agent deployment and orchestration'\n },\n {\n role: 'VP Product',\n reportsTo: 'CEO',\n mission: 'Define product roadmap and ensure market fit for AI agent solutions'\n },\n {\n role: 'VP Operations',\n reportsTo: 'CEO',\n mission: 'Establish operational processes and ensure efficient execution across all teams'\n },\n {\n role: 'VP Sales',\n reportsTo: 'CEO',\n mission: 'Drive revenue growth and build customer relationships for AI agent services'\n }\n];\n\n(async () => {\n for (const hire of hires) {\n const response = await fetch(\\`http://127.0.0.1:\\${port}/api/hire_agent\\`, {\n method: 'POST',\n headers: {\n 'Content-Type': 'application/json',\n 'Authorization': \\`Bearer \\${token}\\`\n },\n body: JSON.stringify({\n companyId,\n role: hire.role,\n reportsTo: hire.reportsTo,\n mission: hire.mission\n })\n });\n const result = await response.json();\n console.log(\\`Hired \\${hire.role}:\\`, result);\n }\n})();\n\"</parameter>\n</invoke>\n</function_calls>\n\n<function_calls>\n<invoke name=\"bash\">\n<parameter name=\"command\">node -e \"\nconst port = process.env.AGENTCOMPANY_API_PORT;\nconst companyId = process.env.AGENTCOMPANY_COMPANY_ID;\nconst token = process.env.AGENTCOMPANY_AGENT_TOKEN;\n\n(async () => {\n const response = await fetch(\\`http://127.0.0.1:\\${port}/api/approve_ceo_strategy\\`, {\n method: 'POST',\n headers: {\n 'Content-Type': 'application/json',\n 'Authorization': \\`Bearer \\${token}\\`\n },\n body: JSON.stringify({\n companyId,\n mission: 'Build and deploy enterprise-grade AI agent solutions that automate complex business workflows and deliver measurable ROI within 90 days of customer onboarding'\n })\n });\n const result = await response.json();\n console.log('CEO Strategy Approved:', result);\n})();\n\"</parameter>\n</invoke>\n</function_calls>"}],"tokenLedger":{"input":497,"cache_read":0,"cache_creation":0,"output":770}}
|
|
{"type": "assertions", "assertions": [{"kind": "message_contains_substring", "role": "assistant", "substring": "hire_agent"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "approve_ceo_strategy"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "bash"}, {"kind": "trace_max_turns", "max": 3}]}
|