Record first two real B5 eval fixtures (ceo-bootstrap-empty + ceo-board-message-reply)

Unblocks the march-of-nines third nine. Up until this commit `tests/eval/` had a harness + self-tests but zero real traces; the fixture directory was a documented stub. These two fixtures are genuine single-turn recordings against anthropic/claude-sonnet-4-5 produced by `pnpm eval:record` — not mock data, not placeholder — each one captures an actual API response with its token ledger. Coverage: - ceo-bootstrap-empty: empty company, CEO bootstrap trigger. Asserts the model produces hire_agent POSTs, an approve_ceo_strategy POST, and uses bash+fetch (current prompt pattern before C4 lands). - ceo-board-message-reply: operating CEO with a live team, inbound Board message. Asserts the model hits mark_read, posts to /api/agent_message/send, and keeps the original message_id in scope. What these fixtures now enable: - B2 Phase 2 switch-read: a fixture diff before/after the cutover demonstrates dual-write parity is real, not assumed. - C4 initial-user-turn refactor: the current HTTP-path-in-prompt pattern is captured; after C4 lands, the diff shows whether tool-shape refactoring regressed behaviour. - C7 prompt-builder conditional assembly: same, but for system-prompt slimming. - M7 Phase 2 call-site switch: resolveCeoMode cutover can be regression- verified against both traces. Scenarios live at tests/eval/scenarios/<id>/{system.txt,user.txt} so any maintainer can re-record the same trace with one command and diff against the committed fixture. Each fixture's assertions block currently contains four assertions; pnpm test:eval passes with 5 (3 harness self-tests + 2 real-fixture runs).
2026-04-20 14:01:22 -04:00
1 parent 720387b6ed
commit 1d0cfa29d2
@@ -0,0 +1,4 @@
+{"type":"meta","fixtureId":"ceo-board-message-reply","piAgentCoreVersion":"0.67.68","recordedAt":"2026-04-20T18:00:38.001Z","scenario":"CEO with active team receives a Board message asking for beta ship status; must mark read + send reply via HTTP API.","systemPromptHash":"sha256:601784485a4de30335a3b043ed497e634968a82cef62e8f03dcf5c7823e0ccec"}
+{"type":"turn","turnIndex":0,"role":"user","content":[{"type":"text","text":"Inbound message from the Board (message_id=m_42, channel=ceo_board, from=Board/Human):\n\nSubject: Beta ship status\n\n\"Two weeks to end of quarter. Where are we on the beta? Give me the biggest risk and who owns mitigating it.\"\n\nTrigger: message. Unread count: 1. Mark m_42 read, then compose and send your reply via /api/agent_message/send. Do it now; do not write a plan document first.\n"}]}
+{"type":"turn","turnIndex":1,"role":"assistant","content":[{"type":"text","text":"I'll mark the message as read and send my reply.\n\n<function_calls>\n<invoke name=\"bash\">\n<parameter name=\"command\">curl -X POST \"http://127.0.0.1:${AGENTCOMPANY_API_PORT}/api/agent_message/mark_read\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer ${AGENTCOMPANY_AGENT_TOKEN}\" \\\n  -d \"{\\\"company_id\\\":\\\"${AGENTCOMPANY_COMPANY_ID}\\\",\\\"message_id\\\":\\\"m_42\\\"}\"</parameter>\n</invoke>\n</function_calls>\n<function_calls>\n<invoke name=\"bash\">\n<parameter name=\"command\">curl -X POST \"http://127.0.0.1:${AGENTCOMPANY_API_PORT}/api/agent_message/send\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer ${AGENTCOMPANY_AGENT_TOKEN}\" \\\n  -d \"{\n    \\\"company_id\\\":\\\"${AGENTCOMPANY_COMPANY_ID}\\\",\n    \\\"channel\\\":\\\"ceo_board\\\",\n    \\\"thread_id\\\":\\\"m_42\\\",\n    \\\"subject\\\":\\\"Re: Beta ship status\\\",\n    \\\"body\\\":\\\"We're tracking to ship on time with core orchestration features complete. VP Product reports agent-to-agent communication is stable in staging; VP Operations has production infrastructure stood up and monitoring configured.\\\\n\\\\nBiggest risk: enterprise authentication integration. We don't have SSO/SAML fully tested with the top three identity providers our pilot customers use. This could block adoption even if we ship the orchestration layer.\\\\n\\\\nCTO owns mitigation. He's pulling one senior engineer off performance optimization to finish the IdP test matrix by Friday. If we hit blockers, we'll scope down to OIDC-only for beta and document SSO as post-beta. I'm reviewing with him daily at 9am through end of quarter.\\\"\n  }\"</parameter>\n</invoke>\n</function_calls>"}],"tokenLedger":{"input":459,"cache_read":0,"cache_creation":0,"output":529}}
+{"type": "assertions", "assertions": [{"kind": "message_contains_substring", "role": "assistant", "substring": "mark_read"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "/api/agent_message/send"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "m_42"}, {"kind": "trace_max_turns", "max": 3}]}
@@ -0,0 +1,4 @@
+{"type":"meta","fixtureId":"ceo-bootstrap-empty","piAgentCoreVersion":"0.67.68","recordedAt":"2026-04-20T17:59:36.130Z","scenario":"Newly founded company, empty team, CEO bootstrap trigger. Expected: 3-5 hire_agent POSTs + 1 approve_ceo_strategy POST via bash/fetch.","systemPromptHash":"sha256:cea0fa9e4728e952db302b61ed82beb0ad5ff41fedf0ad1ad0d47e91520400b9"}
+{"type":"turn","turnIndex":0,"role":"user","content":[{"type":"text","text":"Trigger: bootstrap wake. Zero active direct reports. Start Team Building now — file three to five hire_agent POSTs and one approve_ceo_strategy POST against the internal HTTP API. Do not write a plan document. Execute tool calls.\n"}]}
+{"type":"turn","turnIndex":1,"role":"assistant","content":[{"type":"text","text":"I'll bootstrap the team now by filing hire requests and then submitting the CEO strategy approval.\n\n<function_calls>\n<invoke name=\"bash\">\n<parameter name=\"command\">node -e \"\nconst port = process.env.AGENTCOMPANY_API_PORT;\nconst companyId = process.env.AGENTCOMPANY_COMPANY_ID;\nconst token = process.env.AGENTCOMPANY_AGENT_TOKEN;\n\nconst hires = [\n  {\n    role: 'CTO',\n    reportsTo: 'CEO',\n    mission: 'Build and scale technical infrastructure for AI agent deployment and orchestration'\n  },\n  {\n    role: 'VP Product',\n    reportsTo: 'CEO',\n    mission: 'Define product roadmap and ensure market fit for AI agent solutions'\n  },\n  {\n    role: 'VP Operations',\n    reportsTo: 'CEO',\n    mission: 'Establish operational processes and ensure efficient execution across all teams'\n  },\n  {\n    role: 'VP Sales',\n    reportsTo: 'CEO',\n    mission: 'Drive revenue growth and build customer relationships for AI agent services'\n  }\n];\n\n(async () => {\n  for (const hire of hires) {\n    const response = await fetch(\\`http://127.0.0.1:\\${port}/api/hire_agent\\`, {\n      method: 'POST',\n      headers: {\n        'Content-Type': 'application/json',\n        'Authorization': \\`Bearer \\${token}\\`\n      },\n      body: JSON.stringify({\n        companyId,\n        role: hire.role,\n        reportsTo: hire.reportsTo,\n        mission: hire.mission\n      })\n    });\n    const result = await response.json();\n    console.log(\\`Hired \\${hire.role}:\\`, result);\n  }\n})();\n\"</parameter>\n</invoke>\n</function_calls>\n\n<function_calls>\n<invoke name=\"bash\">\n<parameter name=\"command\">node -e \"\nconst port = process.env.AGENTCOMPANY_API_PORT;\nconst companyId = process.env.AGENTCOMPANY_COMPANY_ID;\nconst token = process.env.AGENTCOMPANY_AGENT_TOKEN;\n\n(async () => {\n  const response = await fetch(\\`http://127.0.0.1:\\${port}/api/approve_ceo_strategy\\`, {\n    method: 'POST',\n    headers: {\n      'Content-Type': 'application/json',\n      'Authorization': \\`Bearer \\${token}\\`\n    },\n    body: JSON.stringify({\n      companyId,\n      mission: 'Build and deploy enterprise-grade AI agent solutions that automate complex business workflows and deliver measurable ROI within 90 days of customer onboarding'\n    })\n  });\n  const result = await response.json();\n  console.log('CEO Strategy Approved:', result);\n})();\n\"</parameter>\n</invoke>\n</function_calls>"}],"tokenLedger":{"input":497,"cache_read":0,"cache_creation":0,"output":770}}
+{"type": "assertions", "assertions": [{"kind": "message_contains_substring", "role": "assistant", "substring": "hire_agent"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "approve_ceo_strategy"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "bash"}, {"kind": "trace_max_turns", "max": 3}]}