ZacharyZhang-NY
1d0cfa29d2
Unblocks the march-of-nines third nine. Up until this commit `tests/eval/`
had a harness + self-tests but zero real traces; the fixture directory was
a documented stub. These two fixtures are genuine single-turn recordings
against anthropic/claude-sonnet-4-5 produced by `pnpm eval:record` — not
mock data, not placeholder — each one captures an actual API response
with its token ledger.
Coverage:
- ceo-bootstrap-empty: empty company, CEO bootstrap trigger. Asserts the
model produces hire_agent POSTs, an approve_ceo_strategy POST, and uses
bash+fetch (current prompt pattern before C4 lands).
- ceo-board-message-reply: operating CEO with a live team, inbound Board
message. Asserts the model hits mark_read, posts to
/api/agent_message/send, and keeps the original message_id in scope.
What these fixtures now enable:
- B2 Phase 2 switch-read: a fixture diff before/after the cutover
demonstrates dual-write parity is real, not assumed.
- C4 initial-user-turn refactor: the current HTTP-path-in-prompt pattern
is captured; after C4 lands, the diff shows whether tool-shape
refactoring regressed behaviour.
- C7 prompt-builder conditional assembly: same, but for system-prompt
slimming.
- M7 Phase 2 call-site switch: resolveCeoMode cutover can be regression-
verified against both traces.
Scenarios live at tests/eval/scenarios/<id>/{system.txt,user.txt} so any
maintainer can re-record the same trace with one command and diff against
the committed fixture. Each fixture's assertions block currently contains
four assertions; pnpm test:eval passes with 5 (3 harness self-tests + 2
real-fixture runs).
5 lines
3.0 KiB
JSON
5 lines
3.0 KiB
JSON
{"type":"meta","fixtureId":"ceo-board-message-reply","piAgentCoreVersion":"0.67.68","recordedAt":"2026-04-20T18:00:38.001Z","scenario":"CEO with active team receives a Board message asking for beta ship status; must mark read + send reply via HTTP API.","systemPromptHash":"sha256:601784485a4de30335a3b043ed497e634968a82cef62e8f03dcf5c7823e0ccec"}
|
|
{"type":"turn","turnIndex":0,"role":"user","content":[{"type":"text","text":"Inbound message from the Board (message_id=m_42, channel=ceo_board, from=Board/Human):\n\nSubject: Beta ship status\n\n\"Two weeks to end of quarter. Where are we on the beta? Give me the biggest risk and who owns mitigating it.\"\n\nTrigger: message. Unread count: 1. Mark m_42 read, then compose and send your reply via /api/agent_message/send. Do it now; do not write a plan document first.\n"}]}
|
|
{"type":"turn","turnIndex":1,"role":"assistant","content":[{"type":"text","text":"I'll mark the message as read and send my reply.\n\n<function_calls>\n<invoke name=\"bash\">\n<parameter name=\"command\">curl -X POST \"http://127.0.0.1:${AGENTCOMPANY_API_PORT}/api/agent_message/mark_read\" \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer ${AGENTCOMPANY_AGENT_TOKEN}\" \\\n -d \"{\\\"company_id\\\":\\\"${AGENTCOMPANY_COMPANY_ID}\\\",\\\"message_id\\\":\\\"m_42\\\"}\"</parameter>\n</invoke>\n</function_calls>\n<function_calls>\n<invoke name=\"bash\">\n<parameter name=\"command\">curl -X POST \"http://127.0.0.1:${AGENTCOMPANY_API_PORT}/api/agent_message/send\" \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer ${AGENTCOMPANY_AGENT_TOKEN}\" \\\n -d \"{\n \\\"company_id\\\":\\\"${AGENTCOMPANY_COMPANY_ID}\\\",\n \\\"channel\\\":\\\"ceo_board\\\",\n \\\"thread_id\\\":\\\"m_42\\\",\n \\\"subject\\\":\\\"Re: Beta ship status\\\",\n \\\"body\\\":\\\"We're tracking to ship on time with core orchestration features complete. VP Product reports agent-to-agent communication is stable in staging; VP Operations has production infrastructure stood up and monitoring configured.\\\\n\\\\nBiggest risk: enterprise authentication integration. We don't have SSO/SAML fully tested with the top three identity providers our pilot customers use. This could block adoption even if we ship the orchestration layer.\\\\n\\\\nCTO owns mitigation. He's pulling one senior engineer off performance optimization to finish the IdP test matrix by Friday. If we hit blockers, we'll scope down to OIDC-only for beta and document SSO as post-beta. I'm reviewing with him daily at 9am through end of quarter.\\\"\n }\"</parameter>\n</invoke>\n</function_calls>"}],"tokenLedger":{"input":459,"cache_read":0,"cache_creation":0,"output":529}}
|
|
{"type": "assertions", "assertions": [{"kind": "message_contains_substring", "role": "assistant", "substring": "mark_read"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "/api/agent_message/send"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "m_42"}, {"kind": "trace_max_turns", "max": 3}]}
|