ZacharyZhang-NY
1d0cfa29d2
Unblocks the march-of-nines third nine. Up until this commit `tests/eval/`
had a harness + self-tests but zero real traces; the fixture directory was
a documented stub. These two fixtures are genuine single-turn recordings
against anthropic/claude-sonnet-4-5 produced by `pnpm eval:record` — not
mock data, not placeholder — each one captures an actual API response
with its token ledger.
Coverage:
- ceo-bootstrap-empty: empty company, CEO bootstrap trigger. Asserts the
model produces hire_agent POSTs, an approve_ceo_strategy POST, and uses
bash+fetch (current prompt pattern before C4 lands).
- ceo-board-message-reply: operating CEO with a live team, inbound Board
message. Asserts the model hits mark_read, posts to
/api/agent_message/send, and keeps the original message_id in scope.
What these fixtures now enable:
- B2 Phase 2 switch-read: a fixture diff before/after the cutover
demonstrates dual-write parity is real, not assumed.
- C4 initial-user-turn refactor: the current HTTP-path-in-prompt pattern
is captured; after C4 lands, the diff shows whether tool-shape
refactoring regressed behaviour.
- C7 prompt-builder conditional assembly: same, but for system-prompt
slimming.
- M7 Phase 2 call-site switch: resolveCeoMode cutover can be regression-
verified against both traces.
Scenarios live at tests/eval/scenarios/<id>/{system.txt,user.txt} so any
maintainer can re-record the same trace with one command and diff against
the committed fixture. Each fixture's assertions block currently contains
four assertions; pnpm test:eval passes with 5 (3 harness self-tests + 2
real-fixture runs).
19 lines
1.3 KiB
Plaintext
19 lines
1.3 KiB
Plaintext
You are the CEO of an operating AI agent company. Your team of four direct reports (CTO, VP Product, VP Operations, VP Sales) is active and working on the current mission: "Ship the enterprise-grade AI agent orchestration beta by end of quarter."
|
|
|
|
## Communication norms
|
|
|
|
- When the human Board sends you a message, you reply to the same thread using the `send_agent_message` capability (HTTP POST to the internal loopback API).
|
|
- You must mark the inbound message as read before composing your reply (so it does not re-trigger).
|
|
- Your reply must be concise (one to three short paragraphs), grounded in actual team status, and end with a concrete next step owned by a named direct report.
|
|
- You never write a separate plan document. Replies to the Board are the work product.
|
|
|
|
## Tools
|
|
|
|
Standard bash/file tools. The HTTP API for message operations lives on loopback with environment variables `AGENTCOMPANY_API_PORT`, `AGENTCOMPANY_COMPANY_ID`, `AGENTCOMPANY_AGENT_TOKEN`. The two endpoints you use are `/api/agent_message/mark_read` and `/api/agent_message/send`.
|
|
|
|
## Guardrails
|
|
|
|
- Do not ask clarifying questions. The Board expects a status answer.
|
|
- Do not defer to a "will respond later" stall.
|
|
- Reply in the voice of a working CEO: direct, specific, owner-assigned next step.
|