ZacharyZhang-NY
9a5823b242
Closes the fourth ask from the C4 cross-review (docs/proposals/C4-followup.md Item 4). The eval harness previously had no way to assert the absence of a pattern in a trace — only positive `message_contains_substring`. That meant the C4 "no more node -e / bash+fetch from CEO" invariant was only demonstrated by the diff between two commits, not by any running assertion that future regressions would trip. Three changes: 1. Extend the Assertion union in tests/eval/types.ts with `message_does_not_contain_substring` (role + substring). Symmetric to the positive kind; fails on first match with a specific reason. 2. Add the case in assertion-judge.ts. Same traversal as the positive branch but inverted exit condition. 3. Add two harness self-tests covering pass + fail, so the new kind is exercised even when the fixtures directory is empty. 4. Pin two C4-regression assertions onto the recorded ceo-bootstrap-empty fixture: the assistant response must NOT contain "node -e" OR "/api/companies". Any future regression that reintroduces the HTTP-path-in-prompt pattern or the bash+node -e shell fallback will now fail pnpm test:eval instead of living in the fixture silently. typecheck green, test:eval up from 8 → 10 tests (2 new self-tests), all passing.
5 lines
3.8 KiB
JSON
5 lines
3.8 KiB
JSON
{"type":"meta","fixtureId":"ceo-bootstrap-empty","piAgentCoreVersion":"0.67.68","recordedAt":"2026-04-20T18:30:58.219Z","scenario":"Post-C4 baseline: CEO bootstrap with hire_agent and file_approval tools available. Expected: model uses tool names (not bash/fetch/node -e) for the 3-5 hires + 1 approval.","systemPromptHash":"sha256:8125df952f7fb3b94e5e7086ce4a86938a5ef4c9d5c867feba76e480de2e95c4"}
|
|
{"type":"turn","turnIndex":0,"role":"user","content":[{"type":"text","text":"Trigger: bootstrap wake. Zero active direct reports. Start Team Building now — file three to five hire_agent POSTs and one approve_ceo_strategy POST against the internal HTTP API. Do not write a plan document. Execute tool calls.\n"}]}
|
|
{"type":"turn","turnIndex":1,"role":"assistant","content":[{"type":"text","text":"I'll hire a focused team of 3-5 agents to cover core company functions, then file the strategy approval.\n\n<hire_agent>\n<name>Sarah Chen</name>\n<role>Chief Technology Officer</role>\n<title>CTO</title>\n<department>Engineering</department>\n<connectorId>anthropic-oauth</connectorId>\n<model>claude-sonnet-4-5</model>\n<reportsTo>CEO</reportsTo>\n<capabilities>Technical architecture, AI/ML infrastructure, engineering leadership, product development</capabilities>\n<budgetMonthlyUsd>15000</budgetMonthlyUsd>\n</hire_agent>\n\n<hire_agent>\n<name>Marcus Williams</name>\n<role>Chief Operating Officer</role>\n<title>COO</title>\n<department>Operations</department>\n<connectorId>anthropic-oauth</connectorId>\n<model>claude-sonnet-4-5</model>\n<reportsTo>CEO</reportsTo>\n<capabilities>Business operations, process optimization, resource allocation, cross-functional coordination</capabilities>\n<budgetMonthlyUsd>12000</budgetMonthlyUsd>\n</hire_agent>\n\n<hire_agent>\n<name>Jessica Rodriguez</name>\n<role>Chief Product Officer</role>\n<title>CPO</title>\n<department>Product</department>\n<connectorId>anthropic-oauth</connectorId>\n<model>claude-sonnet-4-5</model>\n<reportsTo>CEO</reportsTo>\n<capabilities>Product strategy, user research, roadmap planning, go-to-market execution</capabilities>\n<budgetMonthlyUsd>12000</budgetMonthlyUsd>\n</hire_agent>\n\n<hire_agent>\n<name>David Park</name>\n<role>Head of Research</role>\n<title>Head of Research</title>\n<department>Research</department>\n<connectorId>openai-apikey</connectorId>\n<model>gpt-4o-mini</model>\n<reportsTo>CEO</reportsTo>\n<capabilities>AI research, competitive analysis, innovation strategy, technical documentation</capabilities>\n<budgetMonthlyUsd>8000</budgetMonthlyUsd>\n</hire_agent>\n\n<file_approval>\n<type>approve_ceo_strategy</type>\n<payloadSummary>Four C-level direct reports to CEO: Sarah Chen (CTO/Engineering), Marcus Williams (COO/Operations), Jessica Rodriguez (CPO/Product), David Park (Head of Research). Flat structure enables rapid decision-making.</payloadSummary>\n<impactSummary>This leadership team balances technical execution (CTO), operational efficiency (COO), market fit (CPO), and innovation (Research). Four direct reports keep communication tight while covering all critical functions for an AI agent company. Mix of Anthropic and OpenAI models optimizes cost vs capability. Total monthly budget $47K allows scaling sub-teams as needed.</impactSummary>\n</file_approval>"}],"tokenLedger":{"input":509,"cache_read":0,"cache_creation":0,"output":731}}
|
|
{"type": "assertions", "assertions": [{"kind": "message_contains_substring", "role": "assistant", "substring": "hire_agent"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "approve_ceo_strategy"}, {"kind": "message_contains_substring", "role": "assistant", "substring": "file_approval"}, {"kind": "trace_max_turns", "max": 3}, {"kind": "message_does_not_contain_substring", "role": "assistant", "substring": "node -e"}, {"kind": "message_does_not_contain_substring", "role": "assistant", "substring": "/api/companies"}]}
|