Commit Graph
3 Commits
Author SHA1 Message Date
ZacharyZhang-NY 04c4059159 test(eval): add semantic rubric gate 2026-04-23 19:31:31 -04:00
ZacharyZhang-NY 17ffec7ea5 C4 follow-up: re-record manager fixture post-C4, strip bash leftover from SOP, document remaining asks as C4-followup proposal
Closes review asks from cross-review of C4:

- Jobs ask #2: team-building.en.md still referenced the bash+fetch
  pattern for goal creation. Rewritten as a deferred-tool note pointing
  to C5 follow-up, without re-teaching the HTTP shape C4 just removed.

- Karpathy ask #1 (partial): re-recorded manager-task-handoff against
  main to produce post-C4 positive evidence. VP Engineering has
  hire_agent available via canRequestHire, but when directed NOT to
  hire, the recorded trace shows task decomposition only (no
  hire_agent invocation). 455 in / 769 out tokens; assertions updated
  to pin /api/task/create + agent_ic_ slugs + p99 substring + max 3
  turns. The `non-ceo-dm-reply` re-record is deferred to the follow-up
  ticket.

- Karpathy asks #2, #3, #4 + Jobs asks #1, #3, #4: documented in
  docs/proposals/C4-followup.md with concrete completion criteria.
  Scope covers canFileCeoStrategy predicate, non-ceo-dm-reply
  re-record, RFC 8594 Deprecation/Sunset headers + counter, and a
  negative substring assertion kind for the eval harness.

Eval still green at 8/8. C4 main batch (e2c839b..8e99e01) stands
unchanged; this commit is pure follow-up groundwork.
2026-04-20 14:36:00 -04:00
ZacharyZhang-NY 409bec2b25 Record 3 new B5 fixtures: IC task exec, manager handoff, non-CEO DM
These are the role×trigger tripwires Karpathy required before C4 can
ship: covering IC / manager / non-CEO paths, not just the two CEO
fixtures from 1d0cfa2. Together they act as the cross-role regression
net — any C4 change that breaks hire_agent tool registration or
prompt shape across role tiers will now surface as an assertion
failure, not a silent production drift.

Fixtures recorded against anthropic/claude-sonnet-4-5 via the user's
Claude Max OAuth token (read from macOS keychain, not committed):

- ic-task-execute (364 in / 1425 out): IC receives task t_1138,
  expects task id + filename in the response and a mark_complete
  intent within 3 turns.
- manager-task-handoff (457 in / 1022 out): VP Engineering decomposes
  CEO p99 directive into IC-sized tasks; expects /api/task/create
  calls, references to agent_ic_* slugs, and the "p99" term.
- non-ceo-dm-reply (357 in / 545 out): VP Product handles a peer DM
  with blocking ETA ask; expects mark_read + /api/agent_message/send
  + message id preserved.

Total spend ~3.7k tokens; pnpm test:eval --run now green at 8/8
(3 harness self-tests + 5 recorded fixtures).

With these tripwires in place, C4 (hire_agent / file_approval tool
uplift + initial-user-turn rewrite) can proceed without the previous
"only CEO path has a baseline" blind spot.
2026-04-20 14:08:35 -04:00