Production regression of T1 (7829acc..15e3863). User reports "明明生成
了很多文件,但只有一个" — the filesystem shows 19 markdown deliverables
in ~/Documents/Github/test/{business-plan,operations,ceo-coordination}/
but the DocumentCenter surfaces only one row.
Root cause is structural, predicted in spirit by Karpathy's Round 1
take ("把'认知'和'执行'两件事一起锁是数据结构错"). The T1 rewrite
collapsed two concerns into one nullable field on the run context:
workspacePath = workspace?.localPath ?? null
^^^^^^^^^ ^^^^
lock target observation root
For lockMode='none' (every cognitive C-suite agent post-T1) the
resolver returns workspace=null, so workspacePath was null, so
artifact-projector.ts:242 short-circuited every projection with
`{ kind: "skipped", reason: "no_workspace" }`. The agents kept
writing files via bash (cwd falls back to the company default
workspace at worker/runtime/index.ts:142), but nothing observed
them — the DocumentCenter only saw the one file that was projected
back when the agent still held an exclusive workspace.
Fix: split the two concerns. WorkspaceResolution gains a non-lock
`observationPath` field; the resolver populates it from the company
primary workspace (new `getCompanyObservationWorkspace` query —
primary first, oldest second, null only when company has zero
workspace rows) for lockMode='none' runs, and from `workspace.localPath`
itself for lockMode='exclusive' runs. RunDispatchParams threads the
field through; turn-context's `workspacePath` reads it first.
artifact-projector.ts is unchanged — it just sees a non-null path
again and resumes baselining, diffing, and upserting work_products
rows the way it did pre-T1.
Three new unit tests pin the regression: cognitive role with
workspace ⇒ observationPath set; cognitive role with no workspace ⇒
observationPath null; exclusive ⇒ observationPath equals
workspace.localPath. typecheck green, 543/543 unit tests, 10/10
eval. No fallback, no mock, no silent degradation.
Old resolver: task → agent → project → any company workspace, all
marked exclusive. That fourth step folded every runnable agent in a
multi-agent company onto a single workspace row and serialised the
C-suite (four agents, one exclusive lock, three starved).
New resolver: lockMode='exclusive' is now an opt-in decision inferred
per task. Three rules:
1. task.workspaceId set → exclusive on that workspace (explicit).
2. filesystem signal (task.requiresFilesystem OR agent in a
coding role/engineering department) → exclusive on
agent.workspaceId / project primary; throws if neither exists.
3. otherwise → none / null. Pure cognitive work runs with no
workspace lock and fully in parallel.
Removed:
- company_fallback silent-share path. Filesystem-touching runs
with no legitimate binding now throw WorkspaceConfigurationError
at resolve time so the caller decides recovery rather than the
resolver picking a random shared workspace.
- workspacesMod.getAnyCompanyWorkspace (now dead code).
Invariants now enforced by types:
- lockMode='exclusive' ⇒ workspace is a live WorkspaceRecord.
- lockMode='none' ⇒ workspace is null.
Tests updated to cover the new decision table and the four-non-
coding-agents parallel case that was the original bug.
Closes the second and final ask from the C4 cross-review (Karpathy
specific ask #1). The previous non-ceo-dm-reply recording (409bec2)
pre-dated C4 part 3/5 by ~18 minutes, so it could not discriminate
between "non-CEO path correctly does not receive hire_agent /
file_approval" and "the tools simply had not been injected yet."
Re-recorded against anthropic/claude-sonnet-4-5 on the current main.
The VP Product agent (manager — canRequestHire=true by keyword match,
but after F3 canFileCeoStrategy=false because reportsTo is not null
AND role is not CEO) produces:
- mark_read + /api/agent_message/send + m_77 preserved — 3 positive
assertions intact.
- Zero hire_agent invocation (not asked to hire).
- Zero file_approval invocation — positive evidence the F3 split
correctly excludes file_approval from non-CEO managers. Without
F3 this call would have had file_approval in scope and a manager
under pressure might have reached for it.
Fixture assertions updated to pin the two tool-scope invariants using
the new message_does_not_contain_substring kind from F2.
C4-followup.md marked CLOSED with resolution table mapping each of
the four items to its landing commit, plus the companion cleanup
(43337e5 — RpcMethod single-source-of-truth) that came out of the
same session.
Final: typecheck green, 537 unit tests, 10 eval tests.
Old internal-tool-selector attached `file_approval` to any agent with
`reportsTo === null`, and attached both tools through the same
management-keyword predicate chain. That let a VP or director with
org-management authority walk away with `file_approval`, contradicting
the C4 commit narrative that scoped `approve_ceo_strategy` as CEO-only.
Introduce `canFileCeoStrategy(db, companyId, agentId)` as a dedicated
predicate requiring both a CEO-equivalent role/title AND
`reportsTo === null`. Using either half alone is too loose: role alone
lets a demoted CEO with a virtual superior still file strategy,
`reportsTo` alone lets a VP with no manager slip through. The two
conditions together pin the predicate to a genuine top-level CEO.
- `hire_agent` continues to gate on `canRequestHire` (managers hiring
ICs is a legitimate flow and the original "CEO-only" framing in the
C4 commit message was narrower than the production reality).
- `file_approval` now gates on `canFileCeoStrategy`, orthogonal to hire
authority.
- New unit test covers 4 edges: genuine CEO, CEO title with virtual
superior, VP without manager, and plain IC.
Closes the fourth ask from the C4 cross-review (docs/proposals/C4-followup.md
Item 4). The eval harness previously had no way to assert the absence of a
pattern in a trace — only positive `message_contains_substring`. That meant
the C4 "no more node -e / bash+fetch from CEO" invariant was only
demonstrated by the diff between two commits, not by any running assertion
that future regressions would trip.
Three changes:
1. Extend the Assertion union in tests/eval/types.ts with
`message_does_not_contain_substring` (role + substring). Symmetric to
the positive kind; fails on first match with a specific reason.
2. Add the case in assertion-judge.ts. Same traversal as the positive
branch but inverted exit condition.
3. Add two harness self-tests covering pass + fail, so the new kind is
exercised even when the fixtures directory is empty.
4. Pin two C4-regression assertions onto the recorded ceo-bootstrap-empty
fixture: the assistant response must NOT contain "node -e" OR
"/api/companies". Any future regression that reintroduces the
HTTP-path-in-prompt pattern or the bash+node -e shell fallback will
now fail pnpm test:eval instead of living in the fixture silently.
typecheck green, test:eval up from 8 → 10 tests (2 new self-tests), all
passing.
Closes review asks from cross-review of C4:
- Jobs ask #2: team-building.en.md still referenced the bash+fetch
pattern for goal creation. Rewritten as a deferred-tool note pointing
to C5 follow-up, without re-teaching the HTTP shape C4 just removed.
- Karpathy ask #1 (partial): re-recorded manager-task-handoff against
main to produce post-C4 positive evidence. VP Engineering has
hire_agent available via canRequestHire, but when directed NOT to
hire, the recorded trace shows task decomposition only (no
hire_agent invocation). 455 in / 769 out tokens; assertions updated
to pin /api/task/create + agent_ic_ slugs + p99 substring + max 3
turns. The `non-ceo-dm-reply` re-record is deferred to the follow-up
ticket.
- Karpathy asks #2, #3, #4 + Jobs asks #1, #3, #4: documented in
docs/proposals/C4-followup.md with concrete completion criteria.
Scope covers canFileCeoStrategy predicate, non-ceo-dm-reply
re-record, RFC 8594 Deprecation/Sunset headers + counter, and a
negative substring assertion kind for the eval harness.
Eval still green at 8/8. C4 main batch (e2c839b..8e99e01) stands
unchanged; this commit is pure follow-up groundwork.
This is the verification artefact for the C4 refactor (e2c839b through
03864f3). The old fixture (recorded at 1d0cfa2, scenario asked the model
to POST hire_agent requests via bash+node+fetch) captured the model
emitting `node -e 'fetch(\`http://...\`)'` — exactly the antipattern
Jobs called out.
Post-C4 scenario was updated to reflect the new world: the CEO now has
explicit hire_agent and file_approval tools. Re-recording against
anthropic/claude-sonnet-4-5 produced 509 in / 731 out tokens and the
assistant response:
- Contains `hire_agent` (pseudo-XML tool invocations, one per hire)
- Contains `file_approval` with type=approve_ceo_strategy
- No `node -e` / bash+fetch / HTTP POST pattern anywhere
- 3-5 hires as requested, single approval block
Assertions updated to pin the post-C4 tool names as positive signals.
Absence of bash/fetch is demonstrated by diff against 1d0cfa2 (see that
commit's recorded JSONL for the before state).
Eval harness now green at 8/8 (3 self-tests + 5 recorded fixtures) with
this new baseline. Any future regression that reverts CEO bootstrap to
HTTP-path prompts will fail all three hire_agent / approve_ceo_strategy /
file_approval assertions.
These are the role×trigger tripwires Karpathy required before C4 can
ship: covering IC / manager / non-CEO paths, not just the two CEO
fixtures from 1d0cfa2. Together they act as the cross-role regression
net — any C4 change that breaks hire_agent tool registration or
prompt shape across role tiers will now surface as an assertion
failure, not a silent production drift.
Fixtures recorded against anthropic/claude-sonnet-4-5 via the user's
Claude Max OAuth token (read from macOS keychain, not committed):
- ic-task-execute (364 in / 1425 out): IC receives task t_1138,
expects task id + filename in the response and a mark_complete
intent within 3 turns.
- manager-task-handoff (457 in / 1022 out): VP Engineering decomposes
CEO p99 directive into IC-sized tasks; expects /api/task/create
calls, references to agent_ic_* slugs, and the "p99" term.
- non-ceo-dm-reply (357 in / 545 out): VP Product handles a peer DM
with blocking ETA ask; expects mark_read + /api/agent_message/send
+ message id preserved.
Total spend ~3.7k tokens; pnpm test:eval --run now green at 8/8
(3 harness self-tests + 5 recorded fixtures).
With these tripwires in place, C4 (hire_agent / file_approval tool
uplift + initial-user-turn rewrite) can proceed without the previous
"only CEO path has a baseline" blind spot.
Unblocks the march-of-nines third nine. Up until this commit `tests/eval/`
had a harness + self-tests but zero real traces; the fixture directory was
a documented stub. These two fixtures are genuine single-turn recordings
against anthropic/claude-sonnet-4-5 produced by `pnpm eval:record` — not
mock data, not placeholder — each one captures an actual API response
with its token ledger.
Coverage:
- ceo-bootstrap-empty: empty company, CEO bootstrap trigger. Asserts the
model produces hire_agent POSTs, an approve_ceo_strategy POST, and uses
bash+fetch (current prompt pattern before C4 lands).
- ceo-board-message-reply: operating CEO with a live team, inbound Board
message. Asserts the model hits mark_read, posts to
/api/agent_message/send, and keeps the original message_id in scope.
What these fixtures now enable:
- B2 Phase 2 switch-read: a fixture diff before/after the cutover
demonstrates dual-write parity is real, not assumed.
- C4 initial-user-turn refactor: the current HTTP-path-in-prompt pattern
is captured; after C4 lands, the diff shows whether tool-shape
refactoring regressed behaviour.
- C7 prompt-builder conditional assembly: same, but for system-prompt
slimming.
- M7 Phase 2 call-site switch: resolveCeoMode cutover can be regression-
verified against both traces.
Scenarios live at tests/eval/scenarios/<id>/{system.txt,user.txt} so any
maintainer can re-record the same trace with one command and diff against
the committed fixture. Each fixture's assertions block currently contains
four assertions; pnpm test:eval passes with 5 (3 harness self-tests + 2
real-fixture runs).
Linus review T1 surfaced three real bugs in the B2 Phase 1 shadow
dual-write surface. Fix them in one pass.
1. advanceTaskStatus left task_leases behind on terminal (done /
cancelled / blocked) and "return to queue" (todo / backlog)
transitions. The status->phase trigger advanced tasks.phase but the
lease row lingered as an orphan. dev-mode verifyPhaseConsistency
could see the divergence; production silently drifted. Wrap the
status UPDATE and an explicit DELETE FROM task_leases in a single
transaction so no caller observes the intermediate state.
2. The no-restricted-syntax guard in eslint.config.mjs only matched
literals shaped "UPDATE tasks SET status" -- advanceTaskStatus's
dynamic `UPDATE tasks SET ${timestampSets.join(", ")}` slipped past,
and so did saveTask / recoverStaleLocks (status not first column).
Tighten the selectors to (a) match any status-write within UPDATE
tasks regardless of column order, and (b) flag template literals
whose head ends with `SET ` so dynamic helpers must carry an
explicit eslint-disable with rationale. Five grandfathered sites
now carry matching disables.
3. interruptStaleRuns scoped its task+lease cleanup with
`WHERE finished_at = ?` against the freshly stamped timestamp --
safe in isolation, vulnerable to ISO-millisecond ties with other
already-interrupted runs. Capture the stale run ids before
mutating anything and drive every downstream statement from that
explicit id list. Original semantics preserved; scoping is exact.
Unit tests added in tests/unit/tasks-phase.test.ts cover the
advanceTaskStatus lease cleanup across done / cancelled / blocked /
todo, and pin the in_review case as "lease stays".
ZacharyZhang-NY