Production regression of T1 (7829acc..15e3863). User reports "明明生成
了很多文件,但只有一个" — the filesystem shows 19 markdown deliverables
in ~/Documents/Github/test/{business-plan,operations,ceo-coordination}/
but the DocumentCenter surfaces only one row.
Root cause is structural, predicted in spirit by Karpathy's Round 1
take ("把'认知'和'执行'两件事一起锁是数据结构错"). The T1 rewrite
collapsed two concerns into one nullable field on the run context:
workspacePath = workspace?.localPath ?? null
^^^^^^^^^ ^^^^
lock target observation root
For lockMode='none' (every cognitive C-suite agent post-T1) the
resolver returns workspace=null, so workspacePath was null, so
artifact-projector.ts:242 short-circuited every projection with
`{ kind: "skipped", reason: "no_workspace" }`. The agents kept
writing files via bash (cwd falls back to the company default
workspace at worker/runtime/index.ts:142), but nothing observed
them — the DocumentCenter only saw the one file that was projected
back when the agent still held an exclusive workspace.
Fix: split the two concerns. WorkspaceResolution gains a non-lock
`observationPath` field; the resolver populates it from the company
primary workspace (new `getCompanyObservationWorkspace` query —
primary first, oldest second, null only when company has zero
workspace rows) for lockMode='none' runs, and from `workspace.localPath`
itself for lockMode='exclusive' runs. RunDispatchParams threads the
field through; turn-context's `workspacePath` reads it first.
artifact-projector.ts is unchanged — it just sees a non-null path
again and resumes baselining, diffing, and upserting work_products
rows the way it did pre-T1.
Three new unit tests pin the regression: cognitive role with
workspace ⇒ observationPath set; cognitive role with no workspace ⇒
observationPath null; exclusive ⇒ observationPath equals
workspace.localPath. typecheck green, 543/543 unit tests, 10/10
eval. No fallback, no mock, no silent degradation.
Old resolver: task → agent → project → any company workspace, all
marked exclusive. That fourth step folded every runnable agent in a
multi-agent company onto a single workspace row and serialised the
C-suite (four agents, one exclusive lock, three starved).
New resolver: lockMode='exclusive' is now an opt-in decision inferred
per task. Three rules:
1. task.workspaceId set → exclusive on that workspace (explicit).
2. filesystem signal (task.requiresFilesystem OR agent in a
coding role/engineering department) → exclusive on
agent.workspaceId / project primary; throws if neither exists.
3. otherwise → none / null. Pure cognitive work runs with no
workspace lock and fully in parallel.
Removed:
- company_fallback silent-share path. Filesystem-touching runs
with no legitimate binding now throw WorkspaceConfigurationError
at resolve time so the caller decides recovery rather than the
resolver picking a random shared workspace.
- workspacesMod.getAnyCompanyWorkspace (now dead code).
Invariants now enforced by types:
- lockMode='exclusive' ⇒ workspace is a live WorkspaceRecord.
- lockMode='none' ⇒ workspace is null.
Tests updated to cover the new decision table and the four-non-
coding-agents parallel case that was the original bug.
Closes the second and final ask from the C4 cross-review (Karpathy
specific ask #1). The previous non-ceo-dm-reply recording (409bec2)
pre-dated C4 part 3/5 by ~18 minutes, so it could not discriminate
between "non-CEO path correctly does not receive hire_agent /
file_approval" and "the tools simply had not been injected yet."
Re-recorded against anthropic/claude-sonnet-4-5 on the current main.
The VP Product agent (manager — canRequestHire=true by keyword match,
but after F3 canFileCeoStrategy=false because reportsTo is not null
AND role is not CEO) produces:
- mark_read + /api/agent_message/send + m_77 preserved — 3 positive
assertions intact.
- Zero hire_agent invocation (not asked to hire).
- Zero file_approval invocation — positive evidence the F3 split
correctly excludes file_approval from non-CEO managers. Without
F3 this call would have had file_approval in scope and a manager
under pressure might have reached for it.
Fixture assertions updated to pin the two tool-scope invariants using
the new message_does_not_contain_substring kind from F2.
C4-followup.md marked CLOSED with resolution table mapping each of
the four items to its landing commit, plus the companion cleanup
(43337e5 — RpcMethod single-source-of-truth) that came out of the
same session.
Final: typecheck green, 537 unit tests, 10 eval tests.
Old internal-tool-selector attached `file_approval` to any agent with
`reportsTo === null`, and attached both tools through the same
management-keyword predicate chain. That let a VP or director with
org-management authority walk away with `file_approval`, contradicting
the C4 commit narrative that scoped `approve_ceo_strategy` as CEO-only.
Introduce `canFileCeoStrategy(db, companyId, agentId)` as a dedicated
predicate requiring both a CEO-equivalent role/title AND
`reportsTo === null`. Using either half alone is too loose: role alone
lets a demoted CEO with a virtual superior still file strategy,
`reportsTo` alone lets a VP with no manager slip through. The two
conditions together pin the predicate to a genuine top-level CEO.
- `hire_agent` continues to gate on `canRequestHire` (managers hiring
ICs is a legitimate flow and the original "CEO-only" framing in the
C4 commit message was narrower than the production reality).
- `file_approval` now gates on `canFileCeoStrategy`, orthogonal to hire
authority.
- New unit test covers 4 edges: genuine CEO, CEO title with virtual
superior, VP without manager, and plain IC.
Closes the fourth ask from the C4 cross-review (docs/proposals/C4-followup.md
Item 4). The eval harness previously had no way to assert the absence of a
pattern in a trace — only positive `message_contains_substring`. That meant
the C4 "no more node -e / bash+fetch from CEO" invariant was only
demonstrated by the diff between two commits, not by any running assertion
that future regressions would trip.
Three changes:
1. Extend the Assertion union in tests/eval/types.ts with
`message_does_not_contain_substring` (role + substring). Symmetric to
the positive kind; fails on first match with a specific reason.
2. Add the case in assertion-judge.ts. Same traversal as the positive
branch but inverted exit condition.
3. Add two harness self-tests covering pass + fail, so the new kind is
exercised even when the fixtures directory is empty.
4. Pin two C4-regression assertions onto the recorded ceo-bootstrap-empty
fixture: the assistant response must NOT contain "node -e" OR
"/api/companies". Any future regression that reintroduces the
HTTP-path-in-prompt pattern or the bash+node -e shell fallback will
now fail pnpm test:eval instead of living in the fixture silently.
typecheck green, test:eval up from 8 → 10 tests (2 new self-tests), all
passing.
Closes review asks from cross-review of C4:
- Jobs ask #2: team-building.en.md still referenced the bash+fetch
pattern for goal creation. Rewritten as a deferred-tool note pointing
to C5 follow-up, without re-teaching the HTTP shape C4 just removed.
- Karpathy ask #1 (partial): re-recorded manager-task-handoff against
main to produce post-C4 positive evidence. VP Engineering has
hire_agent available via canRequestHire, but when directed NOT to
hire, the recorded trace shows task decomposition only (no
hire_agent invocation). 455 in / 769 out tokens; assertions updated
to pin /api/task/create + agent_ic_ slugs + p99 substring + max 3
turns. The `non-ceo-dm-reply` re-record is deferred to the follow-up
ticket.
- Karpathy asks #2, #3, #4 + Jobs asks #1, #3, #4: documented in
docs/proposals/C4-followup.md with concrete completion criteria.
Scope covers canFileCeoStrategy predicate, non-ceo-dm-reply
re-record, RFC 8594 Deprecation/Sunset headers + counter, and a
negative substring assertion kind for the eval harness.
Eval still green at 8/8. C4 main batch (e2c839b..8e99e01) stands
unchanged; this commit is pure follow-up groundwork.
This is the verification artefact for the C4 refactor (e2c839b through
03864f3). The old fixture (recorded at 1d0cfa2, scenario asked the model
to POST hire_agent requests via bash+node+fetch) captured the model
emitting `node -e 'fetch(\`http://...\`)'` — exactly the antipattern
Jobs called out.
Post-C4 scenario was updated to reflect the new world: the CEO now has
explicit hire_agent and file_approval tools. Re-recording against
anthropic/claude-sonnet-4-5 produced 509 in / 731 out tokens and the
assistant response:
- Contains `hire_agent` (pseudo-XML tool invocations, one per hire)
- Contains `file_approval` with type=approve_ceo_strategy
- No `node -e` / bash+fetch / HTTP POST pattern anywhere
- 3-5 hires as requested, single approval block
Assertions updated to pin the post-C4 tool names as positive signals.
Absence of bash/fetch is demonstrated by diff against 1d0cfa2 (see that
commit's recorded JSONL for the before state).
Eval harness now green at 8/8 (3 self-tests + 5 recorded fixtures) with
this new baseline. Any future regression that reverts CEO bootstrap to
HTTP-path prompts will fail all three hire_agent / approve_ceo_strategy /
file_approval assertions.
These are the role×trigger tripwires Karpathy required before C4 can
ship: covering IC / manager / non-CEO paths, not just the two CEO
fixtures from 1d0cfa2. Together they act as the cross-role regression
net — any C4 change that breaks hire_agent tool registration or
prompt shape across role tiers will now surface as an assertion
failure, not a silent production drift.
Fixtures recorded against anthropic/claude-sonnet-4-5 via the user's
Claude Max OAuth token (read from macOS keychain, not committed):
- ic-task-execute (364 in / 1425 out): IC receives task t_1138,
expects task id + filename in the response and a mark_complete
intent within 3 turns.
- manager-task-handoff (457 in / 1022 out): VP Engineering decomposes
CEO p99 directive into IC-sized tasks; expects /api/task/create
calls, references to agent_ic_* slugs, and the "p99" term.
- non-ceo-dm-reply (357 in / 545 out): VP Product handles a peer DM
with blocking ETA ask; expects mark_read + /api/agent_message/send
+ message id preserved.
Total spend ~3.7k tokens; pnpm test:eval --run now green at 8/8
(3 harness self-tests + 5 recorded fixtures).
With these tripwires in place, C4 (hire_agent / file_approval tool
uplift + initial-user-turn rewrite) can proceed without the previous
"only CEO path has a baseline" blind spot.
Unblocks the march-of-nines third nine. Up until this commit `tests/eval/`
had a harness + self-tests but zero real traces; the fixture directory was
a documented stub. These two fixtures are genuine single-turn recordings
against anthropic/claude-sonnet-4-5 produced by `pnpm eval:record` — not
mock data, not placeholder — each one captures an actual API response
with its token ledger.
Coverage:
- ceo-bootstrap-empty: empty company, CEO bootstrap trigger. Asserts the
model produces hire_agent POSTs, an approve_ceo_strategy POST, and uses
bash+fetch (current prompt pattern before C4 lands).
- ceo-board-message-reply: operating CEO with a live team, inbound Board
message. Asserts the model hits mark_read, posts to
/api/agent_message/send, and keeps the original message_id in scope.
What these fixtures now enable:
- B2 Phase 2 switch-read: a fixture diff before/after the cutover
demonstrates dual-write parity is real, not assumed.
- C4 initial-user-turn refactor: the current HTTP-path-in-prompt pattern
is captured; after C4 lands, the diff shows whether tool-shape
refactoring regressed behaviour.
- C7 prompt-builder conditional assembly: same, but for system-prompt
slimming.
- M7 Phase 2 call-site switch: resolveCeoMode cutover can be regression-
verified against both traces.
Scenarios live at tests/eval/scenarios/<id>/{system.txt,user.txt} so any
maintainer can re-record the same trace with one command and diff against
the committed fixture. Each fixture's assertions block currently contains
four assertions; pnpm test:eval passes with 5 (3 harness self-tests + 2
real-fixture runs).
Linus review T1 surfaced three real bugs in the B2 Phase 1 shadow
dual-write surface. Fix them in one pass.
1. advanceTaskStatus left task_leases behind on terminal (done /
cancelled / blocked) and "return to queue" (todo / backlog)
transitions. The status->phase trigger advanced tasks.phase but the
lease row lingered as an orphan. dev-mode verifyPhaseConsistency
could see the divergence; production silently drifted. Wrap the
status UPDATE and an explicit DELETE FROM task_leases in a single
transaction so no caller observes the intermediate state.
2. The no-restricted-syntax guard in eslint.config.mjs only matched
literals shaped "UPDATE tasks SET status" -- advanceTaskStatus's
dynamic `UPDATE tasks SET ${timestampSets.join(", ")}` slipped past,
and so did saveTask / recoverStaleLocks (status not first column).
Tighten the selectors to (a) match any status-write within UPDATE
tasks regardless of column order, and (b) flag template literals
whose head ends with `SET ` so dynamic helpers must carry an
explicit eslint-disable with rationale. Five grandfathered sites
now carry matching disables.
3. interruptStaleRuns scoped its task+lease cleanup with
`WHERE finished_at = ?` against the freshly stamped timestamp --
safe in isolation, vulnerable to ISO-millisecond ties with other
already-interrupted runs. Capture the stale run ids before
mutating anything and drive every downstream statement from that
explicit id list. Original semantics preserved; scoping is exact.
Unit tests added in tests/unit/tasks-phase.test.ts cover the
advanceTaskStatus lease cleanup across done / cancelled / blocked /
todo, and pin the in_review case as "lease stays".
Migrate all remaining pass-through call sites (social, skills, budget,
entityCrud, documents, knowledge, automation, wakeup, backup, threads,
settings, snapshot, connectors, workspaces, activity) to their owning
domain modules. `AppDatabase` now carries only three methods:
- createHireRequest (multi-module composite: agents + approvalsMod)
- rollbackConfigRevision (multi-module composite: documents + entityCrud)
- queryAll (raw-SQL escape hatch, @internal)
The vault/env-binding infrastructure previously passed argument-by-
argument into `entityCrud.saveSecret`/`resolveSecretEnv` now binds at
module construction, letting callers invoke `db.entityCrud.saveSecret({...})`
without carrying vault plumbing.
Structural types that narrowed AppDatabase to a bag of flat methods
(`ConnectorModelPolicyDeps` callers, `CapabilityMatchDb`, test mocks in
budget-enforcement / budget-incidents / task-quality-gates /
failure-escalation / task-quality-gates-sweep) are reshaped to the new
namespace. `connectorPolicyDepsFromDb` replaces the old pattern of
handing the whole AppDatabase to `validateConnectorModelSelection`.
A new ESLint rule in eslint.config.mjs blocks new top-level methods on
AppDatabase so the pass-through layer cannot silently grow back.
`AppDatabase.ts` dropped from 193 lines to 58; `AppDatabaseBase` lost
the 270-line trailing pass-through block.
Rewrite ~376 call sites for messages, runs, approvalsMod, and company
domains and delete the corresponding pass-through methods on
AppDatabase / AppDatabaseBase. Structural type references
(ReturnType<HandlerDeps["db"]["getRun"]>) and SweepStaleReviewsDeps-style
mocks are updated to route through the owning sub-module.
Rewrite ~525 call sites from `db.<method>()` to `db.<module>.<method>()`
for tasks and agents domains (including `getAgent`/`findAgent`/`getTask`
which live on the `entityCrud` field). Delete the corresponding
pass-through methods on AppDatabaseBase. Structural types that narrowed
`db` to a bag of flat methods (CapabilityMatchDb, EscalationDeps,
SweepStaleReviewsDeps) are reshaped to match the new namespace. Internal
callback references inside database.ts/database-base.ts now invoke the
owning module directly (e.g. `this.agents.saveAgent`) rather than the
deleted pass-through.
The codemod lives at scripts/codemod/remove-appdatabase-passthrough.mjs
and filters by owning module + optional explicit method list. It is
idempotent and does not re-rewrite already-namespaced calls.
Consolidate the two runtime messaging modules under src/main/messaging/. Storage stays in database/messages.ts — that is the SQL layer and belongs with the other DB modules; messaging/ covers the dispatch/normalize runtime around it.
- message-dispatch.ts -> messaging/dispatch.ts
- message-normalize.ts -> messaging/normalize.ts
- add messaging/index.ts barrel for future consumers (existing call sites import the concrete module path directly)
- 10 call sites updated across src/main and tests
Replaces the stub recorder with a real-provider CLI. The command now:
1. Reads system prompt + user message from files on disk (empty files
are rejected — a fixture with no prompt captures no behaviour).
2. Resolves the provider spec from `--provider anthropic|openai|google`
and pulls the API key from the matching environment variable.
Missing env → hard error with a copy-pasteable export hint; no
fallback, no mock.
3. Calls `streamSimple` from @mariozechner/pi-ai with the user-picked
model. On provider error (stopReason=error|aborted) the CLI exits
non-zero and writes no JSONL. Users re-run if they think the
failure was transient.
4. Serialises one user turn + one assistant turn + an empty
assertions block into the JSONL shape documented in
tests/eval/README.md. Token ledger comes straight from the
assistant message's usage field (renamed to snake_case for
provider-neutral on-disk format).
Scope is deliberately single-turn: tool use / multi-turn agent loops
are B5.1's job. Importing the AgentCompany worker's system prompt
builder would require the Electron runtime; the CLI stays node-only
so it can run standalone.
`--help` works and prints the argument matrix. All failure modes exit
non-zero with a specific, actionable message.
Also document the new recording workflow in tests/eval/README.md —
four-step procedure, example scenario files, assertion templates, and
explicit list of what the recorder still does not do.
- harness.test.ts: add in-memory self-test layer (pass + fail + purity), guard
fixtures dir with existsSync so absent dir yields empty list without
crashing the runner. Empty-fixture pnpm test:eval now carries real signal.
- replay-client.ts: remove dead if (!turn) branch; cursor bounds-check above
already guarantees the element. Use non-null assertion with a comment.
- assertion-judge.ts: drop unnecessary (turn as AssistantTurn | { content:
unknown[] }) cast. turn.role === assertion.role already narrows to
UserTurn | AssistantTurn and both expose .content.
- _BLOCKED-BY-CONSTRAINT.md: correct electron-builder note. files: is a
whitelist, so tests/eval is automatically excluded and no new rule is
required.
Collects the five post-run diagnostic + escalation modules into src/main/worker/observability/: run_events persistence, stdout/stderr noise filter, credential-shape log redactor, self-wake circuit breaker, and retry-aware failure escalation pipeline. circuit-breaker.ts and failure-escalation.ts remain as standalone files inside the directory — each owns a distinct state machine that deserves its own boundary — rather than collapsing into a single index.ts. handlers/*.ts and tests/unit/*.test.ts updated to point at the new paths.
Land the B5 harness skeleton: fixture loader (strict JSONL contract with
meta header + ordered turn records + assertions tail), replay-client that
swaps in for pi-ai's streamSimple by replaying recorded assistant turns,
structural assertion judge (tool_called / no_tool_called /
message_contains_substring / tool_call_arg_matches / trace_max_turns), an
eval-runner that sums the token ledger and produces an EvalResult, a
vitest driver (tests/eval/harness.test.ts) that discovers *.jsonl
fixtures, and a record-fixture CLI that writes fixture meta headers.
Hard contracts:
- No fallback: replay-client and fixture-loader throw on exhaustion /
malformed input rather than synthesizing defaults.
- No mock: types reuse @mariozechner/pi-ai AgentMessage / ToolCall
verbatim; shadow types are forbidden.
- No fake fixture data: tests/eval/fixtures/ ships only README +
.gitkeep. Real traces are recorded by the user against live providers.
package.json adds test:eval (vitest --passWithNoTests so an empty
fixture dir is green) and eval:record (CLI entrypoint). electron-builder
already excludes tests/ via its files: glob — fixtures never enter the
shipped .asar.
ZacharyZhang-NY