Symptom: scheduler logs `[heartbeat] Waking agent X (interval: 120s)`
every 15 seconds for three ICs, but no `[heartbeat] Starting taskless
heartbeat run` ever follows. Agents are wedged, UI shows nothing, and
the operator has no way to see why.
The loop had three compounding flaws:
1. `startAgentHeartbeatRun` returned `string | null`. Every gate
rejection was a silent `return null` — most were `logger.debug`,
which prod log levels filter out. Neither the scheduler nor the
renderer could tell the difference between "ran fine" and
"blocked by budget / no tasks / pending approval." The function
name promised "run or not" and the enum of reasons lived in
comments, not in the type.
2. The scheduler logged `[heartbeat] Waking agent ${name}` BEFORE
calling start, so even pure no-ops spammed the log. The "wake"
wasn't actually a wake; it was an attempt.
3. `markHeartbeatAt` is only called when a run starts. When start
returns null, `lastHeartbeatAt` stays frozen. On the next 15s
tick, `now - lastHeartbeatAt >= intervalMs` is still true, and
the scheduler fires again. The loop was a fixed-point at "skip →
log → skip → log" running 8 times per minute until something
external changed.
Root fix (data structure + visibility):
- `startAgentHeartbeatRun` now returns a discriminated union:
`{ status: "started"; runId } | { status: "skipped"; reason }`,
with `HeartbeatSkipReason` closed over 16 named reasons. Every
exit path carries its reason. The type tells operators what the
function can actually do; the enum tells them what it did.
- Scheduler holds a per-agent `attemptedAtByAgent` map. The interval
check uses `max(lastHeartbeatAt, lastAttemptedAt)`, so a streak of
skips still respects the heartbeat interval. Failed attempts cool
down the same as successful runs.
- On `skipped`, the scheduler persists `{reason, at}` into
`agents.runtime_state_json` under `heartbeatSkip`. On `started`,
it clears the field. The UI can read per-agent skip state from
the snapshot without a new IPC or table; operators finally see
"agent waiting: no_actionable_tasks" instead of a blank screen.
- The `Waking agent` log is gone. Its replacement prints once per
actual skip, with the reason inline (`[heartbeat] X not ready:
no_actionable_tasks`), and the existing `Starting taskless
heartbeat run for X` keeps marking the happy path.
`wakeAgentIfPossible` keeps its external `string | null` contract;
only the internal call site adapts. No caller outside of
`heartbeat.ts` / `scheduler-heartbeat.ts` / `scheduler.ts` needed
changes.
File sizes: heartbeat.ts 290, scheduler-heartbeat.ts 133. All under
the 400 cap. Tests: 603/603 green. Typecheck clean.
Split the single 15s setInterval in scheduler.ts (6 serial jobs) into 4
independent schedulers so a slow job no longer stalls the others:
- scheduler-heartbeat.ts (15s): wake eligible agents, CEO bootstrap,
schedule automation trigger
- scheduler-lock-recovery.ts (60s): stale task-lock recovery and stale
workspace-lock release
- scheduler-idle-automation.ts (60s): idle-agent automation dispatch
- scheduler-housekeeping.ts (300s): monthly budget reset, expired budget
incident auto-close, stale review sweep, expired wakeup purge
scheduler.ts is now a thin composer exposing startAllSchedulers /
stopAllSchedulers. Bootstrap and service-graph updated accordingly.
Business logic, error isolation, and atomic actions preserved.
ZacharyZhang-NY