ZacharyZhang-NY
1261c2431f
Symptom: scheduler logs `[heartbeat] Waking agent X (interval: 120s)`
every 15 seconds for three ICs, but no `[heartbeat] Starting taskless
heartbeat run` ever follows. Agents are wedged, UI shows nothing, and
the operator has no way to see why.
The loop had three compounding flaws:
1. `startAgentHeartbeatRun` returned `string | null`. Every gate
rejection was a silent `return null` — most were `logger.debug`,
which prod log levels filter out. Neither the scheduler nor the
renderer could tell the difference between "ran fine" and
"blocked by budget / no tasks / pending approval." The function
name promised "run or not" and the enum of reasons lived in
comments, not in the type.
2. The scheduler logged `[heartbeat] Waking agent ${name}` BEFORE
calling start, so even pure no-ops spammed the log. The "wake"
wasn't actually a wake; it was an attempt.
3. `markHeartbeatAt` is only called when a run starts. When start
returns null, `lastHeartbeatAt` stays frozen. On the next 15s
tick, `now - lastHeartbeatAt >= intervalMs` is still true, and
the scheduler fires again. The loop was a fixed-point at "skip →
log → skip → log" running 8 times per minute until something
external changed.
Root fix (data structure + visibility):
- `startAgentHeartbeatRun` now returns a discriminated union:
`{ status: "started"; runId } | { status: "skipped"; reason }`,
with `HeartbeatSkipReason` closed over 16 named reasons. Every
exit path carries its reason. The type tells operators what the
function can actually do; the enum tells them what it did.
- Scheduler holds a per-agent `attemptedAtByAgent` map. The interval
check uses `max(lastHeartbeatAt, lastAttemptedAt)`, so a streak of
skips still respects the heartbeat interval. Failed attempts cool
down the same as successful runs.
- On `skipped`, the scheduler persists `{reason, at}` into
`agents.runtime_state_json` under `heartbeatSkip`. On `started`,
it clears the field. The UI can read per-agent skip state from
the snapshot without a new IPC or table; operators finally see
"agent waiting: no_actionable_tasks" instead of a blank screen.
- The `Waking agent` log is gone. Its replacement prints once per
actual skip, with the reason inline (`[heartbeat] X not ready:
no_actionable_tasks`), and the existing `Starting taskless
heartbeat run for X` keeps marking the happy path.
`wakeAgentIfPossible` keeps its external `string | null` contract;
only the internal call site adapts. No caller outside of
`heartbeat.ts` / `scheduler-heartbeat.ts` / `scheduler.ts` needed
changes.
File sizes: heartbeat.ts 290, scheduler-heartbeat.ts 133. All under
the 400 cap. Tests: 603/603 green. Typecheck clean.
35 lines
1.4 KiB
TypeScript
35 lines
1.4 KiB
TypeScript
import type { OrchestrationDependencies } from "./types";
|
|
import type { HeartbeatStartResult } from "./heartbeat";
|
|
import { createHeartbeatScheduler } from "./scheduler-heartbeat";
|
|
import { createLockRecoveryScheduler } from "./scheduler-lock-recovery";
|
|
import { createIdleAutomationScheduler } from "./scheduler-idle-automation";
|
|
import { createHousekeepingScheduler } from "./scheduler-housekeeping";
|
|
|
|
export function createScheduler(
|
|
deps: OrchestrationDependencies,
|
|
hasActiveRun: (filter: { taskId?: string; agentId?: string; companyId: string }) => boolean,
|
|
startAgentHeartbeatRun: (agentId: string, companyId: string, trigger: string) => HeartbeatStartResult,
|
|
wakeAgentIfPossible: (agentId: string, companyId: string, trigger: string) => string | null | undefined,
|
|
) {
|
|
const heartbeat = createHeartbeatScheduler(deps, hasActiveRun, startAgentHeartbeatRun);
|
|
const lockRecovery = createLockRecoveryScheduler(deps);
|
|
const idleAutomation = createIdleAutomationScheduler(deps);
|
|
const housekeeping = createHousekeepingScheduler(deps, wakeAgentIfPossible);
|
|
|
|
function startAllSchedulers() {
|
|
heartbeat.start();
|
|
lockRecovery.start();
|
|
idleAutomation.start();
|
|
housekeeping.start();
|
|
}
|
|
|
|
function stopAllSchedulers() {
|
|
heartbeat.stop();
|
|
lockRecovery.stop();
|
|
idleAutomation.stop();
|
|
housekeeping.stop();
|
|
}
|
|
|
|
return { startAllSchedulers, stopAllSchedulers };
|
|
}
|