Free tasks from the checkout_race_lost deadlock

Two bugs combined to park successful-but-deliverable-less tasks in an
unrecoverable state. Operator saw Alex Chen loop on
`checkout_race_lost` every two minutes for 16+ minutes in the field
log after completing its first task.

1. `processRunCompletion` in automation-run-completion.ts fired a no-op
   status transition (`in_progress` → `in_progress`) when a run
   succeeded without producing the expected deliverable. advanceTaskStatus
   no-ops on identical prev/next, and handleTaskStatusChange logged its
   `caller should gate this` warning and returned. finishRun had already
   released active_run_id and checkout_run_id, so the task sat at
   `in_progress` with no lock, no run — the exact orphan pattern that
   checkoutTask's `status IN ('todo','backlog')` clause refuses.
   Transition the row to `blocked` with the diagnostic comment attached
   so the manager's blocker queue surfaces it, the assignee stops
   spinning, and effectiveStatus carries the real state downstream.

2. `checkoutTask` itself refused to reclaim any orphan `in_progress`
   row, even when active_run_id was already null. That refusal was the
   right invariant against concurrent re-entry but wrong for the orphan
   case a crashed or status-unchanged run leaves behind. Extend the CAS
   predicate to accept `status = 'in_progress' AND active_run_id IS NULL`
   — heartbeat candidate selection already filters out tasks whose
   active_run_id points at a live run, so the guard preserves the
   original mutual-exclusion property for the busy case while allowing
   the orphan reclaim the scheduler actually needs.
This commit is contained in:
ZacharyZhang-NY committed 2026-04-19 21:09:16 -04:00
1 parent fdde562d55
commit 6b22fa9f0e
2 files changed
+39 -5

No files matched your search

+18 -1
View File
@@ -159,13 +159,30 @@ export function buildTaskStatusMethods(
checkoutTask(taskId: string, runId: string): boolean {
const ts = nowIso();
// Primary path is `status IN ('todo','backlog')`. The extra
// `in_progress AND active_run_id IS NULL` branch recovers from a
// specific orphan: finishRun has cleared active_run_id and
// checkout_run_id for the previous run but the status transition
// to done/in_review never fired (e.g. the run succeeded without a
// deliverable and automation parked it at in_progress, or a
// crashed run left the row behind). Without the reclaim, heartbeat
// picks the task (activeRunId=null passes the candidate filter)
// and then loops on `checkout_race_lost` forever because status
// never flips back to todo on its own.
//
// The `active_run_id IS NULL` guard preserves the original CAS
// contract for the busy case: a task whose active_run_id still
// points at a live run is refused here, exactly as before.
const result = getDb()
.prepare(
`UPDATE tasks
SET checkout_run_id = ?, execution_locked_at = ?, status = 'in_progress', updated_at = ?
WHERE id = ?
AND checkout_run_id IS NULL
AND status IN ('todo', 'backlog')`,
AND (
status IN ('todo', 'backlog')
OR (status = 'in_progress' AND active_run_id IS NULL)
)`,
)
.run(runId, ts, ts, taskId);
return result.changes > 0;