Free tasks from the checkout_race_lost deadlock

Two bugs combined to park successful-but-deliverable-less tasks in an
unrecoverable state. Operator saw Alex Chen loop on
`checkout_race_lost` every two minutes for 16+ minutes in the field
log after completing its first task.

1. `processRunCompletion` in automation-run-completion.ts fired a no-op
   status transition (`in_progress` → `in_progress`) when a run
   succeeded without producing the expected deliverable. advanceTaskStatus
   no-ops on identical prev/next, and handleTaskStatusChange logged its
   `caller should gate this` warning and returned. finishRun had already
   released active_run_id and checkout_run_id, so the task sat at
   `in_progress` with no lock, no run — the exact orphan pattern that
   checkoutTask's `status IN ('todo','backlog')` clause refuses.
   Transition the row to `blocked` with the diagnostic comment attached
   so the manager's blocker queue surfaces it, the assignee stops
   spinning, and effectiveStatus carries the real state downstream.

2. `checkoutTask` itself refused to reclaim any orphan `in_progress`
   row, even when active_run_id was already null. That refusal was the
   right invariant against concurrent re-entry but wrong for the orphan
   case a crashed or status-unchanged run leaves behind. Extend the CAS
   predicate to accept `status = 'in_progress' AND active_run_id IS NULL`
   — heartbeat candidate selection already filters out tasks whose
   active_run_id points at a live run, so the guard preserves the
   original mutual-exclusion property for the busy case while allowing
   the orphan reclaim the scheduler actually needs.
This commit is contained in:
ZacharyZhang-NY committed 2026-04-19 21:09:16 -04:00
1 parent fdde562d55
commit 6b22fa9f0e
2 files changed
+39 -5

No files matched your search

@@ -349,10 +349,26 @@ export function processRunCompletion(
if (needsDeliverableCheck(taskType)) {
const hasDeliverable = checkRunProducedDeliverable(db, run.id, run.companyId, task.id, taskType);
if (!hasDeliverable) {
// Don't auto-advance -- flag for manager review
db.advanceTaskStatus(task.id, "in_progress",
`${taskType} task run completed but no deliverable artifact was detected. Flagging for manager review.`);
handleTaskStatusChange(task.id, run.companyId, "in_progress", "in_progress");
// Don't auto-advance to in_review: the run succeeded but did not
// produce the required deliverable, so it's not ready for
// manager sign-off.
//
// Park the task as `blocked` rather than leaving it at
// `in_progress`. finishRun has already cleared the run's
// checkout_run_id and active_run_id, so a task still tagged
// `in_progress` has no lock, no run, and no path back to
// `todo` — heartbeat picks it up, checkoutTask rejects it
// (status ∉ todo/backlog), and the assignee is stuck in a
// permanent `checkout_race_lost` loop. Blocking instead puts
// the task in the manager's blocker queue with the diagnostic
// comment attached; the manager patches it back to `todo` (or
// `cancelled`) after deciding how to proceed.
db.advanceTaskStatus(
task.id,
"blocked",
`${taskType} task run completed but no deliverable artifact was detected. Flagging for manager review.`,
);
handleTaskStatusChange(task.id, run.companyId, "in_progress", "blocked");
// Wake manager for review
try {
const agent = db.getAgent(run.agentId);
@@ -366,6 +382,7 @@ export function processRunCompletion(
);
}
publishDomainChanged();
effectiveStatus = "blocked";
// Skip the normal in_review advance
} else {
effectiveStatus = advanceSuccessfulRun(