Free tasks from the checkout_race_lost deadlock
Two bugs combined to park successful-but-deliverable-less tasks in an
unrecoverable state. Operator saw Alex Chen loop on
`checkout_race_lost` every two minutes for 16+ minutes in the field
log after completing its first task.
1. `processRunCompletion` in automation-run-completion.ts fired a no-op
status transition (`in_progress` → `in_progress`) when a run
succeeded without producing the expected deliverable. advanceTaskStatus
no-ops on identical prev/next, and handleTaskStatusChange logged its
`caller should gate this` warning and returned. finishRun had already
released active_run_id and checkout_run_id, so the task sat at
`in_progress` with no lock, no run — the exact orphan pattern that
checkoutTask's `status IN ('todo','backlog')` clause refuses.
Transition the row to `blocked` with the diagnostic comment attached
so the manager's blocker queue surfaces it, the assignee stops
spinning, and effectiveStatus carries the real state downstream.
2. `checkoutTask` itself refused to reclaim any orphan `in_progress`
row, even when active_run_id was already null. That refusal was the
right invariant against concurrent re-entry but wrong for the orphan
case a crashed or status-unchanged run leaves behind. Extend the CAS
predicate to accept `status = 'in_progress' AND active_run_id IS NULL`
— heartbeat candidate selection already filters out tasks whose
active_run_id points at a live run, so the guard preserves the
original mutual-exclusion property for the busy case while allowing
the orphan reclaim the scheduler actually needs.
This commit is contained in:
1 parent
fdde562d55
commit
6b22fa9f0e
2 files changed
+39
-5
No files matched your search
@@ -159,13 +159,30 @@ export function buildTaskStatusMethods(
|
||||
|
||||
checkoutTask(taskId: string, runId: string): boolean {
|
||||
const ts = nowIso();
|
||||
// Primary path is `status IN ('todo','backlog')`. The extra
|
||||
// `in_progress AND active_run_id IS NULL` branch recovers from a
|
||||
// specific orphan: finishRun has cleared active_run_id and
|
||||
// checkout_run_id for the previous run but the status transition
|
||||
// to done/in_review never fired (e.g. the run succeeded without a
|
||||
// deliverable and automation parked it at in_progress, or a
|
||||
// crashed run left the row behind). Without the reclaim, heartbeat
|
||||
// picks the task (activeRunId=null passes the candidate filter)
|
||||
// and then loops on `checkout_race_lost` forever because status
|
||||
// never flips back to todo on its own.
|
||||
//
|
||||
// The `active_run_id IS NULL` guard preserves the original CAS
|
||||
// contract for the busy case: a task whose active_run_id still
|
||||
// points at a live run is refused here, exactly as before.
|
||||
const result = getDb()
|
||||
.prepare(
|
||||
`UPDATE tasks
|
||||
SET checkout_run_id = ?, execution_locked_at = ?, status = 'in_progress', updated_at = ?
|
||||
WHERE id = ?
|
||||
AND checkout_run_id IS NULL
|
||||
AND status IN ('todo', 'backlog')`,
|
||||
AND (
|
||||
status IN ('todo', 'backlog')
|
||||
OR (status = 'in_progress' AND active_run_id IS NULL)
|
||||
)`,
|
||||
)
|
||||
.run(runId, ts, ts, taskId);
|
||||
return result.changes > 0;
|
||||
|
||||
Reference in new issue
Block a user