Two bugs combined to park successful-but-deliverable-less tasks in an
unrecoverable state. Operator saw Alex Chen loop on
`checkout_race_lost` every two minutes for 16+ minutes in the field
log after completing its first task.
1. `processRunCompletion` in automation-run-completion.ts fired a no-op
status transition (`in_progress` → `in_progress`) when a run
succeeded without producing the expected deliverable. advanceTaskStatus
no-ops on identical prev/next, and handleTaskStatusChange logged its
`caller should gate this` warning and returned. finishRun had already
released active_run_id and checkout_run_id, so the task sat at
`in_progress` with no lock, no run — the exact orphan pattern that
checkoutTask's `status IN ('todo','backlog')` clause refuses.
Transition the row to `blocked` with the diagnostic comment attached
so the manager's blocker queue surfaces it, the assignee stops
spinning, and effectiveStatus carries the real state downstream.
2. `checkoutTask` itself refused to reclaim any orphan `in_progress`
row, even when active_run_id was already null. That refusal was the
right invariant against concurrent re-entry but wrong for the orphan
case a crashed or status-unchanged run leaves behind. Extend the CAS
predicate to accept `status = 'in_progress' AND active_run_id IS NULL`
— heartbeat candidate selection already filters out tasks whose
active_run_id points at a live run, so the guard preserves the
original mutual-exclusion property for the busy case while allowing
the orphan reclaim the scheduler actually needs.
advanceTaskStatus cleared checkout_run_id when transitioning out of
in_progress but left active_run_id alone. Heartbeat selects candidate
tasks by !activeRunId, so a task recovered from a crash (moved
blocked → todo by the bootstrap recovery) could carry a dead
active_run_id pointer and remain silently unpickable by every future
wake, even though the run it points to is long since interrupted.
interruptStaleRuns normally clears active_run_id for tasks whose
owning run is now interrupted, but the guarantee relied on a single
startup pass. Make advanceTaskStatus defensive: when the terminal
state is todo or backlog, clear the pointer too. The invariant holds
end-to-end instead of relying on interruptStaleRuns having seen every
edge case.
Also harden the new-CEO team-building cue: make it an explicit
imperative ("call the bash tool to run the fetch POSTs now") and
route app_restart_resume through the same branch so a startup-crash
recovery wake lands on the same focused action script.
`init()` ran three implicit data-repair queries on every boot:
`reconcileOrphanedTasks`, `reconcileUnreviewableTasks`, and a
secrets-table migration whose fallback glued legacy rows onto a
synthetic zero UUID when no current company existed. The first
two papered over state-machine leaks at init time — hiding the
real bug and making startup observably mutate production data;
the third silently connected ciphertext to a non-existent company
with no UI visibility and no deletion path.
Root fix: the startup path is now allowed exactly one mutation —
`interruptStaleRuns`, which is crash recovery (running-at-boot =
worker died mid-turn) and cannot be expressed anywhere else.
The dead-letter `in_review` case moves to the transition point:
`tasks-status.guardInReviewHasReviewer` refuses to advance a task
into `in_review` when its assignee is a top-level agent
(`reports_to IS NULL`) and the task is not flagged for user
review. Callers either pre-flag the task (the CEO-completion path
in `automation-run-completion.advanceToInReview` now does this)
or the transition is rejected. No more silent resurrection at
boot; the failure is visible and actionable at source.
The orphan-task reconcile disappears entirely — the `finishRun`
guard already enforces the invariant at the one place a run can
leave the in-progress state, and `recoverStaleLocks` handles
stuck executions at runtime.
The secrets migration now throws when neither
`settings.current_company_id` nor any company row exists, rather
than binding legacy ciphertext to `00000000-0000-0000-0000-000000000000`.
Legacy rows stay on the old table until the operator creates a
real company, at which point the upgrade succeeds deterministically.
Tests rewritten: the reconciliation regression suite now asserts
the transition-time guard and the lone `interruptStaleRuns`
behavior. `resolve-workspace.test.ts` catches up with the
exclusive-by-default lock policy committed earlier.
Sessions have been agent-scoped since v34, but the schema kept four
dead task-scoped columns that every session code path had to pretend
were meaningful:
- tasks.active_session_id (mirrored agents.session_id)
- tasks.session_json (superseded by task_sessions.session_params_json)
- tasks.session_display_id (superseded by task_sessions.session_display_id)
- task_sessions.task_id (always empty-string on new rows)
v41 drops all four via SQLite table rebuild — the only safe path when
multiple columns need to go in one step and indexes / FK directions
need to be preserved. `PRAGMA foreign_keys = OFF` spans the swap
window so cascade-delete does not fire mid-rebuild. The migration is
idempotent and also backfills the optional task columns
(task_type / requires_user_review / retry_count / last_failure_at /
started_at / completed_at / cancelled_at / checkout_run_id /
execution_locked_at) before rebuilding so it works whether or not
`ensureOptionalColumns` has run yet on this install.
Along with the schema change, every code path that read or wrote
those columns has been rewired to the agent-scoped equivalents:
- task-sessions-crud.ts: deleted updateTaskSession,
getTaskSession, getActiveTaskSession, updateTasksActiveSessionForAgent;
added updateSessionParams(sessionId, ...) for the worker's
finalizer. ensureAgentSession and createNewSession no longer touch
task_id.
- database-base.ts: facade drops the now-unused pass-throughs.
- snapshot.ts / task-queries.ts / tasks-status.ts / entity-crud.ts:
SELECT projections no longer pull session_json / session_display_id
/ active_session_id from tasks.
- tasks-crud.ts (TaskRow) and shared/types/records-core.ts
(TaskRecord): sessionJson / sessionDisplayId fields removed.
- TaskSessionRecord: taskId field removed.
- turn-context.ts::resolveSession: no more
updateTasksActiveSessionForAgent mirror; sessionParams read directly
from the current session via getSessionById.
- run-finished-helpers.ts::finalizeRun: stamps sessionParams via
updateSessionParams on the agent's current session, resolved through
agents.session_id.
- session-management-service.ts: rotateSession stops calling
updateTasksActiveSessionForAgent.
Test suite updated to use the new session-id API. Existing tests used
`getActiveTaskSession(taskId)` and `createNewSession({ taskId, ...})`
as probes; they now use `getSessionById(sessionId)` and the taskId-less
createNewSession input.
Tests: 603/603 green. Typecheck clean. Migration tested on fresh DB
(runs v1→v41 cleanly) and idempotent re-apply (detects absent dead
columns, skips).
The NOT NULL constraint forced heartbeat runs to synthesize a fake task
row (`[Heartbeat] X — Check for work`) so the FK could resolve. Three
layers of code — acquireHeartbeatTask, findReusableHeartbeatTask,
cleanupDuplicateHeartbeatTasks, plus the is_heartbeat flag column + UI
filter — existed to paper over one bad constraint. Remove the
constraint and the whole workaround vanishes.
SQLite table rebuild with data preservation. Live production DB, no
backup — migration is atomic, idempotent, and verified before DROP.
## Migration v35 (rebuild runs with nullable task_id)
Uses `fn:` form so it can:
- Snapshot `COUNT(*) FROM runs` before copy
- Walk `PRAGMA table_info(runs)` to detect drift columns (v34 declared
a column set; any additive ALTER since v14 shows up here)
- Assert `COUNT(*) FROM runs_new == beforeCount` before `DROP TABLE runs`
- `PRAGMA defer_foreign_keys = ON` so run_events(run_id) FK defers to
commit time, by which runs has the same id set under the new name
- Recreate all 6 indexes on the new table
If the assertion fails, the migration throws inside the existing
`transaction()` wrapper. SQLite rolls back the whole migration, old
`runs` table is untouched, `schema_migrations` never records v35. Next
app start retries cleanly.
## Migration v36 (drop tasks.is_heartbeat + synthetic rows)
Runs now carry `task_id = NULL` for heartbeat dispatches. The
`is_heartbeat` flag is dead data going forward.
1. `UPDATE runs SET task_id = NULL WHERE task_id IN (SELECT id FROM tasks WHERE is_heartbeat = 1)`
2. `DELETE FROM tasks WHERE is_heartbeat = 1` (synthetic rows, not real work)
3. Rebuild `tasks` without the `is_heartbeat` column (same defer-FK +
row-count-assert pattern as v35)
## Code cleanup
- `RunRecord.taskId: string | null`; `createRun` / `cancelRunForDeletion`
/ `finishRun` guard every `UPDATE tasks` path with a null-check.
- `TaskRecord.isHeartbeat` removed; every SELECT / filter / UI reference
dropped.
- Delete `acquireHeartbeatTask` / `findReusableHeartbeatTask` /
`cleanupDuplicateHeartbeatTasks` — dead after the constraint lifts.
- Delete `src/main/orchestration/heartbeat-task.ts` (pure workaround
constant file).
- `heartbeat.ts` creates runs with `taskId: null` when the agent has no
assigned work. Turn context already supports nullable task
(heartbeatFallbackSection); prompt surface unchanged.
- `validateTaskCreation` drops the `isHeartbeat` parameter — user-
authored + automation-authored tasks are never heartbeat.
- Renderer: `run.taskId ? taskMap.get(...) : undefined` in Dashboard /
RunCenter / RunDetail / RunQueue.
- Delete `tests/unit/heartbeat-coalescing.test.ts` (12 tests that
exercised the deleted DB methods).
- Drop 1 isHeartbeat-bypass test from `task-quality-gates.test.ts`.
## Grep verification
- `isHeartbeatTaskTitle src/` → 0
- `acquireHeartbeatTask src/` → migration comment only
- `is_heartbeat src/` → schema.ts migration history only
- `isHeartbeat src/` → 0
Tests: 578/578 (was 591 — delta is 13 tests that tested deleted code).
Replace the racy three-step heartbeat path (listAgentTasks ->
findReusableHeartbeatTask -> saveTask -> checkoutTask) with a single
transactional acquireHeartbeatTask method. The prior flow allowed two
concurrent scheduler ticks to both observe "no reusable heartbeat" and
both INSERT a fresh [Heartbeat] task before either CAS-checked out,
producing duplicates that cleanupDuplicateHeartbeatTasks had to mop up.
acquireHeartbeatTask runs inside BEGIN IMMEDIATE so concurrent ticks
serialize on the writer lock. Inside the transaction it first attempts
to claim any existing claimable [Heartbeat] task (todo/backlog/done/
in_review with checkout_run_id IS NULL), reviving done/in_review by
resetting status. If nothing is claimable but an in-flight heartbeat
exists (status NOT IN done/cancelled), it returns null instead of
creating a duplicate. Only when no heartbeat task exists at all does it
INSERT a new one, already in 'in_progress' with checkout_run_id set.
cleanupDuplicateHeartbeatTasks remains as deletion-side defense.
ZacharyZhang-NY