Self-healing queue: transient provider failures park the provider, never burn retries

The July 9 run died at 07:22 when the GPT account hit its plan-side usage
limit: every later request failed with "The usage limit has been reached",
each failure counted against the 3-attempt retry budget, and 31 tasks were
abandoned for good while the worker kept hammering 12 doomed requests a
minute until the queue drained.

Root fix, three parts sharing one classifier:
- isTransientProviderFailure() in provider-runtime-domain recognises
  provider-side verdicts (usage limit, rate limit, 429, overloaded) plus the
  internal monthly-quota message; matching is against recorded
  task_runs.error_message rows, so historical damage heals retroactively.
- enqueueHeartbeatRuns computes the retry budget with that predicate instead
  of an exact match on the internal quota string alone.
- A transient failure parks the provider for a 10-minute cooldown: the
  heartbeat stops enqueuing its tasks and the worker leaves its queued runs
  unclaimed, then a single probe run re-arms or clears the park. Transitions
  are logged once each way.

Tests: predicate unit tests, budget-exemption regression on the raw provider
message, cooldown park/resume with an injected clock, claim exclusion.
This commit is contained in:
ZacharyZhang-NY committed 2026-07-09 19:04:44 -04:00
1 parent 4e11ca13ec
commit 927bdc91f2
6 files changed
+270 -13

No files matched your search

+9 -4
View File
@@ -26,12 +26,17 @@ export function isAutonomousPaused(connection: DatabaseConnection): boolean {
/**
* Atomically claims the oldest queued task_run by transitioning it
* `queued` -> `running` in one transaction; returns the claimed run or null.
* Runs whose provider sits in a transient-failure cooldown are left queued —
* they would fail on arrival, and waiting costs nothing.
*/
export function claimNextQueuedRun(connection: DatabaseConnection): QueuedRun | null {
export function claimNextQueuedRun(connection: DatabaseConnection, excludedProviderIds: string[] = []): QueuedRun | null {
return connection.transaction(() => {
const exclusion = excludedProviderIds.length > 0
? `and provider_id not in (${excludedProviderIds.map(() => "?").join(", ")})`
: "";
const row = connection.db
.prepare("select id, org_id from task_runs where state = 'queued' order by created_at asc limit 1")
.get() as QueuedRun | undefined;
.prepare(`select id, org_id from task_runs where state = 'queued' ${exclusion} order by created_at asc limit 1`)
.get(...excludedProviderIds) as QueuedRun | undefined;
if (!row) return null;
const result = connection.db
.prepare("update task_runs set state = 'running', started_at = coalesce(started_at, ?) where id = ? and state = 'queued'")
@@ -70,7 +75,7 @@ export function startTaskQueueWorker(input: TaskQueueWorkerInput): () => void {
}
if (paused) return;
const enqueued = input.connection.transaction(() => taskExecution.enqueueHeartbeatRuns());
const claimed = claimNextQueuedRun(input.connection);
const claimed = claimNextQueuedRun(input.connection, taskExecution.cooledProviderIds());
if (!claimed) {
if (enqueued.length > 0) notifyDataChanged();
return;