Self-healing queue: transient provider failures park the provider, never burn retries
The July 9 run died at 07:22 when the GPT account hit its plan-side usage limit: every later request failed with "The usage limit has been reached", each failure counted against the 3-attempt retry budget, and 31 tasks were abandoned for good while the worker kept hammering 12 doomed requests a minute until the queue drained. Root fix, three parts sharing one classifier: - isTransientProviderFailure() in provider-runtime-domain recognises provider-side verdicts (usage limit, rate limit, 429, overloaded) plus the internal monthly-quota message; matching is against recorded task_runs.error_message rows, so historical damage heals retroactively. - enqueueHeartbeatRuns computes the retry budget with that predicate instead of an exact match on the internal quota string alone. - A transient failure parks the provider for a 10-minute cooldown: the heartbeat stops enqueuing its tasks and the worker leaves its queued runs unclaimed, then a single probe run re-arms or clears the park. Transitions are logged once each way. Tests: predicate unit tests, budget-exemption regression on the raw provider message, cooldown park/resume with an injected clock, claim exclusion.
This commit is contained in:
1 parent
4e11ca13ec
commit
927bdc91f2
6 files changed
+270
-13
No files matched your search
@@ -26,12 +26,17 @@ export function isAutonomousPaused(connection: DatabaseConnection): boolean {
|
||||
/**
|
||||
* Atomically claims the oldest queued task_run by transitioning it
|
||||
* `queued` -> `running` in one transaction; returns the claimed run or null.
|
||||
* Runs whose provider sits in a transient-failure cooldown are left queued —
|
||||
* they would fail on arrival, and waiting costs nothing.
|
||||
*/
|
||||
export function claimNextQueuedRun(connection: DatabaseConnection): QueuedRun | null {
|
||||
export function claimNextQueuedRun(connection: DatabaseConnection, excludedProviderIds: string[] = []): QueuedRun | null {
|
||||
return connection.transaction(() => {
|
||||
const exclusion = excludedProviderIds.length > 0
|
||||
? `and provider_id not in (${excludedProviderIds.map(() => "?").join(", ")})`
|
||||
: "";
|
||||
const row = connection.db
|
||||
.prepare("select id, org_id from task_runs where state = 'queued' order by created_at asc limit 1")
|
||||
.get() as QueuedRun | undefined;
|
||||
.prepare(`select id, org_id from task_runs where state = 'queued' ${exclusion} order by created_at asc limit 1`)
|
||||
.get(...excludedProviderIds) as QueuedRun | undefined;
|
||||
if (!row) return null;
|
||||
const result = connection.db
|
||||
.prepare("update task_runs set state = 'running', started_at = coalesce(started_at, ?) where id = ? and state = 'queued'")
|
||||
@@ -70,7 +75,7 @@ export function startTaskQueueWorker(input: TaskQueueWorkerInput): () => void {
|
||||
}
|
||||
if (paused) return;
|
||||
const enqueued = input.connection.transaction(() => taskExecution.enqueueHeartbeatRuns());
|
||||
const claimed = claimNextQueuedRun(input.connection);
|
||||
const claimed = claimNextQueuedRun(input.connection, taskExecution.cooledProviderIds());
|
||||
if (!claimed) {
|
||||
if (enqueued.length > 0) notifyDataChanged();
|
||||
return;
|
||||
|
||||
Reference in new issue
Block a user