Self-healing queue: transient provider failures park the provider, never burn retries
The July 9 run died at 07:22 when the GPT account hit its plan-side usage limit: every later request failed with "The usage limit has been reached", each failure counted against the 3-attempt retry budget, and 31 tasks were abandoned for good while the worker kept hammering 12 doomed requests a minute until the queue drained. Root fix, three parts sharing one classifier: - isTransientProviderFailure() in provider-runtime-domain recognises provider-side verdicts (usage limit, rate limit, 429, overloaded) plus the internal monthly-quota message; matching is against recorded task_runs.error_message rows, so historical damage heals retroactively. - enqueueHeartbeatRuns computes the retry budget with that predicate instead of an exact match on the internal quota string alone. - A transient failure parks the provider for a 10-minute cooldown: the heartbeat stops enqueuing its tasks and the worker leaves its queued runs unclaimed, then a single probe run re-arms or clears the park. Transitions are logged once each way. Tests: predicate unit tests, budget-exemption regression on the raw provider message, cooldown park/resume with an injected clock, claim exclusion.
This commit is contained in:
1 parent
4e11ca13ec
commit
927bdc91f2
6 files changed
+270
-13
No files matched your search
@@ -50,4 +50,29 @@ describe("task queue worker claim", () => {
|
||||
expect(states).toEqual(["running", "running"]);
|
||||
connection.close();
|
||||
});
|
||||
|
||||
test("leaves runs of a cooling-down provider queued instead of claiming them", () => {
|
||||
const connection = new DatabaseConnection(join(tmpdir(), `ac-queue-${randomUUID()}.db`));
|
||||
const organization = new OrganizationService(connection.db);
|
||||
let orgId = "";
|
||||
let agentId = "";
|
||||
let providerId = "";
|
||||
let taskId = "";
|
||||
connection.transaction(() => {
|
||||
orgId = organization.createOrganization({ organizationName: "Acme", employeeName: "Dana", mission: "m", goal: "g" });
|
||||
providerId = (connection.db.prepare("select id from providers where org_id = ? and provider_key = 'openai'").get(orgId) as { id: string }).id;
|
||||
agentId = organization.createAgent({ orgId, name: "CEO", role: "CEO", department: "Executive", modelProviderId: providerId, modelId: "gpt-5.4", budgetCents: 25000, managerAgentId: null });
|
||||
taskId = organization.createTask({
|
||||
orgId, title: "Kickoff", description: "Start.", status: TASK_STATUS.BACKLOG, assigneeAgentId: agentId,
|
||||
risk: RISK_LEVEL.LOW, projectId: null, goalId: null, workspaceId: null, priority: TASK_PRIORITY.NORMAL, requiresFilesystem: false,
|
||||
});
|
||||
});
|
||||
const runId = seedQueuedRun(connection, orgId, taskId, agentId, providerId);
|
||||
|
||||
expect(claimNextQueuedRun(connection, [providerId])).toBeNull();
|
||||
expect(connection.db.prepare("select state from task_runs where id = ?").get(runId)).toEqual({ state: "queued" });
|
||||
|
||||
expect(claimNextQueuedRun(connection, [])?.id).toBe(runId);
|
||||
connection.close();
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user