Self-healing queue: transient provider failures park the provider, never burn retries

The July 9 run died at 07:22 when the GPT account hit its plan-side usage
limit: every later request failed with "The usage limit has been reached",
each failure counted against the 3-attempt retry budget, and 31 tasks were
abandoned for good while the worker kept hammering 12 doomed requests a
minute until the queue drained.

Root fix, three parts sharing one classifier:
- isTransientProviderFailure() in provider-runtime-domain recognises
  provider-side verdicts (usage limit, rate limit, 429, overloaded) plus the
  internal monthly-quota message; matching is against recorded
  task_runs.error_message rows, so historical damage heals retroactively.
- enqueueHeartbeatRuns computes the retry budget with that predicate instead
  of an exact match on the internal quota string alone.
- A transient failure parks the provider for a 10-minute cooldown: the
  heartbeat stops enqueuing its tasks and the worker leaves its queued runs
  unclaimed, then a single probe run re-arms or clears the park. Transitions
  are logged once each way.

Tests: predicate unit tests, budget-exemption regression on the raw provider
message, cooldown park/resume with an injected clock, claim exclusion.
This commit is contained in:
ZacharyZhang-NY committed 2026-07-09 19:04:44 -04:00
1 parent 4e11ca13ec
commit 927bdc91f2
6 files changed
+270 -13

No files matched your search

@@ -0,0 +1,21 @@
import { describe, expect, test } from "vitest";
import { isTransientProviderFailure, PROVIDER_MONTHLY_LIMIT_MESSAGE } from "../../src/shared/provider-runtime-domain";
describe("isTransientProviderFailure", () => {
test("classifies provider-side limit and overload verdicts as transient", () => {
expect(isTransientProviderFailure(PROVIDER_MONTHLY_LIMIT_MESSAGE)).toBe(true);
expect(isTransientProviderFailure("The usage limit has been reached")).toBe(true);
expect(isTransientProviderFailure("Our servers are currently overloaded. Please try again later.")).toBe(true);
expect(isTransientProviderFailure("Rate limit exceeded, retry after 20s")).toBe(true);
expect(isTransientProviderFailure("429 Too Many Requests")).toBe(true);
});
test("keeps genuine task failures counting toward the retry budget", () => {
expect(isTransientProviderFailure(null)).toBe(false);
expect(isTransientProviderFailure(undefined)).toBe(false);
expect(isTransientProviderFailure("")).toBe(false);
expect(isTransientProviderFailure("provider rejected task")).toBe(false);
expect(isTransientProviderFailure("Agent run did not converge within 12 iterations.")).toBe(false);
expect(isTransientProviderFailure("Provider returned an empty turn: no text and no tool calls.")).toBe(false);
});
});