Self-healing queue: transient provider failures park the provider, never burn retries
The July 9 run died at 07:22 when the GPT account hit its plan-side usage limit: every later request failed with "The usage limit has been reached", each failure counted against the 3-attempt retry budget, and 31 tasks were abandoned for good while the worker kept hammering 12 doomed requests a minute until the queue drained. Root fix, three parts sharing one classifier: - isTransientProviderFailure() in provider-runtime-domain recognises provider-side verdicts (usage limit, rate limit, 429, overloaded) plus the internal monthly-quota message; matching is against recorded task_runs.error_message rows, so historical damage heals retroactively. - enqueueHeartbeatRuns computes the retry budget with that predicate instead of an exact match on the internal quota string alone. - A transient failure parks the provider for a 10-minute cooldown: the heartbeat stops enqueuing its tasks and the worker leaves its queued runs unclaimed, then a single probe run re-arms or clears the park. Transitions are logged once each way. Tests: predicate unit tests, budget-exemption regression on the raw provider message, cooldown park/resume with an injected clock, claim exclusion.
This commit is contained in:
1 parent
4e11ca13ec
commit
927bdc91f2
6 files changed
+270
-13
No files matched your search
@@ -38,6 +38,28 @@ export interface ProviderRateLimitObservationRecord extends ProviderObservedRate
|
||||
*/
|
||||
export const PROVIDER_MONTHLY_LIMIT_MESSAGE = "Provider monthly request limit has been reached.";
|
||||
|
||||
/**
|
||||
* Failure-message signatures of transient provider conditions — the account's
|
||||
* usage window filling up, rate limiting, or momentary overload. These verdicts
|
||||
* come from the provider's side of the wire and say nothing about the task, so
|
||||
* schedulers must treat them as "try again later", never as attempts consumed.
|
||||
* Matched case-insensitively against recorded task_runs.error_message rows,
|
||||
* which also heals historical rows written before this classification existed.
|
||||
*/
|
||||
const TRANSIENT_PROVIDER_FAILURE_SIGNATURES = [
|
||||
PROVIDER_MONTHLY_LIMIT_MESSAGE,
|
||||
"usage limit", // ChatGPT-account plans: "The usage limit has been reached"
|
||||
"rate limit",
|
||||
"too many requests", // HTTP 429 status text
|
||||
"overloaded", // "Our servers are currently overloaded. Please try again later."
|
||||
].map((signature) => signature.toLowerCase());
|
||||
|
||||
export function isTransientProviderFailure(message: string | null | undefined): boolean {
|
||||
if (!message) return false;
|
||||
const normalized = message.toLowerCase();
|
||||
return TRANSIENT_PROVIDER_FAILURE_SIGNATURES.some((signature) => normalized.includes(signature));
|
||||
}
|
||||
|
||||
export function currentProviderUsageWindow(now = new Date()): { start: string; end: string } {
|
||||
const start = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), 1));
|
||||
const end = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth() + 1, 1));
|
||||
|
||||
Reference in new issue
Block a user