The July 9 run died at 07:22 when the GPT account hit its plan-side usage
limit: every later request failed with "The usage limit has been reached",
each failure counted against the 3-attempt retry budget, and 31 tasks were
abandoned for good while the worker kept hammering 12 doomed requests a
minute until the queue drained.
Root fix, three parts sharing one classifier:
- isTransientProviderFailure() in provider-runtime-domain recognises
provider-side verdicts (usage limit, rate limit, 429, overloaded) plus the
internal monthly-quota message; matching is against recorded
task_runs.error_message rows, so historical damage heals retroactively.
- enqueueHeartbeatRuns computes the retry budget with that predicate instead
of an exact match on the internal quota string alone.
- A transient failure parks the provider for a 10-minute cooldown: the
heartbeat stops enqueuing its tasks and the worker leaves its queued runs
unclaimed, then a single probe run re-arms or clears the park. Transitions
are logged once each way.
Tests: predicate unit tests, budget-exemption regression on the raw provider
message, cooldown park/resume with an injected clock, claim exclusion.
The heartbeat scheduler enqueued runs even while a provider's monthly
usage window was full, so every task burned its 3-attempt retry budget
on instant quota failures within seconds and stayed dead forever -
raising the limit (or restarting) never revived anything.
- Park tasks (skip enqueue) while their provider is at its
monthly_request_limit instead of creating runs doomed to fail
- Exclude quota-blocked failures from the 3-attempt retry budget,
retroactively reviving tasks abandoned by past filled windows
- Share the gate message as PROVIDER_MONTHLY_LIMIT_MESSAGE and log
quota-block transitions once per state change for diagnosability
- Ignore landing/ (self-contained Next.js sub-project) and
built-in skills/ (prompt content) in eslint so verify:ci lints only
application code
The legacy application - its src, tests, landing site, promo assets, docs,
release tooling, CI workflows, and root configs - leaves the tree; it stays
reachable in history. Everything under AgentCompany-New moves up unchanged:
same package name, so the production userData/database path is untouched,
and e2e/build scripts already resolved paths relative to the project. The
developer cred.txt stays gitignored at its new location. Full verify:ci
passes from the new root.
ZacharyZhang-NYandClaude Opus 5.5