The July 9 run died at 07:22 when the GPT account hit its plan-side usage
limit: every later request failed with "The usage limit has been reached",
each failure counted against the 3-attempt retry budget, and 31 tasks were
abandoned for good while the worker kept hammering 12 doomed requests a
minute until the queue drained.
Root fix, three parts sharing one classifier:
- isTransientProviderFailure() in provider-runtime-domain recognises
provider-side verdicts (usage limit, rate limit, 429, overloaded) plus the
internal monthly-quota message; matching is against recorded
task_runs.error_message rows, so historical damage heals retroactively.
- enqueueHeartbeatRuns computes the retry budget with that predicate instead
of an exact match on the internal quota string alone.
- A transient failure parks the provider for a 10-minute cooldown: the
heartbeat stops enqueuing its tasks and the worker leaves its queued runs
unclaimed, then a single probe run re-arms or clears the park. Transitions
are logged once each way.
Tests: predicate unit tests, budget-exemption regression on the raw provider
message, cooldown park/resume with an injected clock, claim exclusion.
Two operator asks, one review gate:
Avatar portraits (migration 052): the avatar editor gains "Upload
photo" - the renderer downscales the chosen picture to a ~96px cover
crop and stores it as a small data URL (PNG keeps transparency, the
rest compresses to JPEG), so the snapshot stays light and the CSP needs
nothing new. The portrait wins over emoji and initials everywhere the
shared Avatar renders (chat, org chart, office card, sidebar), and
"Remove photo" falls back cleanly. The main process validates scheme
and size (base64 image data URL, <=300k chars) so a rogue caller cannot
re-bloat the snapshot. e2e uploads a real PNG through the profile
dialog and sees the portrait land in the sidebar.
Global pause (migration 053): when agents start talking in circles,
one topbar toggle - visible on every page - pulls the handbrake.
app_state.autorun_paused survives restarts; the queue worker checks it
each tick and neither enqueues nor claims while set (the run already in
flight finishes). Queued work simply waits and the paused state renders
as an amber "resume" pill. A unit test drives the real worker: paused
leaves a queued run untouched through many ticks, resuming claims it
within moments; e2e flips the toggle in the live shell.
The legacy application - its src, tests, landing site, promo assets, docs,
release tooling, CI workflows, and root configs - leaves the tree; it stays
reachable in history. Everything under AgentCompany-New moves up unchanged:
same package name, so the production userData/database path is untouched,
and e2e/build scripts already resolved paths relative to the project. The
developer cred.txt stays gitignored at its new location. Full verify:ci
passes from the new root.
ZacharyZhang-NY