Replace builder droplets stuck provisioning (#864)

A droplet DigitalOcean still reports as "new" never registers a runner, but
the controller counted it as one booting and created no replacement until
MAX_AGE_MINUTES. A publish job sat queued for minutes behind one in mkc1.

Delete droplets still provisioning after MAX_BOOT_MINUTES (10) and leave them
out of the live count, so the same tick creates a replacement.
This commit is contained in:
Ryan Hughes authored and GitHub committed 2026-10-08 11:59:48 -04:00
1 parent 259a3fa8b8
commit 5961a3ff5d
4 files changed
+18 -4

No files matched your search

+1 -1
View File
@@ -14,7 +14,7 @@ signing on merge exactly as before.
- `controller.sh` — systemd timer every minute on a small always-on droplet.
Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet per job up
to `MAX_DROPLETS`, deletes droplets that are powered off or older than
`MAX_AGE_MINUTES`. Builders go in any region DigitalOcean lists the size in
`MAX_AGE_MINUTES`, or still provisioning after `MAX_BOOT_MINUTES`. Builders go in any region DigitalOcean lists the size in
stock in (`REGIONS` only sets which to try first); a refused create, logged
with DigitalOcean's message, falls back to the next region, then the next
of `SIZES`. No inbound endpoint. Plain curl against both APIs, no