Replace builder droplets stuck provisioning (#864)
A droplet DigitalOcean still reports as "new" never registers a runner, but the controller counted it as one booting and created no replacement until MAX_AGE_MINUTES. A publish job sat queued for minutes behind one in mkc1. Delete droplets still provisioning after MAX_BOOT_MINUTES (10) and leave them out of the live count, so the same tick creates a replacement.
This commit is contained in:
1 parent
259a3fa8b8
commit
5961a3ff5d
4 files changed
+18
-4
No files matched your search
+1
-1
@@ -14,7 +14,7 @@ signing on merge exactly as before.
|
||||
- `controller.sh` — systemd timer every minute on a small always-on droplet.
|
||||
Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet per job up
|
||||
to `MAX_DROPLETS`, deletes droplets that are powered off or older than
|
||||
`MAX_AGE_MINUTES`. Builders go in any region DigitalOcean lists the size in
|
||||
`MAX_AGE_MINUTES`, or still provisioning after `MAX_BOOT_MINUTES`. Builders go in any region DigitalOcean lists the size in
|
||||
stock in (`REGIONS` only sets which to try first); a refused create, logged
|
||||
with DigitalOcean's message, falls back to the next region, then the next
|
||||
of `SIZES`. No inbound endpoint. Plain curl against both APIs, no
|
||||
|
||||
Reference in new issue
Block a user