Keep builders coming when DigitalOcean sells out a size or region (#861)
* Keep builders coming when DigitalOcean sells out a droplet size ric1 sold out of g5-32vcpu-64gb-50gb, and every create came back 422. curl -f dropped the reason and set -e ended the tick, so builders only appeared when capacity happened to free up, and the journal showed nothing but "curl: (22)". Try each of SIZES in turn, logging DigitalOcean's refusal message, and fail the tick only when every size is refused. Builders now power off however start.sh exits, so a failed registration is reaped instead of counting as a booting runner until MAX_AGE_MINUTES. The controller unit pulls the checkout before each tick, so merged controller fixes reach the box. * Create builders in any region that has the size in stock ric1 sold out of g5-32vcpu-128gb-50gb within minutes of the box switching to it. Builders need nothing from a particular region, so read DigitalOcean's size catalog once per tick and try each of SIZES in every region it lists in stock, REGIONS first if set. A refused pair is dropped for the rest of the tick.
This commit is contained in:
1 parent
024943141d
commit
128c5647ba
6 files changed
+111
-17
No files matched your search
@@ -7,6 +7,9 @@ Wants=network-online.target
|
||||
Type=oneshot
|
||||
User=controller
|
||||
EnvironmentFile=/etc/omarchy-controller.env
|
||||
# Run the branch's current controller, not the one cloned when the box was
|
||||
# built. As root (the checkout's owner); a failed pull keeps the last one.
|
||||
ExecStartPre=-+/usr/bin/git -C /opt/omarchy-pkgs pull --ff-only --quiet
|
||||
ExecStart=/opt/omarchy-pkgs/ci/controller.sh
|
||||
# The reaper's safety net is time, not state; a hung tick must not hold the lock.
|
||||
TimeoutStartSec=240
|
||||
Reference in new issue
Block a user