Keep builders coming when DigitalOcean sells out a size or region (#861)

* Keep builders coming when DigitalOcean sells out a droplet size

ric1 sold out of g5-32vcpu-64gb-50gb, and every create came back 422. curl -f
dropped the reason and set -e ended the tick, so builders only appeared when
capacity happened to free up, and the journal showed nothing but "curl: (22)".

Try each of SIZES in turn, logging DigitalOcean's refusal message, and fail
the tick only when every size is refused. Builders now power off however
start.sh exits, so a failed registration is reaped instead of counting as a
booting runner until MAX_AGE_MINUTES. The controller unit pulls the checkout
before each tick, so merged controller fixes reach the box.

* Create builders in any region that has the size in stock

ric1 sold out of g5-32vcpu-128gb-50gb within minutes of the box switching to
it. Builders need nothing from a particular region, so read DigitalOcean's
size catalog once per tick and try each of SIZES in every region it lists in
stock, REGIONS first if set. A refused pair is dropped for the rest of the
tick.
This commit is contained in:
Ryan Hughes authored and GitHub committed 2026-10-08 10:50:06 -04:00
1 parent 024943141d
commit 128c5647ba
6 files changed
+111 -17

No files matched your search

+12 -2
View File
@@ -12,9 +12,12 @@ signing on merge exactly as before.
- `runner-cloud-init.yaml` — Ubuntu 24.04 user-data: docker + buildx, the
GitHub runner registered `--ephemeral`, runs one job, powers off.
- `controller.sh` — systemd timer every minute on a small always-on droplet.
Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet (ric1) per job up
Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet per job up
to `MAX_DROPLETS`, deletes droplets that are powered off or older than
`MAX_AGE_MINUTES`. No inbound endpoint. Plain curl against both APIs, no
`MAX_AGE_MINUTES`. Builders go in any region DigitalOcean lists the size in
stock in (`REGIONS` only sets which to try first); a refused create, logged
with DigitalOcean's message, falls back to the next region, then the next
of `SIZES`. No inbound endpoint. Plain curl against both APIs, no
doctl and no gh: a token in the environment cannot pick the wrong account
the way a saved doctl context can. Needs curl and jq.
`tests/controller.sh` exercises every decision against canned responses.
@@ -31,6 +34,13 @@ Administration read+write (registration tokens). The DO token is baked into
the box's env file, so it is the account that pays for builder droplets.
Watch it with `journalctl -u omarchy-controller -f` on the box.
Each tick pulls the box's checkout first, so a merged `controller.sh` is live
within a minute. The unit and timer are copies made at creation; after
changing them, on the box:
cp /opt/omarchy-pkgs/ci/controller-box/omarchy-controller.{service,timer} /etc/systemd/system/
systemctl daemon-reload
## What the spike proved (2026-09-17, fork ryanrhughes/omarchy-pkgs)
- `bin/build` works from a bare clone: with no local published tree it