Keep builders coming when DigitalOcean sells out a size or region (#861)
* Keep builders coming when DigitalOcean sells out a droplet size ric1 sold out of g5-32vcpu-64gb-50gb, and every create came back 422. curl -f dropped the reason and set -e ended the tick, so builders only appeared when capacity happened to free up, and the journal showed nothing but "curl: (22)". Try each of SIZES in turn, logging DigitalOcean's refusal message, and fail the tick only when every size is refused. Builders now power off however start.sh exits, so a failed registration is reaped instead of counting as a booting runner until MAX_AGE_MINUTES. The controller unit pulls the checkout before each tick, so merged controller fixes reach the box. * Create builders in any region that has the size in stock ric1 sold out of g5-32vcpu-128gb-50gb within minutes of the box switching to it. Builders need nothing from a particular region, so read DigitalOcean's size catalog once per tick and try each of SIZES in every region it lists in stock, REGIONS first if set. A refused pair is dropped for the rest of the tick.
This commit is contained in:
1 parent
024943141d
commit
128c5647ba
6 files changed
+111
-17
No files matched your search
+12
-2
@@ -12,9 +12,12 @@ signing on merge exactly as before.
|
||||
- `runner-cloud-init.yaml` — Ubuntu 24.04 user-data: docker + buildx, the
|
||||
GitHub runner registered `--ephemeral`, runs one job, powers off.
|
||||
- `controller.sh` — systemd timer every minute on a small always-on droplet.
|
||||
Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet (ric1) per job up
|
||||
Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet per job up
|
||||
to `MAX_DROPLETS`, deletes droplets that are powered off or older than
|
||||
`MAX_AGE_MINUTES`. No inbound endpoint. Plain curl against both APIs, no
|
||||
`MAX_AGE_MINUTES`. Builders go in any region DigitalOcean lists the size in
|
||||
stock in (`REGIONS` only sets which to try first); a refused create, logged
|
||||
with DigitalOcean's message, falls back to the next region, then the next
|
||||
of `SIZES`. No inbound endpoint. Plain curl against both APIs, no
|
||||
doctl and no gh: a token in the environment cannot pick the wrong account
|
||||
the way a saved doctl context can. Needs curl and jq.
|
||||
`tests/controller.sh` exercises every decision against canned responses.
|
||||
@@ -31,6 +34,13 @@ Administration read+write (registration tokens). The DO token is baked into
|
||||
the box's env file, so it is the account that pays for builder droplets.
|
||||
Watch it with `journalctl -u omarchy-controller -f` on the box.
|
||||
|
||||
Each tick pulls the box's checkout first, so a merged `controller.sh` is live
|
||||
within a minute. The unit and timer are copies made at creation; after
|
||||
changing them, on the box:
|
||||
|
||||
cp /opt/omarchy-pkgs/ci/controller-box/omarchy-controller.{service,timer} /etc/systemd/system/
|
||||
systemctl daemon-reload
|
||||
|
||||
## What the spike proved (2026-09-17, fork ryanrhughes/omarchy-pkgs)
|
||||
|
||||
- `bin/build` works from a bare clone: with no local published tree it
|
||||
|
||||
Reference in new issue
Block a user