0 Fork 0

Files

T

Ryan Hughes 128c5647ba Keep builders coming when DigitalOcean sells out a size or region (#861 )

* Keep builders coming when DigitalOcean sells out a droplet size

ric1 sold out of g5-32vcpu-64gb-50gb, and every create came back 422. curl -f
dropped the reason and set -e ended the tick, so builders only appeared when
capacity happened to free up, and the journal showed nothing but "curl: (22)".

Try each of SIZES in turn, logging DigitalOcean's refusal message, and fail
the tick only when every size is refused. Builders now power off however
start.sh exits, so a failed registration is reaped instead of counting as a
booting runner until MAX_AGE_MINUTES. The controller unit pulls the checkout
before each tick, so merged controller fixes reach the box.

* Create builders in any region that has the size in stock

ric1 sold out of g5-32vcpu-128gb-50gb within minutes of the box switching to
it. Builders need nothing from a particular region, so read DigitalOcean's
size catalog once per tick and try each of SIZES in every region it lists in
stock, REGIONS first if set. A refused pair is dropped for the rest of the
tick.

2026-10-08 10:50:06 -04:00

4.9 KiB

Raw Blame History

CI spike: build PRs on ephemeral DigitalOcean droplets

Status: spike. Nothing here publishes. The repository host keeps building and signing on merge exactly as before.

Pieces

.github/workflows/build-pr.yml — on a PR touching pkgbuilds/**, one job per changed package on runners labelled omarchy-builder. aarch64 jobs run on GitHub's native ubuntu-24.04-arm runners instead. Uploads the unsigned .pkg.tar.zst as a workflow artifact (7 days).
runner-cloud-init.yaml — Ubuntu 24.04 user-data: docker + buildx, the GitHub runner registered --ephemeral, runs one job, powers off.
controller.sh — systemd timer every minute on a small always-on droplet. Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet per job up to MAX_DROPLETS, deletes droplets that are powered off or older than MAX_AGE_MINUTES. Builders go in any region DigitalOcean lists the size in stock in (REGIONS only sets which to try first); a refused create, logged with DigitalOcean's message, falls back to the next region, then the next of SIZES. No inbound endpoint. Plain curl against both APIs, no doctl and no gh: a token in the environment cannot pick the wrong account the way a saved doctl context can. Needs curl and jq. tests/controller.sh exercises every decision against canned responses.
controller-box/ — the always-on droplet: unit, timer, env template, cloud-init, and create.sh to stand it up with one API call.

Standing up the controller box

DIGITALOCEAN_TOKEN=<omarchy account> GITHUB_TOKEN=<fine-grained PAT> \
  REPO=omacom/omarchy-pkgs ci/controller-box/create.sh <branch>

The GitHub PAT is fine-grained, scoped to the one repo: Actions read, Administration read+write (registration tokens). The DO token is baked into the box's env file, so it is the account that pays for builder droplets. Watch it with journalctl -u omarchy-controller -f on the box.

Each tick pulls the box's checkout first, so a merged controller.sh is live within a minute. The unit and timer are copies made at creation; after changing them, on the box:

cp /opt/omarchy-pkgs/ci/controller-box/omarchy-controller.{service,timer} /etc/systemd/system/
systemctl daemon-reload

What the spike proved (2026-09-17, fork ryanrhughes/omarchy-pkgs)

bin/build works from a bare clone: with no local published tree it plans against and resolves from https://pkgs.omarchy.org/<mirror>/<arch>.
Droplet create → runner registered: ~70 s. omarchy-fish PR job: 2 min including the builder image build. Droplet powers off after the job.
linux-omarchy on a c-32 droplet: 30 min wall clock for the build job (23:39 → 00:09), 254 MB artifact. Cold start ~90 s before the job began.
A PR whose PKGBUILD fails to build turns the required check red and GitHub refuses the merge (mergeStateStatus=BLOCKED, gh pr merge refuses without --admin).
Controller: one queued job + one busy droplet ⇒ creates exactly one more; reaps powered-off droplets on the next tick.

Not done (required before this touches the real repo)

Tooling from base: check out master's bin/ helpers/ build/ and overlay only the PR's pkgbuilds/<name>; today a PR can edit the build script and it runs on the droplet. The vouch gate limits who can do that, not what they can do.
DigitalOcean cloud firewall on the omarchy-builder tag: no inbound, no egress to private ranges or the metadata address.
A fine-grained GitHub token for the real repository (the one on the controller box is scoped to the fork), and the publish environment's secrets set there.
Disable the host's auto-release timers for any channel CI publishes to, so two writers never touch one database.

Done since the spike README was first written

Controller as a systemd timer on its own droplet, plain curl, self-test.
Build once against edge; one artifact per package per architecture, published into every channel it belongs to (fast ring: all three at once). arch=any builds once for every architecture database.
Publish is incremental and immutable: pull the channel db, refuse different bytes under an existing name, accept identical bytes, upload packages then signatures then the db.
aarch64 under QEMU with credential-preserving binfmt. PR builds now run aarch64 natively on ubuntu-24.04-arm (QEMU was up to ~15x slower). When a merged aarch64 tree has no artifact, publish.yml rebuilds it there too, in its own job, and signs and uploads it on the droplet like a PR artifact.
Vouch gate: collaborators, .github/VOUCHED.td, or the build-approved label; denounced authors cannot be overridden by the label.
Tests run on PRs only; result, self-tests, build-isolation are the required checks with strict up-to-date branches.

Cleanup

doctl compute droplet list --tag-name omarchy-builder
doctl compute droplet delete -f <id>

4.9 KiB Raw Blame History