Marcelo Alcantara
97bcb320ab
A merged aarch64 tree without a PR build artifact (expired after 7 days, or a dispatch) was rebuilt on the x86 droplet under QEMU: omarchy-mac-boot took ~167 of the 240 minutes. A new job builds it on ubuntu-24.04-arm the way build-pr.yml does and uploads it under the same label, so the publish job signs and uploads it on the droplet like a PR artifact. The plan logs reuse or rebuild for every package; x86_64, signing and the publish concurrency are unchanged.
84 lines
4.3 KiB
Markdown
84 lines
4.3 KiB
Markdown
# CI spike: build PRs on ephemeral DigitalOcean droplets
|
|
|
|
Status: spike. Nothing here publishes. The repository host keeps building and
|
|
signing on merge exactly as before.
|
|
|
|
## Pieces
|
|
|
|
- `.github/workflows/build-pr.yml` — on a PR touching `pkgbuilds/**`, one job
|
|
per changed package on runners labelled `omarchy-builder`. aarch64 jobs
|
|
run on GitHub's native `ubuntu-24.04-arm` runners instead. Uploads the unsigned
|
|
`.pkg.tar.zst` as a workflow artifact (7 days).
|
|
- `runner-cloud-init.yaml` — Ubuntu 24.04 user-data: docker + buildx, the
|
|
GitHub runner registered `--ephemeral`, runs one job, powers off.
|
|
- `controller.sh` — systemd timer every minute on a small always-on droplet.
|
|
Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet (ric1) per job up
|
|
to `MAX_DROPLETS`, deletes droplets that are powered off or older than
|
|
`MAX_AGE_MINUTES`. No inbound endpoint. Plain curl against both APIs, no
|
|
doctl and no gh: a token in the environment cannot pick the wrong account
|
|
the way a saved doctl context can. Needs curl and jq.
|
|
`tests/controller.sh` exercises every decision against canned responses.
|
|
- `controller-box/` — the always-on droplet: unit, timer, env template,
|
|
cloud-init, and `create.sh` to stand it up with one API call.
|
|
|
|
## Standing up the controller box
|
|
|
|
DIGITALOCEAN_TOKEN=<omarchy account> GITHUB_TOKEN=<fine-grained PAT> \
|
|
REPO=omacom/omarchy-pkgs ci/controller-box/create.sh <branch>
|
|
|
|
The GitHub PAT is fine-grained, scoped to the one repo: Actions read,
|
|
Administration read+write (registration tokens). The DO token is baked into
|
|
the box's env file, so it is the account that pays for builder droplets.
|
|
Watch it with `journalctl -u omarchy-controller -f` on the box.
|
|
|
|
## What the spike proved (2026-09-17, fork ryanrhughes/omarchy-pkgs)
|
|
|
|
- `bin/build` works from a bare clone: with no local published tree it
|
|
plans against and resolves from `https://pkgs.omarchy.org/<mirror>/<arch>`.
|
|
- Droplet create → runner registered: ~70 s. omarchy-fish PR job: 2 min
|
|
including the builder image build. Droplet powers off after the job.
|
|
- linux-omarchy on a c-32 droplet: 30 min wall clock for the build job
|
|
(23:39 → 00:09), 254 MB artifact. Cold start ~90 s before the job began.
|
|
- A PR whose PKGBUILD fails to build turns the required check red and GitHub
|
|
refuses the merge (`mergeStateStatus=BLOCKED`, `gh pr merge` refuses
|
|
without `--admin`).
|
|
- Controller: one queued job + one busy droplet ⇒ creates exactly one more;
|
|
reaps powered-off droplets on the next tick.
|
|
|
|
## Not done (required before this touches the real repo)
|
|
|
|
- Tooling from base: check out master's `bin/ helpers/ build/` and overlay
|
|
only the PR's `pkgbuilds/<name>`; today a PR can edit the build script
|
|
and it runs on the droplet. The vouch gate limits who can do that, not
|
|
what they can do.
|
|
- DigitalOcean cloud firewall on the `omarchy-builder` tag: no inbound, no
|
|
egress to private ranges or the metadata address.
|
|
- A fine-grained GitHub token for the real repository (the one on the
|
|
controller box is scoped to the fork), and the publish environment's
|
|
secrets set there.
|
|
- Disable the host's auto-release timers for any channel CI publishes to,
|
|
so two writers never touch one database.
|
|
|
|
## Done since the spike README was first written
|
|
|
|
- Controller as a systemd timer on its own droplet, plain curl, self-test.
|
|
- Build once against edge; one artifact per package per architecture,
|
|
published into every channel it belongs to (fast ring: all three at
|
|
once). arch=any builds once for every architecture database.
|
|
- Publish is incremental and immutable: pull the channel db, refuse
|
|
different bytes under an existing name, accept identical bytes, upload
|
|
packages then signatures then the db.
|
|
- aarch64 under QEMU with credential-preserving binfmt. PR builds now run
|
|
aarch64 natively on `ubuntu-24.04-arm` (QEMU was up to ~15x slower). When a
|
|
merged aarch64 tree has no artifact, publish.yml rebuilds it there too, in its
|
|
own job, and signs and uploads it on the droplet like a PR artifact.
|
|
- Vouch gate: collaborators, `.github/VOUCHED.td`, or the `build-approved`
|
|
label; denounced authors cannot be overridden by the label.
|
|
- Tests run on PRs only; `result`, `self-tests`, `build-isolation` are the
|
|
required checks with strict up-to-date branches.
|
|
|
|
## Cleanup
|
|
|
|
doctl compute droplet list --tag-name omarchy-builder
|
|
doctl compute droplet delete -f <id>
|