# CI spike: build PRs on ephemeral DigitalOcean droplets

Status: spike. Nothing here publishes. The repository host keeps building and
signing on merge exactly as before.

## Pieces

- `.github/workflows/build-pr.yml` — on a PR touching `pkgbuilds/**`, one job
  per changed package on runners labelled `omarchy-builder`. aarch64 jobs
  run on GitHub's native `ubuntu-24.04-arm` runners instead. Uploads the unsigned
  `.pkg.tar.zst` as a workflow artifact (7 days).
- `runner-cloud-init.yaml` — Ubuntu 24.04 user-data: docker + buildx, the
  GitHub runner registered `--ephemeral`, runs one job, powers off.
- `controller.sh` — systemd timer every minute on a small always-on droplet.
  Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet (ric1) per job up
  to `MAX_DROPLETS`, deletes droplets that are powered off or older than
  `MAX_AGE_MINUTES`. No inbound endpoint. Plain curl against both APIs, no
  doctl and no gh: a token in the environment cannot pick the wrong account
  the way a saved doctl context can. Needs curl and jq.
  `tests/controller.sh` exercises every decision against canned responses.
- `controller-box/` — the always-on droplet: unit, timer, env template,
  cloud-init, and `create.sh` to stand it up with one API call.

## Standing up the controller box

    DIGITALOCEAN_TOKEN=<omarchy account> GITHUB_TOKEN=<fine-grained PAT> \
      REPO=omacom/omarchy-pkgs ci/controller-box/create.sh <branch>

The GitHub PAT is fine-grained, scoped to the one repo: Actions read,
Administration read+write (registration tokens). The DO token is baked into
the box's env file, so it is the account that pays for builder droplets.
Watch it with `journalctl -u omarchy-controller -f` on the box.

## What the spike proved (2026-09-17, fork ryanrhughes/omarchy-pkgs)

- `bin/build` works from a bare clone: with no local published tree it
  plans against and resolves from `https://pkgs.omarchy.org/<mirror>/<arch>`.
- Droplet create → runner registered: ~70 s. omarchy-fish PR job: 2 min
  including the builder image build. Droplet powers off after the job.
- linux-omarchy on a c-32 droplet: 30 min wall clock for the build job
  (23:39 → 00:09), 254 MB artifact. Cold start ~90 s before the job began.
- A PR whose PKGBUILD fails to build turns the required check red and GitHub
  refuses the merge (`mergeStateStatus=BLOCKED`, `gh pr merge` refuses
  without `--admin`).
- Controller: one queued job + one busy droplet ⇒ creates exactly one more;
  reaps powered-off droplets on the next tick.

## Not done (required before this touches the real repo)

- Tooling from base: check out master's `bin/ helpers/ build/` and overlay
  only the PR's `pkgbuilds/<name>`; today a PR can edit the build script
  and it runs on the droplet. The vouch gate limits who can do that, not
  what they can do.
- DigitalOcean cloud firewall on the `omarchy-builder` tag: no inbound, no
  egress to private ranges or the metadata address.
- A fine-grained GitHub token for the real repository (the one on the
  controller box is scoped to the fork), and the publish environment's
  secrets set there.
- Disable the host's auto-release timers for any channel CI publishes to,
  so two writers never touch one database.

## Done since the spike README was first written

- Controller as a systemd timer on its own droplet, plain curl, self-test.
- Build once against edge; one artifact per package per architecture,
  published into every channel it belongs to (fast ring: all three at
  once). arch=any builds once for every architecture database.
- Publish is incremental and immutable: pull the channel db, refuse
  different bytes under an existing name, accept identical bytes, upload
  packages then signatures then the db.
- aarch64 under QEMU with credential-preserving binfmt. PR builds now run
  aarch64 natively on `ubuntu-24.04-arm` (QEMU was up to ~15x slower). When a
  merged aarch64 tree has no artifact, publish.yml rebuilds it there too, in its
  own job, and signs and uploads it on the droplet like a PR artifact.
- Vouch gate: collaborators, `.github/VOUCHED.td`, or the `build-approved`
  label; denounced authors cannot be overridden by the label.
- Tests run on PRs only; `result`, `self-tests`, `build-isolation` are the
  required checks with strict up-to-date branches.

## Cleanup

    doctl compute droplet list --tag-name omarchy-builder
    doctl compute droplet delete -f <id>