Build PRs on ephemeral droplets; publish merged packages from CI

Every pull request now builds the package directories it touches on
ephemeral DigitalOcean droplets, and every merge to master publishes the
resulting artifacts into the channels each package belongs to. The
repository host's timers become the fallback rather than the pipeline.

Build (.github/workflows/build-pr.yml)
  One job per package per architecture, always against edge. The artifact
  is labelled with the package directory's git tree hash. Tooling (bin/,
  helpers/, build/) is checked out from the base branch; the PR supplies
  only pkgbuilds/, so a PR can change what is built, never how. Builds
  run only for trusted authors: collaborators, .github/VOUCHED.td, or a
  PR carrying the build-approved label. A single required check, result,
  aggregates the matrix.

Publish (.github/workflows/publish.yml, bin/publish-artifact)
  One job per merge. It collects the PR artifacts for the merged tree,
  builds anything that has none, then walks each channel/architecture
  slot once: pull that database, repo-add every package that belongs in
  it, upload packages, signatures, then the database. A published
  filename is immutable; identical bytes under an existing name only
  gain a database entry, different bytes are refused. Fast-ring packages
  reach edge, rc and stable in the same run from the same file.

Matrix (bin/build-matrix)
  Package x architecture, with the channels the artifact ships to,
  decided by package_builds_for_mirror so CI and the host agree.
  arch=any packages build once and land in every architecture database.

Builder (build/build.sh, bin/build, build/Dockerfile)
  With no local published tree, plan against and resolve from the public
  channel over HTTPS. PACKAGER is set to Omarchy. rclone is in the image.

Runners (ci/)
  A controller droplet polls GitHub with curl and creates one g5 droplet
  per queued job from cloud-init, deleting them when off or over-age.
  Builders carry QEMU with credential support for aarch64. Operator SSH
  keys come from GitHub. tests/controller.sh and tests/publish-artifact.sh
  cover the decisions against fixtures and real makepkg output.

Tests run on pull requests only; branch protection requires result,
self-tests and build-isolation with up-to-date branches.
This commit is contained in:
Ryan Hughes committed 2026-09-18 11:25:32 -04:00
1 parent b422d37fa2
commit 537c377fa5
19 files changed
+1119 -7

No files matched your search

+79
View File
@@ -0,0 +1,79 @@
# CI spike: build PRs on ephemeral DigitalOcean droplets
Status: spike. Nothing here publishes. The repository host keeps building and
signing on merge exactly as before.
## Pieces
- `.github/workflows/build-pr.yml` — on a PR touching `pkgbuilds/**`, one job
per changed package on runners labelled `omarchy-builder`. Uploads the
unsigned `.pkg.tar.zst` as a workflow artifact (7 days).
- `runner-cloud-init.yaml` — Ubuntu 24.04 user-data: docker + buildx, the
GitHub runner registered `--ephemeral`, runs one job, powers off.
- `controller.sh` — systemd timer every minute on a small always-on droplet.
Polls for queued jobs with our label, creates one g5-32vcpu-64gb-50gb droplet (ric1) per job up
to `MAX_DROPLETS`, deletes droplets that are powered off or older than
`MAX_AGE_MINUTES`. No inbound endpoint. Plain curl against both APIs, no
doctl and no gh: a token in the environment cannot pick the wrong account
the way a saved doctl context can. Needs curl and jq.
`tests/controller.sh` exercises every decision against canned responses.
- `controller-box/` — the always-on droplet: unit, timer, env template,
cloud-init, and `create.sh` to stand it up with one API call.
## Standing up the controller box
DIGITALOCEAN_TOKEN=<omarchy account> GITHUB_TOKEN=<fine-grained PAT> \
REPO=omacom/omarchy-pkgs ci/controller-box/create.sh <branch>
The GitHub PAT is fine-grained, scoped to the one repo: Actions read,
Administration read+write (registration tokens). The DO token is baked into
the box's env file, so it is the account that pays for builder droplets.
Watch it with `journalctl -u omarchy-controller -f` on the box.
## What the spike proved (2026-09-17, fork ryanrhughes/omarchy-pkgs)
- `bin/build` works from a bare clone: with no local published tree it
plans against and resolves from `https://pkgs.omarchy.org/<mirror>/<arch>`.
- Droplet create → runner registered: ~70 s. omarchy-fish PR job: 2 min
including the builder image build. Droplet powers off after the job.
- linux-omarchy on a c-32 droplet: 30 min wall clock for the build job
(23:39 → 00:09), 254 MB artifact. Cold start ~90 s before the job began.
- A PR whose PKGBUILD fails to build turns the required check red and GitHub
refuses the merge (`mergeStateStatus=BLOCKED`, `gh pr merge` refuses
without `--admin`).
- Controller: one queued job + one busy droplet ⇒ creates exactly one more;
reaps powered-off droplets on the next tick.
## Not done (required before this touches the real repo)
- Tooling from base: check out master's `bin/ helpers/ build/` and overlay
only the PR's `pkgbuilds/<name>`; today a PR can edit the build script
and it runs on the droplet. The vouch gate limits who can do that, not
what they can do.
- DigitalOcean cloud firewall on the `omarchy-builder` tag: no inbound, no
egress to private ranges or the metadata address.
- A fine-grained GitHub token for the real repository (the one on the
controller box is scoped to the fork), and the publish environment's
secrets set there.
- Disable the host's auto-release timers for any channel CI publishes to,
so two writers never touch one database.
## Done since the spike README was first written
- Controller as a systemd timer on its own droplet, plain curl, self-test.
- Build once against edge; one artifact per package per architecture,
published into every channel it belongs to (fast ring: all three at
once). arch=any builds once for every architecture database.
- Publish is incremental and immutable: pull the channel db, refuse
different bytes under an existing name, accept identical bytes, upload
packages then signatures then the db.
- aarch64 under QEMU with credential-preserving binfmt.
- Vouch gate: collaborators, `.github/VOUCHED.td`, or the `build-approved`
label; denounced authors cannot be overridden by the label.
- Tests run on PRs only; `result`, `self-tests`, `build-isolation` are the
required checks with strict up-to-date branches.
## Cleanup
doctl compute droplet list --tag-name omarchy-builder
doctl compute droplet delete -f <id>
+45
View File
@@ -0,0 +1,45 @@
#cloud-config
# The always-on controller droplet (smallest size is fine). Clones the repo
# for ci/controller.sh, installs the unit and timer, and starts polling.
#
# Substitute before use:
# __REPO_URL__ https://github.com/omacom/omarchy-pkgs.git
# __BRANCH__ branch carrying ci/ (master once merged)
# __ENV_B64__ base64 of a filled-in controller.env.example
# __SSH_KEYS_JSON__ JSON array of public keys authorized for root
package_update: true
packages: [curl, jq, git]
# Root stays reachable by key so the journal can be read. Two things stand
# in the way on DO images: disable_root rewrites root's keys into a stub, and
# with no account ssh key attached DO expires root's password, which makes
# sshd refuse every non-interactive session with "password change required".
disable_root: false
chpasswd:
expire: false
ssh_authorized_keys: __SSH_KEYS_JSON__
users:
- name: controller
shell: /bin/bash
write_files:
# defer: write after the users module has created the controller group,
# otherwise chown to root:controller fails and the unit cannot read this.
- path: /etc/omarchy-controller.env
permissions: "0640"
owner: root:controller
encoding: b64
defer: true
content: __ENV_B64__
runcmd:
- chage -d "$(date +%F)" -M -1 root
- chown root:controller /etc/omarchy-controller.env && chmod 0640 /etc/omarchy-controller.env
- git clone --depth 1 --branch __BRANCH__ __REPO_URL__ /opt/omarchy-pkgs
- mkdir -p /run/omarchy-controller && chown controller:controller /run/omarchy-controller
- echo "d /run/omarchy-controller 0755 controller controller -" > /etc/tmpfiles.d/omarchy-controller.conf
# runcmd is executed by /bin/sh: no brace expansion.
- cp /opt/omarchy-pkgs/ci/controller-box/omarchy-controller.service /opt/omarchy-pkgs/ci/controller-box/omarchy-controller.timer /etc/systemd/system/
- systemctl daemon-reload
- systemctl enable --now omarchy-controller.timer
+15
View File
@@ -0,0 +1,15 @@
# /etc/omarchy-controller.env — mode 0600, owned by root, read by systemd.
DIGITALOCEAN_TOKEN=dop_v1_...
# Fine-grained PAT scoped to the repo: Actions: read, Administration: read+write
GITHUB_TOKEN=github_pat_...
REPO=omacom/omarchy-pkgs
LABEL=omarchy-builder
TAG=omarchy-builder
REGION=ric1
SIZE=g5-32vcpu-64gb-50gb
MAX_DROPLETS=6
MAX_AGE_MINUTES=200
LOCK=/run/omarchy-controller/lock
# Operator public keys for root on every builder droplet (JSON array).
# create.sh fills this from the operators' GitHub keys.
SSH_KEYS_JSON=[]
+41
View File
@@ -0,0 +1,41 @@
#!/bin/bash
# Create the controller droplet with plain curl. Run from a laptop, once.
#
# DIGITALOCEAN_TOKEN=... GITHUB_TOKEN=... ci/controller-box/create.sh [branch]
#
# The DO token given here is baked into the box's env file, so it must be the
# token for the account that should pay for builder droplets.
set -euo pipefail
here=$(dirname "$0")
: "${DIGITALOCEAN_TOKEN:?}" "${GITHUB_TOKEN:?}"
REPO=${REPO:-omacom/omarchy-pkgs}
BRANCH=${1:-master}
REGION=${REGION:-ric1}
NAME=${NAME:-omarchy-controller}
# Optional DO ssh key ids as a JSON array, e.g. SSH_KEYS='[123]', for reading
# the journal while bringing the box up. Not needed once it works.
SSH_KEYS=${SSH_KEYS:-[]}
# Public keys authorized for root: the operators' GitHub keys, fetched at
# creation so the box never depends on an ssh_key API scope. Override with
# ADMIN_GITHUB_USERS.
ADMIN_GITHUB_USERS=${ADMIN_GITHUB_USERS:-ryanrhughes dhh}
ssh_keys_json=$(for u in $ADMIN_GITHUB_USERS; do curl -fsS "https://github.com/$u.keys"; done | jq -R . | jq -sc .)
[[ $(jq length <<<"$ssh_keys_json") -gt 0 ]] || { echo "no ssh keys fetched for $ADMIN_GITHUB_USERS" >&2; exit 1; }
env_file=$(sed -e "s|^DIGITALOCEAN_TOKEN=.*|DIGITALOCEAN_TOKEN=$DIGITALOCEAN_TOKEN|" \
-e "s|^GITHUB_TOKEN=.*|GITHUB_TOKEN=$GITHUB_TOKEN|" \
-e "s|^REPO=.*|REPO=$REPO|" \
-e "s|^SSH_KEYS_JSON=.*|SSH_KEYS_JSON=$ssh_keys_json|" "$here/controller.env.example")
userdata=$(sed -e "s|__REPO_URL__|https://github.com/$REPO.git|" -e "s|__BRANCH__|$BRANCH|" \
-e "s|__ENV_B64__|$(printf '%s\n' "$env_file" | base64 -w0)|" \
-e "s|__SSH_KEYS_JSON__|$ssh_keys_json|" "$here/cloud-init.yaml")
body=$(jq -n --arg name "$NAME" --arg region "$REGION" --arg ud "$userdata" --argjson keys "$SSH_KEYS" \
'{name:$name, region:$region, size:"s-1vcpu-1gb", image:"ubuntu-24-04-x64", tags:["omarchy-controller"], user_data:$ud, ssh_keys:$keys}')
# Refuse to create a second one.
existing=$(curl -fsS -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
"https://api.digitalocean.com/v2/droplets?tag_name=omarchy-controller" | jq '.droplets | length')
if (( existing > 0 )); then echo "a controller droplet already exists" >&2; exit 1; fi
curl -fsS -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" -H "Content-Type: application/json" \
-X POST -d "$body" https://api.digitalocean.com/v2/droplets | jq -r '"created \(.droplet.name) id=\(.droplet.id)"'
@@ -0,0 +1,12 @@
[Unit]
Description=Provision ephemeral omarchy-builder runner droplets for queued jobs
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=controller
EnvironmentFile=/etc/omarchy-controller.env
ExecStart=/opt/omarchy-pkgs/ci/controller.sh
# The reaper's safety net is time, not state; a hung tick must not hold the lock.
TimeoutStartSec=240
@@ -0,0 +1,10 @@
[Unit]
Description=Run the omarchy-builder controller every minute
[Timer]
OnBootSec=1min
OnUnitActiveSec=1min
AccuracySec=5s
[Install]
WantedBy=timers.target
+125
View File
@@ -0,0 +1,125 @@
#!/bin/bash
# Droplet-per-job controller for the omarchy-builder runner pool.
#
# Run from a systemd timer every minute on a small always-on droplet. No
# inbound endpoint: it polls GitHub for queued jobs wanting our label, creates
# one ephemeral droplet per job (up to MAX_DROPLETS), and deletes droplets
# that have powered off or exceeded MAX_AGE_MINUTES. The reaper does not
# trust its own bookkeeping: it lists by tag and acts on what DigitalOcean
# reports.
#
# Talks to both APIs with curl. No doctl: its saved contexts silently choose
# an account; a token in the environment cannot. Needs curl and jq.
#
# Environment:
# DIGITALOCEAN_TOKEN DO API token for the account that pays for droplets
# GITHUB_TOKEN fine-grained PAT: Actions read, Administration write
# REPO owner/name
set -euo pipefail
REPO=${REPO:?owner/name}
: "${DIGITALOCEAN_TOKEN:?}" "${GITHUB_TOKEN:?}"
LABEL=${LABEL:-omarchy-builder}
TAG=${TAG:-omarchy-builder}
REGION=${REGION:-ric1}
SIZE=${SIZE:-g5-32vcpu-64gb-50gb}
IMAGE=${IMAGE:-ubuntu-24-04-x64}
MAX_DROPLETS=${MAX_DROPLETS:-4}
MAX_AGE_MINUTES=${MAX_AGE_MINUTES:-200}
RUNNER_VERSION=${RUNNER_VERSION:-2.337.0}
CLOUD_INIT=${CLOUD_INIT:-$(dirname "$0")/runner-cloud-init.yaml}
# Operator public keys authorized on every builder (JSON array of strings).
# The box's env file carries them; empty means no root login.
SSH_KEYS_JSON=${SSH_KEYS_JSON:-[]}
LOCK=${LOCK:-/tmp/omarchy-controller.lock}
log() { echo "$(date '+%F %T') $*"; }
# The only two places the outside world is touched. The self-test overrides
# both, so every decision below is exercised against canned responses.
do_api() { # do_api <path> [curl args...]
local path=$1; shift
curl -fsS -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
-H "Content-Type: application/json" "https://api.digitalocean.com/v2/$path" "$@"
}
gh_api() { # gh_api <path> [curl args...]
local path=$1; shift
curl -fsS -H "Authorization: Bearer $GITHUB_TOKEN" \
-H "Accept: application/vnd.github+json" "https://api.github.com/$path" "$@"
}
# --- reap ------------------------------------------------------------------
reap() {
local now id status created age
now=$(date +%s)
while read -r id status created; do
[[ -n "$id" ]] || continue
age=$(( (now - $(date -d "$created" +%s)) / 60 ))
if [[ $status == off ]] || (( age > MAX_AGE_MINUTES )); then
log "deleting droplet $id (status=$status age=${age}m)"
do_api "droplets/$id" -X DELETE
fi
done < <(do_api "droplets?tag_name=$TAG&per_page=200" |
jq -r '.droplets[] | "\(.id) \(.status) \(.created_at)"')
}
# --- demand ----------------------------------------------------------------
queued_jobs() {
local run
gh_api "repos/$REPO/actions/runs?status=queued&per_page=50" --get \
| jq -r '.workflow_runs[].id' |
while read -r run; do
gh_api "repos/$REPO/actions/runs/$run/jobs" \
| jq -r --arg l "$LABEL" '.jobs[] | select(.status=="queued") | select(.labels | index($l)) | .id'
done | wc -l
}
live_droplets() {
do_api "droplets?tag_name=$TAG&per_page=200" | jq '[.droplets[] | select(.status != "off")] | length'
}
busy_runners() {
gh_api "repos/$REPO/actions/runners?per_page=100" \
| jq --arg l "$LABEL" '[.runners[] | select(.busy) | select(any(.labels[]; .name == $l))] | length'
}
# --- create ----------------------------------------------------------------
create_droplet() {
local token userdata name body
token=$(gh_api "repos/$REPO/actions/runners/registration-token" -X POST | jq -r .token)
userdata=$(sed -e "s|__REPO__|$REPO|g" -e "s|__RUNNER_TOKEN__|$token|g" \
-e "s|__RUNNER_LABELS__|$LABEL|g" -e "s|__RUNNER_VERSION__|$RUNNER_VERSION|g" \
-e "s|__SSH_KEYS_JSON__|$SSH_KEYS_JSON|" "$CLOUD_INIT")
name="$TAG-$(date +%s)-$RANDOM"
body=$(jq -n --arg name "$name" --arg region "$REGION" --arg size "$SIZE" --arg image "$IMAGE" \
--arg tag "$TAG" --arg ud "$userdata" \
'{name:$name, region:$region, size:$size, image:$image, tags:[$tag], user_data:$ud, monitoring:false}')
log "creating $name ($SIZE)"
do_api droplets -X POST -d "$body" | jq -r '"created droplet \(.droplet.id)"'
}
controller_tick() {
reap
local queued live busy available need room
queued=$(queued_jobs)
live=$(live_droplets)
busy=$(busy_runners)
# A live droplet whose runner is busy is spoken for. Only droplets still
# booting or listening can absorb a queued job.
available=$(( live - busy )); (( available < 0 )) && available=0
need=$(( queued - available ))
(( need > 0 )) || return 0
room=$(( MAX_DROPLETS - live ))
(( need > room )) && need=$room
if (( need <= 0 )); then
log "at cap ($live/$MAX_DROPLETS, $busy busy) with $queued queued"
return 0
fi
local i
for (( i = 0; i < need; i++ )); do create_droplet; done
}
if [[ "${CONTROLLER_LIBRARY_ONLY:-}" != 1 ]]; then
exec 9>"$LOCK"; flock -n 9 || exit 0
controller_tick
fi
+77
View File
@@ -0,0 +1,77 @@
#cloud-config
# Ephemeral GitHub Actions runner for omarchy-pkgs package builds.
#
# Boots an Ubuntu droplet, installs docker and the runner, registers ONCE with
# --ephemeral, runs exactly one job, then powers off. The controller (or the
# reaper) deletes the powered-off droplet. Nothing here holds a long-lived
# credential: the registration token is single-use and expires in an hour.
#
# Substitute before use:
# __REPO__ owner/name
# __RUNNER_TOKEN__ registration token (gh api -X POST repos/O/R/actions/runners/registration-token)
# __RUNNER_LABELS__ e.g. omarchy-builder
# __RUNNER_VERSION__ e.g. 2.329.0
# Operators can reach a builder by key while it lives; it powers off after
# one job anyway. Keys are substituted by the controller (__SSH_KEYS_JSON__).
disable_root: false
chpasswd:
expire: false
ssh_authorized_keys: __SSH_KEYS_JSON__
package_update: true
packages:
- docker.io
- docker-buildx
- unzip
- git
- curl
- jq
- rsync
users:
- name: runner
groups: [docker]
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
write_files:
# defer: write after users/groups exist, so /home/runner is created by
# useradd (owned by runner) rather than by this module as root.
- path: /home/runner/start.sh
permissions: "0755"
owner: runner:runner
defer: true
content: |
#!/bin/bash
set -euo pipefail
cd /home/runner
mkdir -p actions-runner && cd actions-runner
arch=$(uname -m); [[ $arch == x86_64 ]] && arch=x64
curl -fsSL -o runner.tgz \
"https://github.com/actions/runner/releases/download/v__RUNNER_VERSION__/actions-runner-linux-${arch}-__RUNNER_VERSION__.tar.gz"
tar xzf runner.tgz && rm runner.tgz
./config.sh --unattended --ephemeral \
--url "https://github.com/__REPO__" \
--token "__RUNNER_TOKEN__" \
--name "do-$(hostname)" \
--labels "__RUNNER_LABELS__" \
--replace
./run.sh
# One job done. Power off; the controller deletes powered-off droplets.
sudo poweroff
runcmd:
# With no account ssh key attached, DO expires root's password, and sshd
# then refuses every non-interactive session. Clear it first so operators
# can read the logs of a builder that never registers.
- chage -d "$(date +%F)" -M -1 root
- systemctl enable --now docker
# aarch64 builds run under user-mode emulation (DO has no arm droplets).
# Register QEMU with the F and C flags via the multiarch image, exactly as
# helpers/docker-helpers.sh setup_qemu does: Ubuntu's qemu-user-static
# package registers without C, so sudo inside the emulated container fails
# with "effective uid is not 0". Best-effort: an x86-only job never needs it.
- docker run --rm --privileged docker.io/multiarch/qemu-user-static --reset -p yes --credential yes || true
- chown -R runner:runner /home/runner
- sudo -u runner /home/runner/start.sh > /home/runner/runner.log 2>&1