Build PRs on ephemeral droplets; publish merged packages from CI

Every pull request now builds the package directories it touches on
ephemeral DigitalOcean droplets, and every merge to master publishes the
resulting artifacts into the channels each package belongs to. The
repository host's timers become the fallback rather than the pipeline.

Build (.github/workflows/build-pr.yml)
  One job per package per architecture, always against edge. The artifact
  is labelled with the package directory's git tree hash. Tooling (bin/,
  helpers/, build/) is checked out from the base branch; the PR supplies
  only pkgbuilds/, so a PR can change what is built, never how. Builds
  run only for trusted authors: collaborators, .github/VOUCHED.td, or a
  PR carrying the build-approved label. A single required check, result,
  aggregates the matrix.

Publish (.github/workflows/publish.yml, bin/publish-artifact)
  One job per merge. It collects the PR artifacts for the merged tree,
  builds anything that has none, then walks each channel/architecture
  slot once: pull that database, repo-add every package that belongs in
  it, upload packages, signatures, then the database. A published
  filename is immutable; identical bytes under an existing name only
  gain a database entry, different bytes are refused. Fast-ring packages
  reach edge, rc and stable in the same run from the same file.

Matrix (bin/build-matrix)
  Package x architecture, with the channels the artifact ships to,
  decided by package_builds_for_mirror so CI and the host agree.
  arch=any packages build once and land in every architecture database.

Builder (build/build.sh, bin/build, build/Dockerfile)
  With no local published tree, plan against and resolve from the public
  channel over HTTPS. PACKAGER is set to Omarchy. rclone is in the image.

Runners (ci/)
  A controller droplet polls GitHub with curl and creates one g5 droplet
  per queued job from cloud-init, deleting them when off or over-age.
  Builders carry QEMU with credential support for aarch64. Operator SSH
  keys come from GitHub. tests/controller.sh and tests/publish-artifact.sh
  cover the decisions against fixtures and real makepkg output.

Tests run on pull requests only; branch protection requires result,
self-tests and build-isolation with up-to-date branches.
This commit is contained in:
Ryan Hughes committed 2026-09-18 11:25:32 -04:00
1 parent b422d37fa2
commit 537c377fa5
19 files changed
+1119 -7

No files matched your search

+3
View File
@@ -92,6 +92,8 @@ while [[ $# -gt 0 ]]; do
echo " OMARCHY_KEEP_BUILD_WORKSPACE=1 keep build-output and reuse packages already there"
echo " OMARCHY_SKIP_BUILDER_IMAGE=1 use the existing builder image instead of building it"
echo " OMARCHY_DEFER_RUNTIME_DEPS=true build the omarchy pair with --nodeps (transaction verified later)"
echo " OMARCHY_PUBLISHED_REPO_URL=<url> channel to plan and resolve against when no local tree exists"
echo " (default https://pkgs.omarchy.org; empty disables the fallback)"
echo ""
exit 0
;;
@@ -256,6 +258,7 @@ DOCKER_ARGS=(
-e MIRROR="$MIRROR"
-e PACKAGES="$PACKAGES"
-e OMARCHY_RC_PINS="${OMARCHY_RC_PINS:-}"
-e OMARCHY_PUBLISHED_REPO_URL="${OMARCHY_PUBLISHED_REPO_URL-https://pkgs.omarchy.org}"
-e DEFER_RUNTIME_DEPS="$DEFER_RUNTIME_DEPS"
-e BUILD_PLAN_DIR=/build-plan
-v "$PLAN_DIR:/build-plan"
+54
View File
@@ -0,0 +1,54 @@
#!/bin/bash
# Print the PR build matrix for a set of package directories as JSON: one
# entry per package per supported architecture. Every package builds exactly
# once, against edge, and that one artifact is what every channel ships:
# channels are databases over a shared pool of files, and a filename must
# mean one set of bytes. "channels" lists where the artifact is published on
# merge: edge for everything, plus rc and stable immediately for the fast
# ring. Eligibility comes from package_builds_for_mirror, the rule the
# release host uses, so CI and the host cannot disagree.
#
# Usage: build-matrix [--arch <arch>|all] <package>...
# Reads package names on stdin when none are given. With no --arch, every
# architecture in CI_ARCHES (default "x86_64 aarch64") the package supports.
# Output: {"include":[{"package":"x","arch":"x86_64","channels":"edge rc stable","publish_arches":"x86_64"},...]}
# arch is where it builds; publish_arches lists every architecture
# database the file goes into (all of them for arch=any).
set -euo pipefail
BUILD_ROOT=$(realpath "${BASH_SOURCE[0]%/*}/..")
source "$BUILD_ROOT/helpers/paths.sh"
source "$BUILD_ROOT/helpers/package-metadata.sh"
ARCHES=${CI_ARCHES:-x86_64 aarch64}
if [[ "${1:-}" == --arch ]]; then [[ $2 == all ]] || ARCHES=$2; shift 2; fi
for a in $ARCHES; do require_valid_arch "$a"; done
if (( $# )); then names=("$@"); else mapfile -t names; fi
entries=()
for name in "${names[@]}"; do
[[ -n "$name" ]] || continue
pkgdir="$PKGBUILDS_DIR/$name"
[[ -d "$pkgdir" ]] || continue
# skip_build packages still build on their own PR (explicit --package
# semantics); the host's unscoped runs are what skip them.
channels=""
for mirror in $VALID_MIRRORS; do
package_builds_for_mirror "$pkgdir" "$mirror" && channels="$channels $mirror"
done
channels=${channels# }
[[ -n "$channels" ]] || continue
# An arch=any package produces one architecture-independent file, so it
# builds once, on the first architecture, and that file serves every
# channel database of every architecture.
if [[ " $(package_arches "$pkgdir" "${ARCHES%% *}") " == *" any "* ]]; then
entries+=("$(jq -nc --arg p "$name" --arg a "${ARCHES%% *}" --arg c "$channels" --arg pa "$ARCHES" '{package:$p, arch:$a, channels:$c, publish_arches:$pa}')")
continue
fi
for arch in $ARCHES; do
package_supports_arch "$pkgdir" "$arch" || continue
entries+=("$(jq -nc --arg p "$name" --arg a "$arch" --arg c "$channels" '{package:$p, arch:$a, channels:$c, publish_arches:$a}')")
done
done
printf '%s\n' "${entries[@]}" | jq -sc '{include: .}'
+118
View File
@@ -0,0 +1,118 @@
#!/bin/bash
# Publish built packages into one channel of the remote repository,
# incrementally and immutably.
#
# publish-artifact --mirror <edge|rc|stable> --arch <arch> <pkg files...>
#
# What it does, in order:
# 1. pull the channel's current database from the remote
# 2. refuse if any package filename already exists on the remote
# 3. sign each package (GPG_PRIVATE_KEY / GPG_PASSPHRASE)
# 4. repo-add the packages into the pulled database (replaces the entry
# for that name; nothing else in the channel is touched)
# 5. upload packages, then signatures, then the database last
#
# Never overwrites: uploads use --ignore-existing for packages and the
# pre-check in step 2 makes a same-name collision a hard failure rather than
# a silent skip. The database is the only object rewritten, and it is
# uploaded only after every file it references is present.
#
# The remote is an rclone remote (REMOTE, default the production one);
# OMARCHY_PUBLISH_PREFIX can point a proof run at a scratch prefix.
set -euo pipefail
BUILD_ROOT=$(realpath "${BASH_SOURCE[0]%/*}/..")
source "$BUILD_ROOT/helpers/message-helpers.sh"
source "$BUILD_ROOT/helpers/paths.sh"
REMOTE=${REMOTE:-pkgs.omarchy.org:omarchy-pkgs}
PREFIX=${OMARCHY_PUBLISH_PREFIX:-}
FILES=()
while [[ $# -gt 0 ]]; do
case $1 in
--mirror) MIRROR=$2; require_valid_mirror "$MIRROR"; shift 2 ;;
--arch) ARCH=$2; require_valid_arch "$ARCH"; shift 2 ;;
--remote) REMOTE=$2; shift 2 ;;
-h|--help) sed -n '2,22p' "$0"; exit 0 ;;
-*) print_error "Unknown option: $1"; exit 1 ;;
*) FILES+=("$1"); shift ;;
esac
done
(( ${#FILES[@]} )) || { print_error "No package files given"; exit 1; }
: "${GPG_PRIVATE_KEY:?}"; GPG_PASSPHRASE=${GPG_PASSPHRASE-}
DEST="$REMOTE/${PREFIX:+$PREFIX/}$MIRROR/$ARCH"
WORK=$(mktemp -d)
trap 'rm -rf "$WORK"' EXIT
print_header "Publish to $DEST"
# --- 0. sanity: every file is a package, named as makepkg names it ---------
for f in "${FILES[@]}"; do
[[ -f "$f" && "$f" == *.pkg.tar.* && "$f" != *.sig ]] || { print_error "Not a package file: $f"; exit 1; }
name=$(bsdtar -xOf "$f" .PKGINFO | awk -F' = ' '$1=="pkgname"{print $2}')
ver=$(bsdtar -xOf "$f" .PKGINFO | awk -F' = ' '$1=="pkgver"{print $2}')
pkgarch=$(bsdtar -xOf "$f" .PKGINFO | awk -F' = ' '$1=="arch"{print $2}')
[[ "$(basename "$f")" == "$name-$ver-$pkgarch.pkg.tar."* ]] || {
print_error "Filename does not match PKGINFO ($name-$ver-$pkgarch): $(basename "$f")"; exit 1; }
[[ "$pkgarch" == any || "$pkgarch" == "$ARCH" ]] || { print_error "$f is $pkgarch, publishing to $ARCH"; exit 1; }
done
# --- 1. pull the current database -----------------------------------------
mkdir -p "$WORK/repo"
listing=$(rclone lsf "$DEST/" --s3-no-head 2>/dev/null || true)
if grep -q '^omarchy.db.tar.zst$' <<<"$listing"; then
rclone copy "$DEST/omarchy.db.tar.zst" "$WORK/repo" --s3-no-head
rclone copy "$DEST/omarchy.files.tar.zst" "$WORK/repo" --s3-no-head 2>/dev/null || true
print_info "Pulled current database ($(tar -tf "$WORK/repo/omarchy.db.tar.zst" | grep -c '/$') entries)"
else
print_warning "No database at $DEST — creating a new one"
fi
# --- 2. same-name collisions ----------------------------------------------
# A filename must mean one set of bytes across every channel. The same file
# reaching a channel that already holds it (a fast-ring publish after edge,
# a re-run, a later promotion) is fine: it is skipped on upload and only the
# database entry is added. Different bytes under a name the channel already
# has is the one thing this must never do.
for f in "${FILES[@]}"; do
b=$(basename "$f")
grep -qxF "$b" <<<"$listing" || continue
remote_sum=$(rclone hashsum md5 "$DEST/$b" --s3-no-head 2>/dev/null | awk '{print $1}')
local_sum=$(md5sum "$f" | awk '{print $1}')
if [[ -n "$remote_sum" && "$remote_sum" == "$local_sum" ]]; then
print_info "Already published with identical bytes, adding to the database only: $b"
else
print_error "Already published with DIFFERENT bytes, refusing to overwrite: $b"
echo " Bump pkgrel; published filenames are immutable."
exit 1
fi
done
# --- 3. sign ---------------------------------------------------------------
export GNUPGHOME="$WORK/gnupg"; mkdir -m 700 "$GNUPGHOME"
echo "$GPG_PRIVATE_KEY" | gpg --batch --quiet --import
KEY_ID=$(gpg --list-secret-keys --with-colons | awk -F: '$1=="sec"{print $5; exit}')
[[ -n "$KEY_ID" ]] || { print_error "No secret key imported"; exit 1; }
for f in "${FILES[@]}"; do
cp "$f" "$WORK/repo/"
gpg --batch --yes --pinentry-mode loopback --passphrase "$GPG_PASSPHRASE" \
--detach-sign --no-armor --local-user "$KEY_ID" "$WORK/repo/$(basename "$f")"
print_step "signed $(basename "$f")"
done
# --- 4. repo-add (replaces the entry for each pkgname) ---------------------
( cd "$WORK/repo" && repo-add --quiet omarchy.db.tar.zst "${FILES[@]/#*\//}" )
ln -sf omarchy.db.tar.zst "$WORK/repo/omarchy.db"
ln -sf omarchy.files.tar.zst "$WORK/repo/omarchy.files"
print_info "Database now has $(tar -tf "$WORK/repo/omarchy.db.tar.zst" | grep -c '/$') entries"
# --- 5. upload: packages, signatures, database last -----------------------
rclone copy "$WORK/repo" "$DEST" --s3-no-head --ignore-existing --filter '- *.sig' --filter '+ *.pkg.tar.*' --filter '- *'
rclone copy "$WORK/repo" "$DEST" --s3-no-head --ignore-existing --filter '+ *.pkg.tar.*.sig' --filter '- *'
# Re-verify every referenced file is really there before the db goes up.
listing=$(rclone lsf "$DEST/" --s3-no-head)
for f in "${FILES[@]}"; do
b=$(basename "$f")
grep -qxF "$b" <<<"$listing" && grep -qxF "$b.sig" <<<"$listing" || { print_error "Upload incomplete: $b"; exit 1; }
done
rclone copy "$WORK/repo" "$DEST" --s3-no-head --checksum --copy-links --filter '+ omarchy.db*' --filter '+ omarchy.files*' --filter '- *'
print_success "Published ${#FILES[@]} package(s) to $DEST"