A one-package merge took 9 to 15 minutes to publish for about 15 seconds of
signing and upload. The run-wide concurrency group made each merge wait for
every earlier run, builds included (#854 waited 13 minutes behind an aarch64
batch), and the publish job waited about 3 minutes for a builder droplet
even when every package had a PR artifact and nothing needed building.
Build x86_64 trees that have no artifact in the rebuild job, one droplet per
entry, as aarch64 already builds there on arm64 runners: in parallel, with
no secrets, and outside any lock. The publish job now only collects
artifacts, signs and uploads, on ubuntu-latest with the GHCR builder image
(#850), and only it holds the publish group.
A droplet DigitalOcean still reports as "new" never registers a runner, but
the controller counted it as one booting and created no replacement until
MAX_AGE_MINUTES. A publish job sat queued for minutes behind one in mkc1.
Delete droplets still provisioning after MAX_BOOT_MINUTES (10) and leave them
out of the live count, so the same tick creates a replacement.
* Keep builders coming when DigitalOcean sells out a droplet size
ric1 sold out of g5-32vcpu-64gb-50gb, and every create came back 422. curl -f
dropped the reason and set -e ended the tick, so builders only appeared when
capacity happened to free up, and the journal showed nothing but "curl: (22)".
Try each of SIZES in turn, logging DigitalOcean's refusal message, and fail
the tick only when every size is refused. Builders now power off however
start.sh exits, so a failed registration is reaped instead of counting as a
booting runner until MAX_AGE_MINUTES. The controller unit pulls the checkout
before each tick, so merged controller fixes reach the box.
* Create builders in any region that has the size in stock
ric1 sold out of g5-32vcpu-128gb-50gb within minutes of the box switching to
it. Builders need nothing from a particular region, so read DigitalOcean's
size catalog once per tick and try each of SIZES in every region it lists in
stock, REGIONS first if set. A refused pair is dropped for the rest of the
tick.
A merged aarch64 tree without a PR build artifact (expired after 7 days,
or a dispatch) was rebuilt on the x86 droplet under QEMU: omarchy-mac-boot
took ~167 of the 240 minutes. A new job builds it on ubuntu-24.04-arm the
way build-pr.yml does and uploads it under the same label, so the publish
job signs and uploads it on the droplet like a PR artifact. The plan logs
reuse or rebuild for every package; x86_64, signing and the publish
concurrency are unchanged.
The droplet pool is x86, so aarch64 builds ran under QEMU about 30x slower;
omarchy-mac-boot's check() took 2h47m of the 180-minute job limit. Heavy
packages stay on the droplets until a native build of each is shown to fit.
multiarch/qemu-user-static stopped at QEMU 7.2 (January 2023). Under it,
qmake's compiler probe (`g++ -E -v` in toolchain.prf) returns nothing on the
current gcc 16 toolchain, so every qmake package fails on aarch64 with
"failed to parse default include paths from compiler output" (hype, PR #517).
The same PKGBUILD builds under QEMU 10.2.3 and 11.1.
Register through tonistiigi/binfmt at a pinned tag, on both the ephemeral
builder droplets and the rootful-Docker path of setup_qemu, uninstalling any
existing entry first because the tool keeps an older registration in place.
The tag is now the one place that decides what every emulated build runs
under. Flags stay F and C, which rootless sudo inside the builder needs.
Every pull request now builds the package directories it touches on
ephemeral DigitalOcean droplets, and every merge to master publishes the
resulting artifacts into the channels each package belongs to. The
repository host's timers become the fallback rather than the pipeline.
Build (.github/workflows/build-pr.yml)
One job per package per architecture, always against edge. The artifact
is labelled with the package directory's git tree hash. Tooling (bin/,
helpers/, build/) is checked out from the base branch; the PR supplies
only pkgbuilds/, so a PR can change what is built, never how. Builds
run only for trusted authors: collaborators, .github/VOUCHED.td, or a
PR carrying the build-approved label. A single required check, result,
aggregates the matrix.
Publish (.github/workflows/publish.yml, bin/publish-artifact)
One job per merge. It collects the PR artifacts for the merged tree,
builds anything that has none, then walks each channel/architecture
slot once: pull that database, repo-add every package that belongs in
it, upload packages, signatures, then the database. A published
filename is immutable; identical bytes under an existing name only
gain a database entry, different bytes are refused. Fast-ring packages
reach edge, rc and stable in the same run from the same file.
Matrix (bin/build-matrix)
Package x architecture, with the channels the artifact ships to,
decided by package_builds_for_mirror so CI and the host agree.
arch=any packages build once and land in every architecture database.
Builder (build/build.sh, bin/build, build/Dockerfile)
With no local published tree, plan against and resolve from the public
channel over HTTPS. PACKAGER is set to Omarchy. rclone is in the image.
Runners (ci/)
A controller droplet polls GitHub with curl and creates one g5 droplet
per queued job from cloud-init, deleting them when off or over-age.
Builders carry QEMU with credential support for aarch64. Operator SSH
keys come from GitHub. tests/controller.sh and tests/publish-artifact.sh
cover the decisions against fixtures and real makepkg output.
Tests run on pull requests only; branch protection requires result,
self-tests and build-isolation with up-to-date branches.
Ryan Hughes