Since publishing moved to CI on merge, a package whose PKGBUILD never
changes while its source moves was never rebuilt: omarchy-dev and
omarchy-settings-dev followed quattro through "#branch=" and a pkgver()
function, and nothing in this repository changed when quattro did. The
host timers that used to notice are off, so edge fell days behind.
The rule now: no git source without a commit or tag pin
(tests/pinned-sources.sh, run in CI). A package that has to follow a
branch declares a git_branch upstream watch, and the pin moves through
the same PR/build/publish path as every other version bump.
Watch (helpers/upstream-watch.py)
git_branch gains tag_pattern: the newest release tag in the pinned
commit's own history, exposed as {tag}/{version}/{distance}, so a
branch build is versioned <tag>.r<n>.g<sha>, above the release it
follows and below the next one. One blobless clone per branch per
run, shared by every package on it. min_release_age selects the
newest commit older than the window, so a push burst builds once.
Lane (helpers/package-metadata.sh, bin/sync-upstream --lane)
"auto_merge": true moves a package from the reviewed 6-hourly sync
PR to the unattended lane. Packages pinned from the same branch move
together: a failure on one restores the others and fails the group,
so the dev pair can never ship from two quattro commits.
Tracker (.github/workflows/track-branches.yml)
Every two hours: pin, open one PR with a GitHub App token, enable
auto-merge. Branch protection still gates the merge on result,
self-tests and build-isolation. A tip that fails to build stays an
open red PR until the next tick supersedes it. The App is required:
a PR opened with GITHUB_TOKEN has its checks held for approval and
its auto-merge would not fire publish.yml.
The reviewed workflows (sync-upstream, sync-rebuilds) open their PRs
with the same App so their builds start without a maintainer clicking
"Approve workflows to run"; without the App they fall back to
GITHUB_TOKEN and behave as before.
Recipes
The dev pair pins _commit and a real sha256sum, keeps the OMARCHY_SRC
override, and drops pkgver(). Its r-number stays the branch's total
commit count because the published history used it and pacman must
never see the version go down. omasnap-git is new: omacom/omasnap
main, versioned <tag>.r<distance>.g<sha>, provides/conflicts omasnap.
set_pkgbuild_array ended its skip over the old array at the first line
holding a ")", comments included. voxtype-bin annotates its arrays with
"# Quickshell OSD launcher + audio-bridge sidecar (new in v0.7.5)", so
the rewrite stopped there and left the rest of the old array behind a
stray ")". Every scheduled sync since voxtype 1.1.0 failed with
"Rewritten PKGBUILD is not valid shell".
Strip comments before looking for the closing paren, add a self-test
fixture that fails on the old awk, and take the update it was blocking:
voxtype-bin 1.0.1 -> 1.1.0.
Every pull request now builds the package directories it touches on
ephemeral DigitalOcean droplets, and every merge to master publishes the
resulting artifacts into the channels each package belongs to. The
repository host's timers become the fallback rather than the pipeline.
Build (.github/workflows/build-pr.yml)
One job per package per architecture, always against edge. The artifact
is labelled with the package directory's git tree hash. Tooling (bin/,
helpers/, build/) is checked out from the base branch; the PR supplies
only pkgbuilds/, so a PR can change what is built, never how. Builds
run only for trusted authors: collaborators, .github/VOUCHED.td, or a
PR carrying the build-approved label. A single required check, result,
aggregates the matrix.
Publish (.github/workflows/publish.yml, bin/publish-artifact)
One job per merge. It collects the PR artifacts for the merged tree,
builds anything that has none, then walks each channel/architecture
slot once: pull that database, repo-add every package that belongs in
it, upload packages, signatures, then the database. A published
filename is immutable; identical bytes under an existing name only
gain a database entry, different bytes are refused. Fast-ring packages
reach edge, rc and stable in the same run from the same file.
Matrix (bin/build-matrix)
Package x architecture, with the channels the artifact ships to,
decided by package_builds_for_mirror so CI and the host agree.
arch=any packages build once and land in every architecture database.
Builder (build/build.sh, bin/build, build/Dockerfile)
With no local published tree, plan against and resolve from the public
channel over HTTPS. PACKAGER is set to Omarchy. rclone is in the image.
Runners (ci/)
A controller droplet polls GitHub with curl and creates one g5 droplet
per queued job from cloud-init, deleting them when off or over-age.
Builders carry QEMU with credential support for aarch64. Operator SSH
keys come from GitHub. tests/controller.sh and tests/publish-artifact.sh
cover the decisions against fixtures and real makepkg output.
Tests run on pull requests only; branch protection requires result,
self-tests and build-isolation with up-to-date branches.
Build on #306 without replacing its package or updater workaround. Exhaust the component feed before selecting a release and cover quarantine, version ordering, invalid input, and transport failures with offline fixtures.
Incremental advances collide with same-version artifacts that reached the
destination through another lineage — the stable -> rc bootstrap seed and
native rc builds are not byte-identical to edge's builds of the same
version. A collision means the package has not moved in the source since,
so keep the destination's published copy and continue; the hard failure
also aborted before the db rebuild and sync, leaving the advance half-done.
Also stop advancing pinned packages into rc: eligibility deferred to
package_builds_for_mirror, whose OMARCHY_RC_PINS gate is never set during
an advance, so edge's omarchy/omarchy-settings looked movable — and could
have overtaken an in-flight RC pin under a fresh filename.
makepkg always exports CARCH, so PKGBUILDs may branch on it at file
scope. Every place the tooling sourced a PKGBUILD did so without CARCH,
taking the wrong branch or aborting partway, and check-versions turned
the resulting empty pkgver/pkgrel into the version '-', which never
matches a published version and queues an endless rebuild.
package_pkgbuild_var reads one variable the way makepkg would see it,
for the architecture being checked, and reports whether the source
succeeded. check-versions, sync-rebuilds and omarchy-pkgs use it.
check-versions now warns and skips a package whose pkgver or pkgrel is
empty instead of comparing a partial version. A self-test covers a
PKGBUILD that branches on CARCH before assigning its version.
One list, PUBLISHED_ARCHES in helpers/paths.sh (default x86_64,
overridable with OMARCHY_ARCHES), now drives everything the repository
host schedules. check-versions compares PKGBUILDs against each
architecture's channel databases and writes one queue per channel and
architecture; auto-release works through the queues one architecture at
a time, each with its own backoff, so a failing build on one never
blocks the other; advance-channel --arch all re-runs an advance for every
published architecture and omarchy-release uses it for start and ship,
building the pinned pair once per architecture in its rc trigger; the
train observes channels through the reference (first) architecture
instead of a hard-coded x86_64. Queue and backoff files written under
the old per-channel names are treated as x86_64 until consumed.
Two things made an aarch64 builder image impossible to create: the
keyring bootstrap fetched omarchy-keyring from the target architecture's
own channel tree, which does not exist before that architecture has
published anything, and the QEMU probe only knew the x86_64-host,
aarch64-target case. The keyring (arch=any) now always comes from the
x86_64 tree, and the probe compares host and target architectures and
runs a container for the target platform.
clean-repo grouped versions with a regex that only knew any, x86_64 and
i686, so aarch64 packages would never have been pruned.
Schist is a layered image editor with PSD, Affinity and camera raw support,
developed by Infrawrench and packaged by its upstream author. The package
re-wraps the pacman-format payloads Schist's release workflow publishes for
x86_64 and aarch64, so the builder does no compiling, and both assets are
pinned by SHA-256.
Releases are tracked declaratively through the GitHub upstream provider,
which gains a "digests": true mode here: a vendor that publishes no checksum
manifest can have each asset's SHA-256 read from the digest GitHub's release
API reports, so the sync never downloads the artifacts. Exactly one of
"checksums" or "digests" must be set, and the provider enforces that itself
because scheduled runs reach it without the metadata validator.
Fresh releases wait 24 hours before the scheduled sync picks them up, as
mise-bin already does. vulkan-driver is an optional dependency rather than a
hard one: makepkg -s would otherwise satisfy the virtual package with
nvidia-utils in the build container, and Omarchy installs a Vulkan driver per
machine.
Co-authored-by: David Heinemeier Hansson <david@hey.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Three opt-in knobs for bin/build, each defaulting to today's behaviour:
OMARCHY_KEEP_BUILD_WORKSPACE=1 keeps build-output/$MIRROR/$ARCH instead
of wiping it, and build/build.sh now folds any packages already there
into omarchy-build.db even when no database exists yet, so packages
built by an earlier job (or a previous, interrupted run) resolve as
dependencies of what builds next.
OMARCHY_SKIP_BUILDER_IMAGE=1 uses the omarchy-pkg-builder image already
present instead of building it, so a workflow can build the image once
with an external BuildKit cache and fan out over package jobs that all
run the same bytes. A missing image is an error, not a silent rebuild.
OMARCHY_DEFER_RUNTIME_DEPS=true builds the omarchy/omarchy-settings pair
with --nodeps, installing only their makedepends and checkdepends
explicitly. The pair depends on each other and on packages a sharded
pipeline builds in other jobs, so they cannot resolve in isolation; the
assembled set is installed in one verified transaction downstream. The
request is refused for anything but exactly that pair, on the host
before Docker starts and again inside the container.
Also fix make_dir_writable: chown -R can succeed on part of the tree
and fail on files a previous container left behind as another uid, and
the old `|| chmod` fallback only ran when chown failed outright. Always
follow with chmod.
cmd_rc handed maybe_iso the bare train version, so omarchy-iso-release
named every candidate omarchy-X.Y.Z-rc.iso — rc2 uploaded over rc1's
URL. Pass the pinned rcN version instead; paired with omarchy-iso's
f9be60b, the artifact becomes omarchy-4.0.2rc2.iso. The ship path
keeps passing the final version.
pkgs.omarchy.org sits behind Cloudflare, and right after a sync the
plain db URL keeps serving the previous file from the edge cache
(observed: a cache HIT with age 900s+ returning the pre-release db).
That made status report the old version, wait_for_published poll a
frozen file for its full timeout, and would let RC auto-numbering
count from last release's versions. A unique query string per fetch
skips the cached entry.
package_builds_for_mirror gates pinned packages' rc builds on
OMARCHY_RC_PINS, but that check runs inside the container via
build.sh, and docker run only forwarded ARCH/MIRROR/PACKAGES —
so the rc trigger's pinned builds were always skipped as 'not
configured for direct rc builds'.
The rc trigger passes '--package omarchy omarchy-settings', and bin/build
already consumes every name up to the next --option, but bin/release only
took one — the second name fell through to 'Unknown option' and the
release exited before building anything. Parse greedily, like bin/build.
`gh pr list` orders by creation date, so `--limit 30` cut the candidate set by
when PRs were opened, not when they merged. The `sort_by(.mergedAt)` that
followed only reordered whatever survived that cut. A PR opened before the
window but merged inside it — exactly the kind a release branch still needs —
never reached the already-on-branch check at all. #7649 and #7709 were both
missing from v4-0-2's list for this reason.
The window is now bounded by the branch point instead of a count: pull a wide
page and keep what merged after the merge-base's commit date, since anything
merged into the dev branch before the release branch left it is already there
by ancestry. v4-0-2 went from 11 candidates to 31.
Widening it surfaced a second gap. A change re-applied by hand carries neither
a PR number nor a cherry-pick trailer, so already_on_branch could not see it
and offered it again (#6939, applied as 33d7363c). It now also compares the PR
title against the branch's subjects, with any trailing "(#N)" stripped.
The rc channel's Arch base can sit anywhere between stable's snapshot and
edge's, so a package built against stable's libraries is not necessarily
correct for rc. Copying stable's fast-ring artifacts into rc therefore shipped
possibly-mislinked packages to RC testers. Fast-ring packages now build
natively for all three channels, each in its own image against its own base
mirror, and the stable release's replication step is gone.
That required separating 'may be built here' from 'whose version wins'. The
release pair is now marked "pinned": its version is set per release on the rc
branch, so it builds for rc only from that branch's worktree
(OMARCHY_RC_PINS=1, set by omarchy-release rc) — master's shipped pins can
never overwrite an in-flight RC, even though check-versions now discovers rc
work like it does for edge and stable.
The fast-ring replication announced itself before the release that triggered
it, so chat read as though an rc job had run on its own — the reverse of what
happened. The publish report now fires first, and the replication says only
how many packages were kept in parity: the release report immediately above it
already lists them by name.
Reporting to BASECAMP_CHATBOT_URL is a working setup — the second variable
exists only for those who want release traffic in its own chat. setup now
states which chat receives reports instead of warning about the common case,
and the README frames the split as optional rather than expected.
Two failures from the first live run:
notify_basecamp declared 'local BASECAMP_CHATBOT_URL' and then called
release_chatbot_url, whose fallback reads that same global — bash locals are
visible to called functions, so the fallback saw the empty local and every
notification silently went nowhere for anyone with only the legacy variable
set. The local is now named 'url'.
check-versions compared PKGBUILD and published versions with !=, so a checkout
BEHIND the channel queued a rebuild of an older version every cycle: the
builder produced it and promotion refused it, because that exact filename is
already published with different bytes. It now skips (with a warning naming
the package) when an artifact for the PKGBUILD's version already exists in the
channel, whichever direction the versions differ.
The rc service fetched and reset /root/omarchy-pkgs-rc in ExecStartPre, but
that worktree does not exist until the first RC creates the rc branch — so on
a freshly set up host the unit failed every five minutes, forever, and showed
up as a failed unit in the timer report.
It now runs bin/auto-release-rc from the main checkout, which always exists:
nothing queued exits silently, no rc branch is a clean no-op, and a missing
worktree is created on demand before handing off to the worktree's own
auto-release.
Restores the no-change report now that it is clear it cannot fire on an idle
timer tick: releases only run when the version check queued work, so a run
that publishes nothing means check-versions and the builder disagree about
what is out of date. The report names the packages that were queued but never
built, so a recurring disagreement is diagnosable rather than invisible.
A release that published nothing is not news, and at a 5-minute cadence those
messages would bury the ones that matter — the log and bin/repo timers still
show the run happened.
Removing it would have left a start report with no follow-up, so the start
report now carries its own answer: check-versions writes the package names it
queued into the state file instead of touching an empty one, and the release
run reads them. 'A build is running' becomes 'your package is in this build',
which is the question the reports exist to answer.
Only failures were reported, so a push could reach the mirror with no way to
know short of querying the database by hand. Release runs now report:
- start: channel, arch, host, and the commit being built
- published: the packages and versions that went out, duration, channel URL
- no-changes: the run found nothing to build
- promoted: what advance moved between channels, including the rc bootstrap
and fast-ring replication
- failed: unchanged, plus the commit context the other reports carry
Release traffic goes to OMARCHY_RELEASE_CHATBOT_URL, falling back to
BASECAMP_CHATBOT_URL, so build reports stop drowning the repository chat the
sync workflows post to. bin/setup reports which destination is configured.
The published list is captured after the build step because promote moves the
files out of build-output, and is capped at 25 entries so a full rebuild does
not produce an unreadable wall of chat.
A push reaching the mirror should take minutes, not up to six hours. All four
units now fire every 5 minutes, staggered a minute apart. Three guards make
that cadence safe:
- Scheduled runs take the release lock NON-BLOCKING (try_release_lock) and
skip the tick when a build is running. Blocking would stack one stalled
process per tick behind a long build and stampede when it finished. Manual
commands still wait, as an operator expects.
- check-versions takes the lock too, and now owns its git pull (--pull, passed
by the unit) instead of an ExecStartPre: at this cadence an unlocked pull
would swap PKGBUILDs out from under a running build.
- A failed release records .build-failed-<channel> and backs off
exponentially (10m, 20m, 40m … capped at 6h) rather than rebuilding the same
broken tree every 5 minutes. Any new commit clears the backoff, since a push
is the most likely fix.
Idle ticks exit without output so the journal keeps showing the runs that
matter, and bin/repo timers reports backoff state — a paused channel is
otherwise indistinguishable from an idle one.
Wraps systemctl list-timers with the things you actually want when checking on
the build host: per-unit enabled state, last run and whether it succeeded,
which channels have builds queued (state files), whether the release lock is
held by a live process, and any failed units. Units are discovered from
systemd/*.timer so the report cannot drift from what setup installs.
It forwards over ssh like the other host commands, so the build box's timer
state is one command away from a workstation (--local to inspect this machine).
--check warned 'rc worktree would be created' whenever the directory was
absent, implying a plain setup run would create it — but with no rc branch in
existence setup skips it, so the warning described something that would not
happen and asked for action that was not possible. It now reports the branch
state: present, would-create (branch exists), or nothing-to-do (no branch
yet — the first RC cut creates the branch and the build trigger creates the
worktree on demand). Branch detection is read-only, as --check must be.
Eligibility used package_moves_to_channel, which excludes packages built
natively in the destination — right for ongoing edge -> rc advances (a native
build must not be raced under the same filename) but wrong for the bootstrap,
whose entire purpose is rc == stable. omarchy and omarchy-settings were
therefore left out, so a machine switched to the rc channel could not install
or update the release pair until the first RC was cut. The bootstrap now
requires only destination membership; nothing is built in rc yet, so there is
no native artifact to conflict with. The dev pair stays edge-only and
fast-ring replication is unchanged.
The builder stage never declared ARG MIRROR, so the keyring [omarchy] repo
pointed at the channel-less legacy pkgs.omarchy.org/$arch path — it works
only because a stale copy of the old layout still answers there, and it would
miss a keyring rotation. Each image now pulls omarchy-keyring from its own
channel (edge/rc/stable), matching the base mirror it already selects.
update-repo and remove-package switch to the edge x86_64 image: repo-add and
repo-remove compile nothing, and using the channel image would deadlock
bootstrap-rc — the rc image can only build once the rc channel it pulls the
keyring from exists remotely.
With a repository host configured (OMARCHY_REPO_HOST / .repo-host), release,
build, sign, promote, update, clean, advance, bootstrap-rc, remove, sync, and
migrate exec on the host over ssh — same code, run where the published tree
lives, after sourcing the host credentials and a --ff-only pull. --local
forces local execution. list/push/deploy/setup never forward. The host itself
has no .repo-host, so ssh'd-in manual use is unchanged. omarchy-release's
advance now rides the same forwarding (one code path), and --host exports
OMARCHY_REPO_HOST so child bin/repo calls follow it.
This closes the gap where bootstrap-rc ran against a workstation's stale
local tree despite .repo-host being set.
The published-db marker also exists on any workstation that once ran a full
local release, so --host / OMARCHY_REPO_HOST / .repo-host are now checked
before on_repo_host everywhere (triggers, advances, doctor). README documents
the detection, the caveat, and the .repo-host tie-breaker.
Release commands now work from anywhere: on_repo_host (the published database
living in this checkout) routes build triggers, advances, and promotion to
local execution; other machines go over ssh to the configured destination.
The host setting is any ssh destination — root@<ip>, root@<hostname>, or an
~/.ssh/config alias — resolved from --host, OMARCHY_REPO_HOST, then the
one-line .repo-host file. doctor reports which mode applies, and the README
documents the format and precedence. All host connections are plain ssh.
The work/mirror clones were HTTPS end to end, so pushes went through git's
credential-helper config — which breaks the moment a stale absolute gh path
is baked into it (as gh auth setup-git once did with /usr/bin/gh). Reads stay
anonymous HTTPS; pushes now use an SSH push URL (derived from the clone URL,
overridable with OMARCHY_UPSTREAM_PUSH_URL), set idempotently on every run so
existing cached clones self-repair.
Changes reach a release branch as backports — cherry-picks with new SHAs — so
the original quattro merge commit is never an ancestor of a patch branch and
the ancestry filter let already-applied PRs through. Candidates are now also
matched against the branch's own commit messages since it left quattro:
'backport of #N' / squash '(#N)' references and 'cherry picked from commit
<sha>' trailers (which pick -x itself writes). Explicitly named PRs/commits
that are already on the branch are skipped with a note instead of re-picked.
Verified against the live v4-0-2 branch: the five backported PRs it carries
filter out; un-backported ones are still offered.
- ship: no interactive override of the untested-commit guard; the tag targets
the pinned commit the artifacts were built from (never the branch head); a
tagged-but-incomplete train is found and resumed instead of vanishing from
open-train detection; a fully shipped train reports as such
- start: a failed edge→rc advance fails the command loudly (both start and
the advance are idempotent) instead of opening a train against stale rc
- rc trigger: bootstraps the server's rc worktree on first use, so a host set
up before the rc branch existed can run its first RC build
- advance-channel: fast-ring packages are excluded from edge→rc (the stable
build replicated by parity is authoritative for rc — same filename, other
bytes); differing destination bytes abort instead of warn; a package whose
signature copy was interrupted gets its .sig restored on resume
- the release lock now also covers direct promote/update/clean/remove/sync
invocations, not just release/advance/upload-prebuilt
Ryan Hughes