The dropbox watch failed with "no matching releases" on some runs
(e.g. actions run 37236931512). It is not the network: Dropbox's download
redirect sends a share of requests to a newer build the pattern rejects
on purpose. On 2026-10-04, 96 of 100 HEADs ended at
dropbox-lnx.x86_64-272.4.3798 and 4 at 274.3.4801, which is not an x.4.y
stable build. The redirect provider made one probe, so one unlucky
answer failed the whole watch.
It now probes up to five times and keeps the first final URL that
matches, and curl gets --retry 2 as Fetcher.file already has. Live,
100 discovers against Dropbox all found 272.4.3798 in 103 probes. Two
tests cover a rollout answer before the stable one and every probe
missing.
Since publishing moved to CI on merge, a package whose PKGBUILD never
changes while its source moves was never rebuilt: omarchy-dev and
omarchy-settings-dev followed quattro through "#branch=" and a pkgver()
function, and nothing in this repository changed when quattro did. The
host timers that used to notice are off, so edge fell days behind.
The rule now: no git source without a commit or tag pin
(tests/pinned-sources.sh, run in CI). A package that has to follow a
branch declares a git_branch upstream watch, and the pin moves through
the same PR/build/publish path as every other version bump.
Watch (helpers/upstream-watch.py)
git_branch gains tag_pattern: the newest release tag in the pinned
commit's own history, exposed as {tag}/{version}/{distance}, so a
branch build is versioned <tag>.r<n>.g<sha>, above the release it
follows and below the next one. One blobless clone per branch per
run, shared by every package on it. min_release_age selects the
newest commit older than the window, so a push burst builds once.
Lane (helpers/package-metadata.sh, bin/sync-upstream --lane)
"auto_merge": true moves a package from the reviewed 6-hourly sync
PR to the unattended lane. Packages pinned from the same branch move
together: a failure on one restores the others and fails the group,
so the dev pair can never ship from two quattro commits.
Tracker (.github/workflows/track-branches.yml)
Every two hours: pin, open one PR with a GitHub App token, enable
auto-merge. Branch protection still gates the merge on result,
self-tests and build-isolation. A tip that fails to build stays an
open red PR until the next tick supersedes it. The App is required:
a PR opened with GITHUB_TOKEN has its checks held for approval and
its auto-merge would not fire publish.yml.
The reviewed workflows (sync-upstream, sync-rebuilds) open their PRs
with the same App so their builds start without a maintainer clicking
"Approve workflows to run"; without the App they fall back to
GITHUB_TOKEN and behave as before.
Recipes
The dev pair pins _commit and a real sha256sum, keeps the OMARCHY_SRC
override, and drops pkgver(). Its r-number stays the branch's total
commit count because the published history used it and pacman must
never see the version go down. omasnap-git is new: omacom/omasnap
main, versioned <tag>.r<distance>.g<sha>, provides/conflicts omasnap.
sync-rebuilds followed PUBLISHED_ARCHES (x86_64 by default), so the scheduled
run skipped every aarch64-only package, and it read aarch64 triggers from Arch
Linux ARM even when this repository carries them. It now follows the
architectures a merge builds (CI_ARCHES, shared with build-matrix), and reads a
carried trigger from its recipe once edge publishes that version; until then
its dependents wait. Published databases are verified before use and loaded
once per run.
`restore=$(shopt -p nullglob)` exits 1 when nullglob is off, which it is
in the workflow shell. Under the `bash -e` that GitHub runs steps with,
that ended "Pack artifact" before tar ran: every job on #512 and #550
built fine and then failed with nothing in the log but the command.
The self-test never saw it because `pack_packages ... || fail` suppresses
errexit. Enumerate package files with a loop instead of toggling shell
options, and add a test that calls both helpers under `bash -e` exactly
as the workflows do; it fails against the old helper.
actions/upload-artifact rejects any path containing ':', and makepkg names
a package with an epoch `name-1:ver-rel-arch.pkg.tar.zst`. Every PR that
built such a package (cursor-cli in the sync PRs, omasnap once it gained an
epoch) failed at "Upload artifact" after a successful build, and publish
then rebuilt from scratch on merge.
The files now ride inside packages.tar for the artifact hop and come back
out with makepkg's names untouched: pacman clients and bin/publish-artifact
both require the filename to match PKGINFO, and the channels already carry
these names. publish.yml still accepts bare pre-packing artifacts until the
7-day retention drains them.
helpers/artifact-helpers.sh holds both halves; tests/artifact-helpers.sh
covers the round trip and runs with the other self-tests.
multiarch/qemu-user-static stopped at QEMU 7.2 (January 2023). Under it,
qmake's compiler probe (`g++ -E -v` in toolchain.prf) returns nothing on the
current gcc 16 toolchain, so every qmake package fails on aarch64 with
"failed to parse default include paths from compiler output" (hype, PR #517).
The same PKGBUILD builds under QEMU 10.2.3 and 11.1.
Register through tonistiigi/binfmt at a pinned tag, on both the ephemeral
builder droplets and the rootful-Docker path of setup_qemu, uninstalling any
existing entry first because the tool keeps an older registration in place.
The tag is now the one place that decides what every emulated build runs
under. Flags stay F and C, which rootless sudo inside the builder needs.
Incremental advances collide with same-version artifacts that reached the
destination through another lineage — the stable -> rc bootstrap seed and
native rc builds are not byte-identical to edge's builds of the same
version. A collision means the package has not moved in the source since,
so keep the destination's published copy and continue; the hard failure
also aborted before the db rebuild and sync, leaving the advance half-done.
Also stop advancing pinned packages into rc: eligibility deferred to
package_builds_for_mirror, whose OMARCHY_RC_PINS gate is never set during
an advance, so edge's omarchy/omarchy-settings looked movable — and could
have overtaken an in-flight RC pin under a fresh filename.
makepkg always exports CARCH, so PKGBUILDs may branch on it at file
scope. Every place the tooling sourced a PKGBUILD did so without CARCH,
taking the wrong branch or aborting partway, and check-versions turned
the resulting empty pkgver/pkgrel into the version '-', which never
matches a published version and queues an endless rebuild.
package_pkgbuild_var reads one variable the way makepkg would see it,
for the architecture being checked, and reports whether the source
succeeded. check-versions, sync-rebuilds and omarchy-pkgs use it.
check-versions now warns and skips a package whose pkgver or pkgrel is
empty instead of comparing a partial version. A self-test covers a
PKGBUILD that branches on CARCH before assigning its version.
One list, PUBLISHED_ARCHES in helpers/paths.sh (default x86_64,
overridable with OMARCHY_ARCHES), now drives everything the repository
host schedules. check-versions compares PKGBUILDs against each
architecture's channel databases and writes one queue per channel and
architecture; auto-release works through the queues one architecture at
a time, each with its own backoff, so a failing build on one never
blocks the other; advance-channel --arch all re-runs an advance for every
published architecture and omarchy-release uses it for start and ship,
building the pinned pair once per architecture in its rc trigger; the
train observes channels through the reference (first) architecture
instead of a hard-coded x86_64. Queue and backoff files written under
the old per-channel names are treated as x86_64 until consumed.
Two things made an aarch64 builder image impossible to create: the
keyring bootstrap fetched omarchy-keyring from the target architecture's
own channel tree, which does not exist before that architecture has
published anything, and the QEMU probe only knew the x86_64-host,
aarch64-target case. The keyring (arch=any) now always comes from the
x86_64 tree, and the probe compares host and target architectures and
runs a container for the target platform.
clean-repo grouped versions with a regex that only knew any, x86_64 and
i686, so aarch64 packages would never have been pruned.
Schist is a layered image editor with PSD, Affinity and camera raw support,
developed by Infrawrench and packaged by its upstream author. The package
re-wraps the pacman-format payloads Schist's release workflow publishes for
x86_64 and aarch64, so the builder does no compiling, and both assets are
pinned by SHA-256.
Releases are tracked declaratively through the GitHub upstream provider,
which gains a "digests": true mode here: a vendor that publishes no checksum
manifest can have each asset's SHA-256 read from the digest GitHub's release
API reports, so the sync never downloads the artifacts. Exactly one of
"checksums" or "digests" must be set, and the provider enforces that itself
because scheduled runs reach it without the metadata validator.
Fresh releases wait 24 hours before the scheduled sync picks them up, as
mise-bin already does. vulkan-driver is an optional dependency rather than a
hard one: makepkg -s would otherwise satisfy the virtual package with
nvidia-utils in the build container, and Omarchy installs a Vulkan driver per
machine.
Co-authored-by: David Heinemeier Hansson <david@hey.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Three opt-in knobs for bin/build, each defaulting to today's behaviour:
OMARCHY_KEEP_BUILD_WORKSPACE=1 keeps build-output/$MIRROR/$ARCH instead
of wiping it, and build/build.sh now folds any packages already there
into omarchy-build.db even when no database exists yet, so packages
built by an earlier job (or a previous, interrupted run) resolve as
dependencies of what builds next.
OMARCHY_SKIP_BUILDER_IMAGE=1 uses the omarchy-pkg-builder image already
present instead of building it, so a workflow can build the image once
with an external BuildKit cache and fan out over package jobs that all
run the same bytes. A missing image is an error, not a silent rebuild.
OMARCHY_DEFER_RUNTIME_DEPS=true builds the omarchy/omarchy-settings pair
with --nodeps, installing only their makedepends and checkdepends
explicitly. The pair depends on each other and on packages a sharded
pipeline builds in other jobs, so they cannot resolve in isolation; the
assembled set is installed in one verified transaction downstream. The
request is refused for anything but exactly that pair, on the host
before Docker starts and again inside the container.
Also fix make_dir_writable: chown -R can succeed on part of the tree
and fail on files a previous container left behind as another uid, and
the old `|| chmod` fallback only ran when chown failed outright. Always
follow with chmod.
The rc channel's Arch base can sit anywhere between stable's snapshot and
edge's, so a package built against stable's libraries is not necessarily
correct for rc. Copying stable's fast-ring artifacts into rc therefore shipped
possibly-mislinked packages to RC testers. Fast-ring packages now build
natively for all three channels, each in its own image against its own base
mirror, and the stable release's replication step is gone.
That required separating 'may be built here' from 'whose version wins'. The
release pair is now marked "pinned": its version is set per release on the rc
branch, so it builds for rc only from that branch's worktree
(OMARCHY_RC_PINS=1, set by omarchy-release rc) — master's shipped pins can
never overwrite an in-flight RC, even though check-versions now discovers rc
work like it does for edge and stable.
Two failures from the first live run:
notify_basecamp declared 'local BASECAMP_CHATBOT_URL' and then called
release_chatbot_url, whose fallback reads that same global — bash locals are
visible to called functions, so the fallback saw the empty local and every
notification silently went nowhere for anyone with only the legacy variable
set. The local is now named 'url'.
check-versions compared PKGBUILD and published versions with !=, so a checkout
BEHIND the channel queued a rebuild of an older version every cycle: the
builder produced it and promotion refused it, because that exact filename is
already published with different bytes. It now skips (with a warning naming
the package) when an artifact for the PKGBUILD's version already exists in the
channel, whichever direction the versions differ.
Only failures were reported, so a push could reach the mirror with no way to
know short of querying the database by hand. Release runs now report:
- start: channel, arch, host, and the commit being built
- published: the packages and versions that went out, duration, channel URL
- no-changes: the run found nothing to build
- promoted: what advance moved between channels, including the rc bootstrap
and fast-ring replication
- failed: unchanged, plus the commit context the other reports carry
Release traffic goes to OMARCHY_RELEASE_CHATBOT_URL, falling back to
BASECAMP_CHATBOT_URL, so build reports stop drowning the repository chat the
sync workflows post to. bin/setup reports which destination is configured.
The published list is captured after the build step because promote moves the
files out of build-output, and is capped at 25 entries so a full rebuild does
not produce an unreadable wall of chat.
A push reaching the mirror should take minutes, not up to six hours. All four
units now fire every 5 minutes, staggered a minute apart. Three guards make
that cadence safe:
- Scheduled runs take the release lock NON-BLOCKING (try_release_lock) and
skip the tick when a build is running. Blocking would stack one stalled
process per tick behind a long build and stampede when it finished. Manual
commands still wait, as an operator expects.
- check-versions takes the lock too, and now owns its git pull (--pull, passed
by the unit) instead of an ExecStartPre: at this cadence an unlocked pull
would swap PKGBUILDs out from under a running build.
- A failed release records .build-failed-<channel> and backs off
exponentially (10m, 20m, 40m … capped at 6h) rather than rebuilding the same
broken tree every 5 minutes. Any new commit clears the backoff, since a push
is the most likely fix.
Idle ticks exit without output so the journal keeps showing the runs that
matter, and bin/repo timers reports backoff state — a paused channel is
otherwise indistinguishable from an idle one.
bin/repo advance --from/--to moves packages forward through the pipeline
(edge → rc → stable), driven by the source channel's database rather than the
raw directory, copying packages AND their detached signatures (fixing the old
migrate bug that left promoted packages unverifiable), refusing to rewrite any
published filename, and requiring a .sig for everything it moves. stable → rc
is allowed only as --fast-ring parity replication or the one-time
--bootstrap seed (bin/repo bootstrap-rc). 'migrate' stays as a deprecated
alias for the transition.
helpers/lock-helpers.sh adds a host-wide flock shared by bin/release,
advance-channel, and upload-prebuilt (reentrant via OMARCHY_RELEASE_LOCK_HELD)
so timers and operators serialize instead of interleaving partial publishes.
bin/release gains a stable-only step 7: replicate fast-ring artifacts to rc so
rc and stable stay in parity between release trains (skipped until rc is
bootstrapped).
A package's .omarchy/package.json may now pin where it lives with
"channels": [...]. With the key present the package builds in each listed
channel except stable (stable is only fed by promotion); without it, today's
defaults hold (build for edge; fast-ring also builds stable directly).
package_moves_to_channel() is the advance/promote eligibility rule: a member
of the destination channel that is not built there natively.
The release pair (omarchy, omarchy-settings) is edge+rc+stable — edge stays
during the client-migration overlap window and drops later. The dev pair is
pinned to edge only.
- helpers/paths.sh: validate_mirror/require_valid_mirror for the edge|rc|stable
set, and REPO_ROOT (OMARCHY_REPO_ROOT override) so a secondary checkout like
the rc branch worktree publishes into the same channel tree as the primary
- validate --mirror everywhere it previously accepted any string (sync-repo,
promote-build, update-repo, clean-repo, remove-package) and widen the
edge|stable checks in build, deploy, push-build, auto-release
- build/Dockerfile: rc builds compile against rc-mirror.omarchy.org
GNU date accepts relative expressions like '2 days ago', which would let a
buggy hook fabricate a release age; the backstop now insists on an ISO 8601
timestamp before date parses it. The provider header now states, rather than
contradicts, the code's behavior for a feed with no stable releases: that is
a loud failure by design, while quarantined releases report no update.
An empty min_release_age string now maps to unparseable rather than absent,
so "min_release_age": "" fails validation instead of silently running
with a zero-second quarantine. The end-to-end fixtures extend the
checked-in pkgver (.90/.91) so the test keeps working at any future mise
version. A Tests workflow runs bin/sync-upstream self-test and
bin/omarchy-pkgs self-test on every PR in the Arch container, making the
proof machine-checked instead of author-supplied. The README package
metadata field list documents upstream and min_release_age.
The self-test now runs sync_package over the checked-in mise-bin package --
its real metadata and PKGBUILD, the full selection/validation/backstop/
rewrite/read-back path -- with only the two network fetches replaced by
mise-shaped fixtures, asserting the final PKGBUILD holds the quarantine-
cleared version, pkgrel 1, and both architecture checksums.
Review fixes: the release-row builder uses "" fallbacks instead of empty
so a malformed row cannot shift columns past the per-field checks, and
provider discovery now keys on the presence of an upstream declaration
rather than a well-formed one, with sync_package failing loudly on a
declaration it cannot use -- a malformed manifest can no longer silently
drop a package out of scheduled synchronization.
bin/sync-upstream self-test swaps the two network fetches in
helpers/upstream-github.sh for fixture readers and runs the production code
paths: fallback past a quarantined release, draft/prerelease filtering, the
deliberate bypass, unchanged-version and all-quarantined no-update paths,
unusable tags/timestamps and missing checksums failing the sync, {tag} and
{pkgver} asset templates with ./ and * manifest prefixes across both
architectures, the min_release_age backstop verdicts (now a testable
release_age_status function), the duration parser, and manifest validation.
Also fixes from the review: the duration parser forces base-10 arithmetic
(leading zeros no longer parse as octal) and bounds values to nine digits so
no suffix can overflow; jq // treating false as absent can no longer let
"min_release_age": false or "upstream": false slip through as unset; the
release feed page grew to the API maximum of 100 with the bounded search
documented; and the README package-metadata section documents the upstream
block, min_release_age, the bypass, and provider-versus-hook exclusivity.
After the quarantine moved into the manifest, all mise-bin's hook still knew
was data: the repository, the checksum manifest name, and the asset filename
patterns. That now lives in .omarchy/package.json as an upstream block --
"upstream": {
"github": "jdx/mise",
"checksums": "SHASUMS256.txt",
"assets": { "x86_64": "mise-{tag}-linux-x64.tar.xz", ... }
}
-- handled by helpers/upstream-github.sh inside bin/sync-upstream. The
provider walks the release feed (drafts/prereleases excluded), honors
min_release_age and BYPASS_MIN_RELEASE_AGE during selection, reports
published_at so the framework backstop still applies, fails closed on any
unreadable tag or timestamp, and skips the checksum fetch when the newest
qualifying release is already checked in.
upstream.sh remains the escape hatch for feeds that fit no convention
(openai-codex-desktop's Debian index, tmog's version.txt, t3code's
electron-builder manifest); declaring both is an error.
Move the hold from a mise-only hardcode to min_release_age in
.omarchy/package.json ("24h", "2d", or bare seconds), alongside source and
release_ring where package policy already lives. bin/sync-upstream exports
the window to every hook as MIN_RELEASE_AGE_SECONDS so a hook that can walk
its release feed selects the newest release that has cleared it, and
enforces it as a backstop: with a policy set, the hook must report
published_at, and a release younger than the window is treated as no
update. A hook that cannot prove the age fails the sync rather than
shipping unverified. BYPASS_MIN_RELEASE_AGE=1 replaces the package-specific
bypass for deliberate emergency updates; scheduled automation never sets it.
The mise hook keeps its release-list walk but reads the window from the
environment and reports published_at; the other upstream hooks are
untouched and unaffected until they opt in.
A package that links Qt private API has to be rebuilt whenever qt6-base moves, because Qt_6_PRIVATE_API symbols are not covered by the soname and pacman upgrades Qt out from under the installed binary while the dependency stays unversioned. Nothing here noticed. Both version gates ask whether the package's own source moved, and for a VCS package pinned to a commit that answer stays no through every Qt release.
Unlocking the build gate would not have been enough on its own. A rebuild that reuses the published version string produces a package pacman never offers anyone, so the trigger has to edit git and bump pkgrel, which is why it sits beside sync-aur and sync-upstream rather than inside check-versions or the builder. Once pkgrel moves, both existing gates already do the right thing untouched.
Packages opt in with rebuild_on in .omarchy/package.json. bin/sync-rebuilds records what each was last bumped for in rebuilt_against and compares that to core, extra and multilib, ignoring testing and kde-unstable because those are not what the builder links against. A package with no record yet is only recorded, never bumped: what its published build linked against is not knowable from here, so the first run establishes the baseline. For an AUR-synced package the bump is written as the dotted Omarchy pkgrel suffix in the metadata as well, since the next sync replaces the PKGBUILD wholesale and would otherwise drop it.
🤖 Generated by Opus 5 in Claude Code.
Some vendors publish a release feed of their own that is faster and more
precise than anyone's packaging of it. A package opts in with an
.omarchy/upstream.sh hook that reports the newest release as JSON, and the
driver rewrites pkgver, the checksum arrays the hook names, and pkgrel.
Writes are guarded on both ends: every assignment the update will touch is
verified to exist before anything is written, so a hook naming an array the
PKGBUILD lacks fails with the file untouched rather than half rewritten; and
pkgver is held to pacman's character set, because it lands in a file makepkg
sources as shell.
Ordering is vercmp's, not sort -V's -- they disagree about whether 1.0a
precedes 1.0, and pacman is what decides if a published package is an upgrade.
That is also why the workflow runs in an Arch container rather than straight on
the runner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bin/build asks the local repository database which packages are already built.
A build machine has no such database, so every package looks out of date: an
unscoped 'bin/repo deploy' on this laptop would have built all 108 packages and
published them. Verified with a dry run.
deploy now refuses to run unscoped when that database is absent, and push
refuses the same combination under --yes, where nobody would see the list it
prints before publishing. Both are allowed on the repository host, which has
the database that makes the comparison meaningful.
Also states the split in the README: build, push and deploy are the three
commands that may run off the repository host; everything else works on the
published tree directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The warning dated from when .build-host was the release trigger's own setting
and push was a second consumer of it. One setting now names one machine for all
three commands, so there is nothing to keep separate -- and the advice was
wrong regardless: OMARCHY_REPO_HOST is read by the same resolver, so it arms
the trigger exactly as the file does. Only --host is per-invocation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OMARCHY_BUILD_HOST and .build-host were carried as fallbacks through the
rename. Nothing in this checkout ever set either one, so they were a second
name for a setting that has only one real name.
Resolution is now --host, OMARCHY_REPO_HOST, .repo-host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Builds now happen wherever the operator likes, so naming the destination after
building described the old arrangement rather than the current one. What the
push and deploy commands reach is the machine that serves pkgs.omarchy.org and
holds the signing key: the repository host. It also runs the scheduled builds,
which is why the trigger in omarchy-pkgs release points at the same place.
Resolution moves into helpers/host-helpers.sh, which all three commands now
share instead of repeating: --host, then OMARCHY_REPO_HOST, then .repo-host.
OMARCHY_BUILD_HOST and .build-host keep working as fallbacks, so existing
environments and checkouts are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Emir Beganovic