The planner took 'git diff base.sha head.sha', a two-dot diff between
the current master tip and the PR head. For a PR that branched before
later merges, that counts every package master has changed since, so
the PR plans those too and builds its own stale copies of them: wasted
builds, and 'already up to date' Pack failures that turn the PR red for
packages it never touched. #397 (lazyjournal) planned 22 packages for a
one-package change. The checkout is shallow, so ask GitHub for the PR's
files, which are listed against the merge base.
'git status --short | head' under set -o pipefail: once the PR's
pkgbuilds/ differs from base in more than ten files, head exits, git
takes SIGPIPE and the step fails with 141 before anything builds. Every
PR branched far enough behind master hit it (omadev #423, llmman-bin
#428 and others on 2026-09-28). sed -n '1,10p' prints the same preview
and drains the stream.
Moving the whole module broke the rest of it: Writer::create needs a
runtime directory on a different filesystem from the copy, and with the
test root on /dev/shm none qualifies. Only
a_garbage_stream_falls_back_with_the_tree_untouched opens its manifest
in its own test dir, which the builder's /tmp refuses (EOPNOTSUPP).
0.3.5's copymanifest::tests::a_garbage_stream_falls_back_... opens its
anonymous (O_TMPFILE) manifest directly in its /tmp test dir, which the
x86_64 builder's /tmp refuses with EOPNOTSUPP. The sibling tests pass
because Writer::create falls back across directories. Same reason the
other filesystem_tests already run on /dev/shm; none of these spawn.
The upstream hook refused v0.3.5 as missing a required security fix. It
was not missing: copy_file_at now opens through
open_if_regular_with_meta(src.at, O_NOFOLLOW), which open_if_regular
wraps, and the gate's regex wanted '(' right after open_if_regular.
The gate dates from 0.1.x, when Omarchy carried four upstream security
patches and needed releases to prove they had absorbed them. Every
release since 0.1.5 has, and matching literal source lines has since
caught only renames (#488 and this one), never a regression. Keep the
real checks -- SHASUMS256.txt match, tarball root, minimum release age
-- and drop the greps.
Update written by bin/sync-upstream; the checksum matches upstream's
SHASUMS256.txt.
The first pins (quattro ee8ebf6, omasnap eb22bfe) date from 2026-09-21.
Edge has since published omarchy-dev from quattro 7b336b1 through a
manual rebuild, so merging the old pin would ship older code under a
higher version. Moved with the tracker's own code:
bin/sync-upstream --lane auto-merge (BYPASS_MIN_RELEASE_AGE=1)
Since publishing moved to CI on merge, a package whose PKGBUILD never
changes while its source moves was never rebuilt: omarchy-dev and
omarchy-settings-dev followed quattro through "#branch=" and a pkgver()
function, and nothing in this repository changed when quattro did. The
host timers that used to notice are off, so edge fell days behind.
The rule now: no git source without a commit or tag pin
(tests/pinned-sources.sh, run in CI). A package that has to follow a
branch declares a git_branch upstream watch, and the pin moves through
the same PR/build/publish path as every other version bump.
Watch (helpers/upstream-watch.py)
git_branch gains tag_pattern: the newest release tag in the pinned
commit's own history, exposed as {tag}/{version}/{distance}, so a
branch build is versioned <tag>.r<n>.g<sha>, above the release it
follows and below the next one. One blobless clone per branch per
run, shared by every package on it. min_release_age selects the
newest commit older than the window, so a push burst builds once.
Lane (helpers/package-metadata.sh, bin/sync-upstream --lane)
"auto_merge": true moves a package from the reviewed 6-hourly sync
PR to the unattended lane. Packages pinned from the same branch move
together: a failure on one restores the others and fails the group,
so the dev pair can never ship from two quattro commits.
Tracker (.github/workflows/track-branches.yml)
Every two hours: pin, open one PR with a GitHub App token, enable
auto-merge. Branch protection still gates the merge on result,
self-tests and build-isolation. A tip that fails to build stays an
open red PR until the next tick supersedes it. The App is required:
a PR opened with GITHUB_TOKEN has its checks held for approval and
its auto-merge would not fire publish.yml.
The reviewed workflows (sync-upstream, sync-rebuilds) open their PRs
with the same App so their builds start without a maintainer clicking
"Approve workflows to run"; without the App they fall back to
GITHUB_TOKEN and behave as before.
Recipes
The dev pair pins _commit and a real sha256sum, keeps the OMARCHY_SRC
override, and drops pkgver(). Its r-number stays the branch's total
commit count because the published history used it and pacman must
never see the version go down. omasnap-git is new: omacom/omasnap
main, versioned <tag>.r<distance>.g<sha>, provides/conflicts omasnap.
With the search race skipped, the aarch64 main suite ran to the end:
2339 passed, 2 failed. a_missing_working_directory_... expects spawn to
fail on a missing cwd, which posix_spawn cannot report under QEMU
user-mode. in_place_retirement_... identifies its process by
/proc/<pid>/exe, which names /usr/bin/qemu-aarch64 there, not sleep.
strata: the example actions' image test needs ImageMagick with WebP
(checkdepends), the checksum example dies printing a non-UTF-8 name
under the builder's en_US.UTF-8 locale (upstream bug, skipped), and on
aarch64 a search refresh test loses a race to the index worker.
flea: on aarch64 the 1900-deep dirsize walk hits its 2 s deadline, and a
process-group cancel test loses its 5 s race under emulation.
owe: on aarch64 the transition test cannot render enough blended frames
under emulation; run every other test there.
Versions and checksums as in sync PR #660.
Three things kept the upstream sync PR (#589) from ever finishing a build:
Scoped dispatches wiped the shared PR. A workflow_dispatch with `packages`
regenerates only those packages from master, and pushing that to
auto/sync-upstream replaced 38 pending updates with one. Scoped runs now
push to their own auto/sync-{upstream,rebuilds}-<packages> branch and PR;
scheduled runs keep the shared branch.
build-approved stopped working after the first bot push. A GITHUB_TOKEN
push creates pull_request runs held for approval but no pull_request_target
run, so approve-pr.yml never saw it: its last run on the branch was the
label itself (2026-09-25T19:26), and each of the next four syncs sat at
action_required. The sync workflows now release the held runs for the
commit they just pushed, from a separate job holding actions: write, and
only for their own bot-authored, same-repo PR while build-approved is on
it.
Each approved push cancelled the in-flight build. Approving the 21:43
sync's build cancelled the label-triggered one still queued on strata and
schist-bin. On auto/sync-* branches a new build now waits for the running
one instead, then reuses its artifacts. The approval script no longer
waits for a lone approved build to start before releasing tests, which a
queued build would have turned into a timeout.
TMOG 1.0.0 moved the site under /rtm/ and publishes each Linux tarball
under a versioned name with a .sha256 sidecar. The versionless
/downloads/TMOG-Task-Manager-Linux-x86_64.tar.gz path the hook fetched
now returns 404, so every upstream sync since 1.0.0 has failed.
Point source=() at the versioned tarball and have the hook read the
version from /rtm/version.txt and the checksum from the sidecar, which
drops the 8 MB download and the directory-name check the mutable URL
needed.
set_pkgbuild_array ended its skip over the old array at the first line
holding a ")", comments included. voxtype-bin annotates its arrays with
"# Quickshell OSD launcher + audio-bridge sidecar (new in v0.7.5)", so
the rewrite stopped there and left the rest of the old array behind a
stray ")". Every scheduled sync since voxtype 1.1.0 failed with
"Rewritten PKGBUILD is not valid shell".
Strip comments before looking for the closing paren, add a self-test
fixture that fails on the old awk, and take the update it was blocking:
voxtype-bin 1.0.1 -> 1.1.0.
Both upstreams set fat LTO in their release profile (strata also
codegen-units = 1), which compiles the final crate on one thread. That
is a reasonable trade for the binary users run, and build() keeps it.
check() then compiled the same crate a second time under the same
profile for the test harness, and threw it away. On the aarch64 builds,
which run under QEMU, those two serial compiles were 106 of strata's
118 minutes.
check() now builds the harness with thin LTO and 16 codegen units in
its own target directory. The shipped binary is unchanged. Measured on
the x86_64 builder for strata's test binary: 708 s serial, 223 s with
this profile. Local run: shipped strata still 7m30s from target/, tests
3m43s from target-check/, 1998 passed.
ttfx has the same upstream profile but no check(), so nothing to do.
flea 0.3.0 never published: its shelfundo suite (new in 0.3.0) failed check() on the 2026-09-18 publish build, and 0.3.1 fails the same way on the sync PR. undo_refuses_to_walk_a_stranger_back deletes a file, creates another under the same name, and expects undo to tell them apart by (dev, ino, kind). ext4 hands the freed inode number straight back to the next create, so on the droplet's /tmp the stranger is identical and undo walks it back into a directory that does not exist. tmpfs allocates inode numbers from a counter and never reuses one. The suite joins the filesystem tests that already run with TMPDIR on /dev/shm.
On aarch64 three more tests fail: two in backend::child and one in menu_registry expect spawning a missing program to be reported as not started. Under QEMU user-mode emulation glibc's posix_spawn cannot observe the child's failed execve; the spawn succeeds with exit 127. Skipped on aarch64 only.
0.19.0 adds two restart_waiter tests that run `sh -c 'while kill -0 "$1"; do sleep; done'` with the pid u32::MAX. Arch's sh is bash 5.3, whose kill builtin in POSIX mode prints "not a pid or valid job spec" for that number but exits 0, so the loop never ends; every droplet build of 0.19.0 sat there until the 180-minute job timeout. Upstream CI runs on Ubuntu, where sh is dash and rejects the number.
It also adds ownership_probe_errors_disable_in_place_updates, which points the pacman path at a directory and expects Command::output() to fail. aarch64 builds run under QEMU user-mode emulation, where glibc's posix_spawn cannot observe the child's failed execve; the spawn succeeds with exit 127 and the assertion fails. Skipped on aarch64 only; it passes natively.
Both skips carry comments; the sync bot rewrites only version fields, so they survive future syncs.
Every push to a PR rebuilt every package the PR touches, on every
architecture, even when only one of them changed. The daily sync PR
carries around thirty package/arch pairs; fixing one package meant
rebuilding all of them, and an aarch64 build under QEMU takes up to an
hour. Nine runs of that PR cost about 36 droplet-hours in two days.
The planner now asks the artifact store for <pkg>-<arch>-<tree hash>
before adding an entry to the matrix and drops entries that already
have one. That is the same lookup publish.yml makes on merge, so a
reused entry publishes exactly the file it would have anyway. Dry run
against the current sync PR: 27 of 31 entries reused, 4 built.
Pack and Upload no longer run with always(): only a successful build
uploads, so an artifact's existence means that tree built.
workflow_dispatch always builds; it is an explicit request.
Upstream moved to omacom/omasnap and cut 1.21.0 from the same line as 1.20.1, so the version simply increases and no epoch is needed.
check() flushes dirty pages before the smoke suite. The suite fsyncs its working documents under /tmp; on the CI droplets the build leaves about a gigabyte of dirty pages, and the flush that starts a few seconds into the suite made those fsyncs stall past the suite's 5-second settle windows on the first run after every build.
`restore=$(shopt -p nullglob)` exits 1 when nullglob is off, which it is
in the workflow shell. Under the `bash -e` that GitHub runs steps with,
that ended "Pack artifact" before tar ran: every job on #512 and #550
built fine and then failed with nothing in the log but the command.
The self-test never saw it because `pack_packages ... || fail` suppresses
errexit. Enumerate package files with a loop instead of toggling shell
options, and add a test that calls both helpers under `bash -e` exactly
as the workflows do; it fails against the old helper.
github.event.pull_request.base.sha is a snapshot taken when the PR was
last pushed, not the current tip of the base branch. A reopened or rerun
PR therefore builds with whatever master looked like at its last push,
and a tooling fix that landed on master since then never reaches it:
the daily sync PR reopened after #553 merged checked out a base without
helpers/artifact-helpers.sh and failed at "Pack artifact".
Check out base.ref instead. The plan's diff and the empty-PR check still
compare base.sha to head.sha, so the list of changed packages is
unaffected; only the tooling that runs on the droplet moves to the tip.
actions/upload-artifact rejects any path containing ':', and makepkg names
a package with an epoch `name-1:ver-rel-arch.pkg.tar.zst`. Every PR that
built such a package (cursor-cli in the sync PRs, omasnap once it gained an
epoch) failed at "Upload artifact" after a successful build, and publish
then rebuilt from scratch on merge.
The files now ride inside packages.tar for the artifact hop and come back
out with makepkg's names untouched: pacman clients and bin/publish-artifact
both require the filename to match PKGINFO, and the channels already carry
these names. publish.yml still accepts bare pre-packing artifacts until the
7-day retention drains them.
helpers/artifact-helpers.sh holds both halves; tests/artifact-helpers.sh
covers the round trip and runs with the other self-tests.
The package was added pointing at v0.1.0 with a placeholder checksum,
but that tag was never cut; upstream went straight to v1.0.0. The
package has never built anywhere. Point at the real tag and fill the
digest. The upstream watch keeps it current from here.
A workflow_dispatch runs from master's head. That commit's PR merged
something unrelated, so looking the PR up by commit attached a failed
elsewhen report to #515, whose merge had nothing to do with elsewhen.
Dispatch runs now go to the log only. The log append also moves ahead
of the PR comment so the record exists by the time anyone follows the
comment to it.
Ryan Hughes