The planner took 'git diff base.sha head.sha', a two-dot diff between
the current master tip and the PR head. For a PR that branched before
later merges, that counts every package master has changed since, so
the PR plans those too and builds its own stale copies of them: wasted
builds, and 'already up to date' Pack failures that turn the PR red for
packages it never touched. #397 (lazyjournal) planned 22 packages for a
one-package change. The checkout is shallow, so ask GitHub for the PR's
files, which are listed against the merge base.
'git status --short | head' under set -o pipefail: once the PR's
pkgbuilds/ differs from base in more than ten files, head exits, git
takes SIGPIPE and the step fails with 141 before anything builds. Every
PR branched far enough behind master hit it (omadev #423, llmman-bin
#428 and others on 2026-09-28). sed -n '1,10p' prints the same preview
and drains the stream.
A merged aarch64 tree without a PR build artifact (expired after 7 days,
or a dispatch) was rebuilt on the x86 droplet under QEMU: omarchy-mac-boot
took ~167 of the 240 minutes. A new job builds it on ubuntu-24.04-arm the
way build-pr.yml does and uploads it under the same label, so the publish
job signs and uploads it on the droplet like a PR artifact. The plan logs
reuse or rebuild for every package; x86_64, signing and the publish
concurrency are unchanged.
Since publishing moved to CI on merge, a package whose PKGBUILD never
changes while its source moves was never rebuilt: omarchy-dev and
omarchy-settings-dev followed quattro through "#branch=" and a pkgver()
function, and nothing in this repository changed when quattro did. The
host timers that used to notice are off, so edge fell days behind.
The rule now: no git source without a commit or tag pin
(tests/pinned-sources.sh, run in CI). A package that has to follow a
branch declares a git_branch upstream watch, and the pin moves through
the same PR/build/publish path as every other version bump.
Watch (helpers/upstream-watch.py)
git_branch gains tag_pattern: the newest release tag in the pinned
commit's own history, exposed as {tag}/{version}/{distance}, so a
branch build is versioned <tag>.r<n>.g<sha>, above the release it
follows and below the next one. One blobless clone per branch per
run, shared by every package on it. min_release_age selects the
newest commit older than the window, so a push burst builds once.
Lane (helpers/package-metadata.sh, bin/sync-upstream --lane)
"auto_merge": true moves a package from the reviewed 6-hourly sync
PR to the unattended lane. Packages pinned from the same branch move
together: a failure on one restores the others and fails the group,
so the dev pair can never ship from two quattro commits.
Tracker (.github/workflows/track-branches.yml)
Every two hours: pin, open one PR with a GitHub App token, enable
auto-merge. Branch protection still gates the merge on result,
self-tests and build-isolation. A tip that fails to build stays an
open red PR until the next tick supersedes it. The App is required:
a PR opened with GITHUB_TOKEN has its checks held for approval and
its auto-merge would not fire publish.yml.
The reviewed workflows (sync-upstream, sync-rebuilds) open their PRs
with the same App so their builds start without a maintainer clicking
"Approve workflows to run"; without the App they fall back to
GITHUB_TOKEN and behave as before.
Recipes
The dev pair pins _commit and a real sha256sum, keeps the OMARCHY_SRC
override, and drops pkgver(). Its r-number stays the branch's total
commit count because the published history used it and pacman must
never see the version go down. omasnap-git is new: omacom/omasnap
main, versioned <tag>.r<distance>.g<sha>, provides/conflicts omasnap.
The droplet pool is x86, so aarch64 builds ran under QEMU about 30x slower;
omarchy-mac-boot's check() took 2h47m of the 180-minute job limit. Heavy
packages stay on the droplets until a native build of each is shown to fit.
makepkg runs check() inside the build container and meson writes each
test's output to the build tree, not to stdout. A failing test therefore
leaves only a summary line in the job log. Diagnosing it means reading
the package source and inferring the cause.
bin/build bind-mounts $SRC_DIR at /src, so the logs outlive the
container at src/**/meson-logs/ on the runner. Upload them when the
build step fails.
owe 0.2.7 showed the cost: owe:transition failed on aarch64 and passed
on x86_64, and CI carried no record of which assertion tripped.
Co-authored-by: Bjarne Oeverli <1419214+bjarneo@users.noreply.github.com>
Three things kept the upstream sync PR (#589) from ever finishing a build:
Scoped dispatches wiped the shared PR. A workflow_dispatch with `packages`
regenerates only those packages from master, and pushing that to
auto/sync-upstream replaced 38 pending updates with one. Scoped runs now
push to their own auto/sync-{upstream,rebuilds}-<packages> branch and PR;
scheduled runs keep the shared branch.
build-approved stopped working after the first bot push. A GITHUB_TOKEN
push creates pull_request runs held for approval but no pull_request_target
run, so approve-pr.yml never saw it: its last run on the branch was the
label itself (2026-09-25T19:26), and each of the next four syncs sat at
action_required. The sync workflows now release the held runs for the
commit they just pushed, from a separate job holding actions: write, and
only for their own bot-authored, same-repo PR while build-approved is on
it.
Each approved push cancelled the in-flight build. Approving the 21:43
sync's build cancelled the label-triggered one still queued on strata and
schist-bin. On auto/sync-* branches a new build now waits for the running
one instead, then reuses its artifacts. The approval script no longer
waits for a lone approved build to start before releasing tests, which a
queued build would have turned into a timeout.
On aarch64 the Limine hooks and the mkinitcpio wrapper run through
limine-apple-gate. Only an Apple Silicon Mac (device tree "apple,")
behaves differently: Limine's kernel and removal hooks wait until
Omarchy activates Limine (/var/lib/omarchy/limine.enabled and
/etc/default/limine), mkinitcpio's own kernel hook keeps /boot current
before and after activation, Limine's EFI deploy hook is left to
omarchy-mac-boot, and the wrapper is plain mkinitcpio. Skipped hooks
drain the socket pacman streams targets over.
x86_64 packages exactly what 1.39.0-1 did; generic aarch64 and
Snapdragon keep upstream behaviour.
Every push to a PR rebuilt every package the PR touches, on every
architecture, even when only one of them changed. The daily sync PR
carries around thirty package/arch pairs; fixing one package meant
rebuilding all of them, and an aarch64 build under QEMU takes up to an
hour. Nine runs of that PR cost about 36 droplet-hours in two days.
The planner now asks the artifact store for <pkg>-<arch>-<tree hash>
before adding an entry to the matrix and drops entries that already
have one. That is the same lookup publish.yml makes on merge, so a
reused entry publishes exactly the file it would have anyway. Dry run
against the current sync PR: 27 of 31 entries reused, 4 built.
Pack and Upload no longer run with always(): only a successful build
uploads, so an artifact's existence means that tree built.
workflow_dispatch always builds; it is an explicit request.
github.event.pull_request.base.sha is a snapshot taken when the PR was
last pushed, not the current tip of the base branch. A reopened or rerun
PR therefore builds with whatever master looked like at its last push,
and a tooling fix that landed on master since then never reaches it:
the daily sync PR reopened after #553 merged checked out a base without
helpers/artifact-helpers.sh and failed at "Pack artifact".
Check out base.ref instead. The plan's diff and the empty-PR check still
compare base.sha to head.sha, so the list of changed packages is
unaffected; only the tooling that runs on the droplet moves to the tip.
actions/upload-artifact rejects any path containing ':', and makepkg names
a package with an epoch `name-1:ver-rel-arch.pkg.tar.zst`. Every PR that
built such a package (cursor-cli in the sync PRs, omasnap once it gained an
epoch) failed at "Upload artifact" after a successful build, and publish
then rebuilt from scratch on merge.
The files now ride inside packages.tar for the artifact hop and come back
out with makepkg's names untouched: pacman clients and bin/publish-artifact
both require the filename to match PKGINFO, and the channels already carry
these names. publish.yml still accepts bare pre-packing artifacts until the
7-day retention drains them.
helpers/artifact-helpers.sh holds both halves; tests/artifact-helpers.sh
covers the round trip and runs with the other self-tests.
* Package Omawake 0.0.3 and Omaspeak 0.0.2
Bump both -bin packages to the model-support roadmap delivery:
Omawake 0.0.3:
- W02-W05 setup/activation/cache gates audited and closed
- W07 pinned catalog URL health checks and import diagnostics
- W08 Moonshine Small/Medium benchmarked; both deferred (Tiny default)
- W09 Spanish wake profile (multilingual Whisper Base INT8, es)
- W10 connection-owned playback pauses (HoldPause)
Omaspeak 0.0.2:
- S09 Kokoro 82M: Kokoro-capable packaged provider (supertonic;kokoro_tts)
with espeak-ng-data.bin shipped beside the executable, 54 named voices
- S10 catalog URL checks, Spanish speech profile, consistent status shape
- S07 streaming deferred at the current pin
Upstream: omawake v0.0.3, omaspeak v0.0.2 (aarch64 + x86_64 verified on
promaxgb10-d666 CUDA and local NPU installs).
* omaspeak-bin: install espeak-ng-data.bin beside the packaged library
* omaspeak-bin: bump to 0.0.3-rc.1 (catalog-managed eSpeak data)
- The tarball no longer ships espeak-ng-data.bin: the Kokoro catalog row
pins the data package as a model asset (downloaded/verified/installed
into the model directory with the GGUF), so the core package ships no
model data at all.
- Both arch checksums taken from the v0.0.3-rc.1 SHA256SUMS.txt.
* omaspeak-bin: finalize at 0.0.3
* Stop setup-created Oma services before pacman removes their binaries
* Drop stale release-verification fixtures from the branch
These were swept in by git add -A during the version bumps: packaged
copies of old releases (0.0.1 tarballs and extracted trees, ~80 MB)
belong to the local verification workflow, not to the package repo.
The consolidated upstream PR should carry only the package changes,
hooks and the removal regression suite.
* Update removal-test fixture versions to the packaged finals
* Ask systemd to reset only an Oma unit that actually failed
The removal helper reset the failed state of every unit it stopped, but
systemd accepts ResetFailed for a unit that is in the failed state alone.
For any other state it answers that the unit is not loaded and exits
non-zero, and because the helper runs under errexit while the hook aborts
on failure, a healthy unit then aborted the whole transaction:
(2/2) Stop and remove omaspeak user services before package removal
Failed to reset failed state of unit omaspeak.service: Unit omaspeak.service not loaded.
:: Could not clean up omaspeak for jacob; removal aborted.
That is the ordinary case, as the packaged service ships disabled and an
enabled one is commonly stopped rather than failed. Read the active state
after the stop and ask for the reset only where it applies, so a failed
unit still loses its failed state along with its rate and restart counters
while a clean unit no longer fails the removal. A reset that a reachable
manager still refuses stays fatal.
Model the rule in the removal suite, where reset-failed now follows the
active state the way a real manager does, and cover both outcomes: an
inactive unit must not be asked for, a failed one must be reset between
the stop and the disable, and a refused reset must still fail the hook.
* Keep removal cleanup faithful to how systemd reads configuration
Both defects from the review of e025111 sat in the shared package-remove
helper, so both packages were affected the same way.
An offline user's drop-in was recognised by grep '^ExecStart=', while the
configuration parser throws away the whitespace around an assignment
(parse_line() strips the line and both halves of the assignment). A drop-in
naming a development build as
ExecStart =
ExecStart = /home/alice/build/omawake daemon
therefore went unmatched, and the helper cleared away the generated base unit
beside with its enablement links, right behind a service that was never meant
to be the package's. Match an assignment the way the parser accepts one. The
gate that decides whether a unit file is the generated one stays strict on
purpose: only the exact generated shape is ever deleted.
systemctl show-environment also prints every value the way a shell would read
it, through shell_maybe_quote(SHELL_ESCAPE_POSIX), so an XDG_CONFIG_HOME with a
space arrives as $'/home/alice/custom config'. The XDG_CONFIG_HOME=/* case saw
neither form and kept the home's .config directory quietly, leaving the unit in
the directory the manager really reads pointing at the removed binary. Decode
that quoting character by character, without letting the text become shell
syntax, and refuse a value that is neither a plain path nor a closed $'...'
quote rather than delete what would have to be guessed at. A value that is not
an absolute path stays the fallback it is in systemd itself.
The fixtures now hand the helper the very text a manager prints, quoted by a
mirror of that printer, and cover a quoted path with a space, an apostrophe and
a backslash, an unreadable quoted value, a relative one, and each spacing of an
offline override. Verified with the removal suite, 15 tests; the eight new
assertions fail against the helper as it was. The other suites were not run
here, as they reach for the network.
pkgrel 3 -> 4 and the helper's checksum, in both recipes.
Reported-by: spencerbull
* Decode systemd control escapes during service removal
systemctl C-escapes control bytes in show-environment output. Rejecting those valid values aborted package removal for every user, even when the affected account had no Oma service. Decode the printer’s named and octal escapes without evaluating shell syntax or stripping trailing newlines, and cover the real printer format in the fixtures.
Co-Authored-By: GPT-6 XHigh <noreply@openai.com>
---------
Co-authored-by: Spencer Bull <spencer@omarchy.org>
Co-authored-by: GPT-6 XHigh <noreply@openai.com>
A workflow_dispatch runs from master's head. That commit's PR merged
something unrelated, so looking the PR up by commit attached a failed
elsewhen report to #515, whose merge had nothing to do with elsewhen.
Dispatch runs now go to the log only. The log append also moves ahead
of the PR comment so the record exists by the time anyone follows the
comment to it.
Re-running publish for a package that is already live (a dispatch for
something that turned out fine, or a retry after a partial failure) made
bin/build report nothing to build and exit 2, which the publish step
treated as an error. The collect step now dry-runs first: if the channel
already holds master's version the package is recorded as
already-published and skipped, and a run where every package is in that
state exits cleanly with a record saying so.
When a package had no PR artifact and its build failed, the publish
step never ran, no record was written, and the report job failed
looking for it. The collect step now records each package's source
(PR artifact, built here, or build-failed) and writes the record itself
when a build fails, so the report can say plainly that nothing was
published and why.
A PR whose diff against its base is empty has already landed some other
way, typically a sync PR carrying the same bump or a merge from master
that swallowed it. Merging it records a change that isn't one and could
mask a real mistake. result now fails with a message saying to close it.
The publish job now writes publish-record.json describing every
channel/architecture slot it touched: the packages, whether the slot was
published or failed, the target (live or a proof prefix), the commit and
the run. A report job renders that as a comment on the PR the merge
commit came from (looked up by commit, so squash and rebase merges work)
and appends the record as one line to publish-log.jsonl in the bucket,
served next to the packages at https://pkgs.omarchy.org/publish-log.jsonl.
Failures are reported too, with the slots that landed before the failure,
which is when a human most needs to know.
Every pull request now builds the package directories it touches on
ephemeral DigitalOcean droplets, and every merge to master publishes the
resulting artifacts into the channels each package belongs to. The
repository host's timers become the fallback rather than the pipeline.
Build (.github/workflows/build-pr.yml)
One job per package per architecture, always against edge. The artifact
is labelled with the package directory's git tree hash. Tooling (bin/,
helpers/, build/) is checked out from the base branch; the PR supplies
only pkgbuilds/, so a PR can change what is built, never how. Builds
run only for trusted authors: collaborators, .github/VOUCHED.td, or a
PR carrying the build-approved label. A single required check, result,
aggregates the matrix.
Publish (.github/workflows/publish.yml, bin/publish-artifact)
One job per merge. It collects the PR artifacts for the merged tree,
builds anything that has none, then walks each channel/architecture
slot once: pull that database, repo-add every package that belongs in
it, upload packages, signatures, then the database. A published
filename is immutable; identical bytes under an existing name only
gain a database entry, different bytes are refused. Fast-ring packages
reach edge, rc and stable in the same run from the same file.
Matrix (bin/build-matrix)
Package x architecture, with the channels the artifact ships to,
decided by package_builds_for_mirror so CI and the host agree.
arch=any packages build once and land in every architecture database.
Builder (build/build.sh, bin/build, build/Dockerfile)
With no local published tree, plan against and resolve from the public
channel over HTTPS. PACKAGER is set to Omarchy. rclone is in the image.
Runners (ci/)
A controller droplet polls GitHub with curl and creates one g5 droplet
per queued job from cloud-init, deleting them when off or over-age.
Builders carry QEMU with credential support for aarch64. Operator SSH
keys come from GitHub. tests/controller.sh and tests/publish-artifact.sh
cover the decisions against fixtures and real makepkg output.
Tests run on pull requests only; branch protection requires result,
self-tests and build-isolation with up-to-date branches.
The root-run package hook changed ownership of paths below a
user-controlled home directory. A config symlink could redirect chown to
an arbitrary root-owned file during installation or upgrade.
Run the config writer as the target desktop user and remove the privileged
ownership changes. This also prevents the missing-config path from writing
through a user-controlled pathname as root. Add regression coverage and
bump the package release.
Reported-by: piratemoo (Esther) <22439214+piratemoo@users.noreply.github.com>
Link: https://github.com/piratemoo/Arbitrary-File-Ownership-Change-via-Symlink-LPE
Exercise both settings recipes with synthetic runtime files. Check the exact Limine templates, boot drop-ins and backup metadata, installer-owned configuration, notifier copies and architecture-specific Thunderbolt handling. Run package regressions in the existing self-test job.
Revert the native installer and updater packaging introduced by #325. Keep the prebuilt desktop and existing launcher, updating only the release tag, commit and archive checksum from the previous recipe.
Install Hermes into its writable native layout instead of shipping a frozen /opt desktop. Preserve the native update path, migrate tagged bootstraps, and keep existing CLI launchers until the desktop is ready.
Co-Authored-By: GPT-6 Codex (xhigh) <noreply@openai.com>
Ryan Hughes