A channel that holds the version the machine already has upgrades
nothing, and every other check passes on a machine that was only
rebooted: rc is in that state today. A twelfth check requires the
Omarchy package to have changed.
Two runs on different branches both push pictures to one branch. The
second to push was refused and its comment never written. The push is
now tried again on top of whatever got there first; each run's folder is
its own, so there is nothing to merge.
The workflow no longer runs for pull requests. It is started by hand,
and still posts to the pull request of the branch it was started on.
The fork fallback and the fork guard go with the trigger.
Two checks could pass without the thing they named being true:
The wallpaper check asked only whether the file the link points at
exists, so an update that swapped the user's picture for another would
pass. It now compares before and after: the same picture, whatever
format it is kept in, and present.
The failed-units check read an empty answer as nothing failed, which is
also what a service manager that cannot be asked gives. A query that
fails is now recorded as that, and fails the check.
Every test so far installs fresh. Nothing took a machine on the last
release and updated it, which is what most users of a new release do.
test/upgrade-test takes an installed system, moves it to edge or rc with
omarchy-channel-set in a login shell, as a user would, reboots it
through the firmware so the new kernel is the one that starts, and
checks eleven things: the update's exit status, skipped migrations, that
the reboot comes back by itself, into a new boot and an installed
kernel, the channel, the system's state, failed units, the desktop, the
wallpaper the user had, and the boot menu. It writes a report and takes
a picture of the unlocked desktop before and after. A reboot that only
comes back after a key press is a failure, and is said to be that.
The Upgrade test workflow installs a release from its signed ISO with
the install test, then runs it. It is started by hand, and runs for a
pull request that changes the test. A second job, the only one that can
write, commits the pictures to the upgrade-test-screenshots branch and
posts the report as one comment on the branch's pull request, updated in
place on later runs.
Run by hand from 4.0.4 to today's edge it found two things: the
wallpaper link left pointing at a file the update renamed
(omacom/omarchy#14596), and one reboot in three stuck on a black screen
until a key was pressed (omacom/omarchy#14771).
A unit test run stopped before it began: Docker Hub refused to hand out
archlinux:latest, "You have reached your unauthenticated pull rate
limit". GitHub's runners share addresses, so the limit is not ours to
stay under.
Arch publishes the same image as ghcr.io/archlinux/archlinux, which has
no such limit from a runner. The three workflow containers and the ISO
build's own container pull it from there.
The warning was written for the bulk upload only, and counted lines:
latest.json is uploaded by a second rclone call, which could still pass
on a retry unseen, and a file that failed on two attempts was counted as
two uploads.
Both calls go through test/vm-image/nightly-rclone, which passes
rclone's output and exit status through and names each file once. A
stand-in rclone tests it.
The four jobs on GitHub's own runners move from ubuntu-24.04 and
ubuntu-latest to ubuntu-26.04.
On 26.04 /tmp is held in memory, so the publish job's artifact, 11 GB for
edge, goes to the runner's temp directory on disk instead. nightly-fetch
now checks there is room before it starts and says how much is missing.
Ubuntu's rclone is 1.60.1, from 2022. In the first publish R2 answered
every upload of the first attempt "501 Not Implemented", having stored
the file each time, and the job passed because rclone's second attempt
found everything in place.
Run against a stand-in server, 1.60.1 and the current 1.75.2 make the
same requests but for one header: 1.60.1 sends "x-amz-acl: private"
with each upload and 1.75.2 sends none to Cloudflare. The publish job
now installs 1.75.2, by version and checksum.
Should an upload pass only on a retry again, the run says so in a
warning.
The first publish spent 9 min 42 s in the download action for the 4.6 GB
stable release; the 11.5 GB edge release would not have fitted in the
job's 45 minutes. GitHub's artifact storage limits each connection to a
few megabytes a second, and the action uses few.
An artifact is one zip at a signed address. test/vm-image/nightly-fetch
asks the API for that address and takes it with aria2c over 16
connections, checks the size and unpacks it. The same 4.6 GB artifact
came down in 123 s that way. The publish job gets actions: read for it,
and checks out test/vm-image for the script.
test/unit/nightly-test.sh plays the network with stand-ins: the newest
artifact of a name is the one fetched, a failed download is tried again
with a new address, a wrong size is refused and not unpacked.
actions/checkout v5.0.0 -> v7.0.1
actions/upload-artifact v6.0.0 -> v7.0.2
actions/download-artifact v4.3.0 -> v8.0.2
docker/setup-buildx-action v4.3.0 -> v4.4.1
Each stays pinned to a commit, with the version in its comment.
From the release notes, what the new majors change and why none of it
touches these workflows: download-artifact v5 moved the path of an
artifact fetched by ID, and ours is fetched by name; v8 fails on a
checksum mismatch where v4 said nothing, which is what a publish job
wants. checkout v7 refuses a fork's code under pull_request_target and
workflow_run, and the one workflow_run here checks nothing out. The
Node 24 releases need runner 2.327.1; ours run 2.337.0.
Pinned actions do not update themselves, so .github/dependabot.yml asks
for one grouped pull request a week.
An install test failed 13 seconds in: a public Arch mirror dropped two
signature files while the test container was installing QEMU. The three
test containers now ask mirror.omarchy.org first, with the public
mirrors after it, and try up to three times.
The image was a folder, vm/, holding base.qcow2 and what boots it. It is
now one file next to the ISO, with the ISO's name and its own checksum
file, the way Ubuntu, Debian, Arch, Fedora and CentOS publish theirs:
edge/<version>/omarchy-edge-<version>.qcow2
stable/<version>/<run>/omarchy-<version>.qcow2
The image boots by itself through UEFI. What a direct boot needs, the
kernel, initramfs, kernel arguments and vm-boot, is in an unpacked/
folder beside it, Ubuntu's name for the same thing. The firmware
variables are no longer published: a boot with the firmware's defaults
finds the disk.
latest.json follows: vm.url, vm.sha256 and vm.size are the image, as
url, sha256 and size are the ISO; vm.unpacked and vm.unpacked_files are
the folder. vm-boot takes the image file. The publish job compares every
file it uploaded, whatever the layout. README, tests and the workflow's
header say the same.
The top is for someone who wants Omarchy: where to get the ISO and how
to check it, how to try a VM image, and autoinstall. Everything about
building, testing and publishing follows under "For maintainers", with
what is published where, how the nightly works, the fields of
latest.json and the files of a VM image.
A read against the code corrected several statements, some of them
older: the installer runs omarchy-provision-user; a defer-provisioning
marker can stand in for the credentials file; an authorized_keys file
fails the install when it holds no keys, not when its keys are bad; the
stock firewall also allows Docker's DNS; built_at is when latest.json
was written; vm-boot's needs are listed in full.
The publish job now fetches the two checksum files back as well, so
"every file of the version is compared" is exact.
The offline mirror index test compares databases with diff and the
nightly test reads JSON with jq. Neither is in the Arch base image, so
both tests failed in CI while passing on a workstation.
The cache kept the ISO's packages between runs. A package that has gone
missing from the mirror would then still be found on the runner, and the
nightly would build and pass while a fresh machine could not. Every
package is fetched again each night: with the T2 repository asked on
GitHub first, the build is 3 to 4 minutes without a cache.
Which channels a run makes and where they go, and what is laid out and
what latest.json then says, were shell inside the workflow file, tried
only by hand. A green run published a latest.json with the Omarchy
version in it twice, and nothing noticed.
Both are scripts now, test/vm-image/nightly-plan and nightly-layout,
and test/unit/nightly-test.sh tries them without a runner: a first run,
a run started again, a release already published, a tag that is not a
version; every field of latest.json for each channel, the checksums
against the files, the version a consumer reads from the ISO's name.
nightly-layout also refuses to describe an image it cannot describe
fully, and checks its own latest.json before anything is published.
The same test covers two things the install test decides: which Omarchy
packages an ISO installs, and when an install is over.
On edge, pacman answers for both omarchy and omarchy-dev with the one
installed package, so the version and the commit were each written
twice, with a line break between.
Starting a run again keeps its run number, so a second attempt would
have built new images for the paths the first attempt had already
partly filled, and the upload refuses to replace a file. From the second
attempt on, the attempt is part of the number: 431, then 431.2.
The same runner the nightly uses. Its kernel has no Landlock, so the
test's container steps around pacman's sandbox the way the ISO build
does. It gets no cache volume: this workflow runs for pull requests, and
a cache one of them wrote would be trusted by the next run.
Next to edge, the nightly now makes a VM image of the last release. The
latest release of omacom/omarchy names the version, iso.omarchy.org has
the ISO, and it is used only if its signature is the release key's, the
one in builder/omarchy.gpg. It gets the same install test, seal and
checks, and is published under stable/ with its own latest.json, whose
url is the ISO's existing place: the ISO is not uploaded again.
A release is done once. A plan job decides what a run makes: edge
always, stable unless a scheduled run finds that version already in
stable/latest.json. Build and publish run once per channel, and one
channel failing does not hold back the other.
The seal read /boot/EFI/Linux/omarchy_linux-omarchy.efi. An edge install
has linux-omarchy-bore, so the first nightly run stopped there. It now
asks the running system which kernel package it booted and takes that
UKI, and takes the command line from /proc/cmdline, since the boot
entry, not the UKI, may be what holds it. The nightly's package report
names the kernel the same way.
A nightly downloads the packages of the ISO's offline mirror, a few
gigabytes, and the test container's own, and would download most of them
again the next night. On Namespace both directories live on the runner
profile's cache volume. The build already prunes from its cache what an
ISO no longer needs. The step is skipped, not failed, on a runner
without a cache volume.
GitHub's hosted runner has 4 cores. The build job moves to
namespace-profile-linux-amd64-16x32; the publish job, which only copies
files and holds the bucket's key, stays where it is.
Those runners' kernel has no Landlock, so pacman 7 fails its first sync
in a container there. The ISO build already steps around that; the
install test's container does the same now. The job also prints what it
is running on: CPU, memory, LSMs and /dev/kvm.
The nightly built the stable ISO and kept it as an artifact for a day.
It now builds the edge ISO, runs the install test on it, seals the
installed disk that test leaves behind into a VM image, checks the
image, and publishes ISO and image under a version that is never reused:
nightly.omarchy.org/edge/latest.json
nightly.omarchy.org/edge/<version>/omarchy-edge-<version>.iso
nightly.omarchy.org/edge/<version>/vm/
latest.json gives the version, the ISO's URL and checksum, the
omarchy-iso commit, the Omarchy version and commit inside, and where the
image is. It is written last, after every file has been fetched back
through the CDN and compared.
Publishing is a second job, the only one with the bucket's key, and only
on quattro. There is no pull_request trigger. A manual run on a branch
builds and checks everything and publishes nothing.
./test/all runs the cidata and orchestrator tests under test/unit/ in a couple of seconds, with no ISO. It runs in an Arch container, where the code it covers runs: the container boots systemd for the tests that ask localectl for its keymaps, and the tests run as an ordinary user. The install test keeps covering everything that needs a VM.
The harness runs as root in its container, so test-runs and the serial
logs QEMU writes there belong to root. The runner's user could then
neither record the pull request number beside them nor read
install-serial.log into the artifact (EACCES at zip time), which failed
the job after a passing install and boot. A chown -R step after the
container step, run even on failure, fixes both. The install loop now
prints a line every 30 s and still screenshots every 2 min.
Review fixes for the comment workflow. Artifacts are downloaded by name
for the exact attempt that finished, and for the baseline run's latest
attempt, since a rerun keeps earlier attempts' artifacts under the same
run. Every lookup prints nothing instead of failing, so no quattro
baseline yet, an expired one, or a missing boot file shows as "–"
rather than ending the script under set -e before any comment. The
install test's own marker step takes ownership of test-runs first,
which the harness created as root inside its container.
The harness now runs in archlinux:latest with /dev/kvm passed through,
so CI uses Arch's current QEMU and OVMF (QEMU 11.1, edk2 202608) at the
paths the harness expects, the same as an Omarchy workstation, instead
of Ubuntu 24.04's QEMU 8.2 with symlinks and shims. Arch splits QEMU's
devices into modules: virtio-vga needs both qemu-hw-display-virtio-gpu
and qemu-hw-display-virtio-vga, and QEMU 11.1 segfaults on
-device virtio-vga when only the latter is installed.
A second workflow, triggered when the install test completes, posts one
comment on the pull request and updates it on every run: install, first
boot and everyday boot next to quattro's latest numbers, with the change,
and the install phases in a collapsed table. Pushes to quattro now run
the install test too, so there is always a baseline.
It runs from quattro's copy with write access, which works for pull
requests from forks, so it never checks out or runs pull request code.
It reads only the finished run's artifact, trusts the pull request
number in it only when that pull request's head is the commit the run
tested, and keeps phase names to plain text.
Builds the ISO a pull request produces, installs it unattended in a KVM
guest on a GitHub-hosted runner, boots the installed system again, and
writes the install time per phase plus first-boot and everyday-boot
systemd-analyze to the job summary. Every later installer change is
measured against these numbers in its own PR.
Locally against the 2026-10-04 quattro ISO: install 51.3 s (42.9 s of
it pacstrap), first boot 12.1 s, everyday boot 11.7 s with ~6 s in the
loader, systemd running, no failed units.
A disk check replaces a cleanup step: the runner has ~86 GB free, so
the 2-minute "Free Disk Space" action the nightly uses is not needed.
The workflow has been running ./build_iso.sh, which has not existed for a long
time — the build moved to bin/omarchy-iso-make. Every scheduled run in the last
100 has failed.
Run the real build driver instead, with NO_BOOT_OFFER=1 so it skips the
interactive "Boot this ISO?" prompt at the end, and collect from ./release/
where the ISO actually lands. Drop the outer archlinux container: omarchy-iso-make
starts its own privileged container, so the job only needs Docker on the runner.
Add a disk cleanup step, since a ~6.5GB ISO plus a ~4.5GB offline mirror does not
fit in a stock runner's free space.
Taken from #67 by @karthikkalidas, which was opened against main. main is 143
commits behind and is no longer the default branch, and scheduled workflows only
run from the default branch, so merging it there would have fixed nothing.
Co-Authored-By: Karthik Kalidas <karthik.kalidas@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Emir Beganovic