The four jobs on GitHub's own runners move from ubuntu-24.04 and
ubuntu-latest to ubuntu-26.04.
On 26.04 /tmp is held in memory, so the publish job's artifact, 11 GB for
edge, goes to the runner's temp directory on disk instead. nightly-fetch
now checks there is room before it starts and says how much is missing.
Ubuntu's rclone is 1.60.1, from 2022. In the first publish R2 answered
every upload of the first attempt "501 Not Implemented", having stored
the file each time, and the job passed because rclone's second attempt
found everything in place.
Run against a stand-in server, 1.60.1 and the current 1.75.2 make the
same requests but for one header: 1.60.1 sends "x-amz-acl: private"
with each upload and 1.75.2 sends none to Cloudflare. The publish job
now installs 1.75.2, by version and checksum.
Should an upload pass only on a retry again, the run says so in a
warning.
The first publish spent 9 min 42 s in the download action for the 4.6 GB
stable release; the 11.5 GB edge release would not have fitted in the
job's 45 minutes. GitHub's artifact storage limits each connection to a
few megabytes a second, and the action uses few.
An artifact is one zip at a signed address. test/vm-image/nightly-fetch
asks the API for that address and takes it with aria2c over 16
connections, checks the size and unpacks it. The same 4.6 GB artifact
came down in 123 s that way. The publish job gets actions: read for it,
and checks out test/vm-image for the script.
test/unit/nightly-test.sh plays the network with stand-ins: the newest
artifact of a name is the one fetched, a failed download is tried again
with a new address, a wrong size is refused and not unpacked.
actions/checkout v5.0.0 -> v7.0.1
actions/upload-artifact v6.0.0 -> v7.0.2
actions/download-artifact v4.3.0 -> v8.0.2
docker/setup-buildx-action v4.3.0 -> v4.4.1
Each stays pinned to a commit, with the version in its comment.
From the release notes, what the new majors change and why none of it
touches these workflows: download-artifact v5 moved the path of an
artifact fetched by ID, and ours is fetched by name; v8 fails on a
checksum mismatch where v4 said nothing, which is what a publish job
wants. checkout v7 refuses a fork's code under pull_request_target and
workflow_run, and the one workflow_run here checks nothing out. The
Node 24 releases need runner 2.327.1; ours run 2.337.0.
Pinned actions do not update themselves, so .github/dependabot.yml asks
for one grouped pull request a week.
An install test failed 13 seconds in: a public Arch mirror dropped two
signature files while the test container was installing QEMU. The three
test containers now ask mirror.omarchy.org first, with the public
mirrors after it, and try up to three times.
The image was a folder, vm/, holding base.qcow2 and what boots it. It is
now one file next to the ISO, with the ISO's name and its own checksum
file, the way Ubuntu, Debian, Arch, Fedora and CentOS publish theirs:
edge/<version>/omarchy-edge-<version>.qcow2
stable/<version>/<run>/omarchy-<version>.qcow2
The image boots by itself through UEFI. What a direct boot needs, the
kernel, initramfs, kernel arguments and vm-boot, is in an unpacked/
folder beside it, Ubuntu's name for the same thing. The firmware
variables are no longer published: a boot with the firmware's defaults
finds the disk.
latest.json follows: vm.url, vm.sha256 and vm.size are the image, as
url, sha256 and size are the ISO; vm.unpacked and vm.unpacked_files are
the folder. vm-boot takes the image file. The publish job compares every
file it uploaded, whatever the layout. README, tests and the workflow's
header say the same.
The install is a numbered list of steps, each starting with its verb.
Verifying the download and the manual's pages have their own headings.
Long sentences are split, compressed ones written out, and one thing has
one name: "nightly build", "latest release".
The part for users pointed at nightly.omarchy.org for a VM image. Users
are sent to the release ISO only: the commands read the latest release
of omacom/omarchy, fetch that ISO and its checksum from iso.omarchy.org
and check it. The nightly site and the VM images are for testing, and
everything about them is now under "For maintainers".
Download, write to a stick, turn off Secure Boot, boot, answer the
questions: the manual's own words, shortened, then links to its pages on
getting started, dual booting and unattended installs. The detailed
autoinstall reference moves under "For maintainers" and points at the
manual's page for users.
The top is for someone who wants Omarchy: where to get the ISO and how
to check it, how to try a VM image, and autoinstall. Everything about
building, testing and publishing follows under "For maintainers", with
what is published where, how the nightly works, the fields of
latest.json and the files of a VM image.
A read against the code corrected several statements, some of them
older: the installer runs omarchy-provision-user; a defer-provisioning
marker can stand in for the credentials file; an authorized_keys file
fails the install when it holds no keys, not when its keys are bad; the
stock firewall also allows Docker's DNS; built_at is when latest.json
was written; vm-boot's needs are listed in full.
The publish job now fetches the two checksum files back as well, so
"every file of the version is compared" is exact.
One table of the release ISO, the edge ISO and the two VM images with
their addresses, then a part for each: the release ISO and its checksum,
the nightly, every field of latest.json, and the files of a VM image
with the commands that fetch, check and boot one.
The three configs the build downloads the ISO's packages with had
"SigLevel = Optional TrustAll" on [omarchy]. Optional checks a signature
when there is one and accepts a package that has none; TrustAll accepts
any key in the keyring. So a package whose .sig was missing went onto the
ISO unchecked.
Without the line, [omarchy] follows the configs' own default, Required
DatabaseOptional, which is what an installed system uses. Every package
in the repository is signed, and the build adds and signs the release
key before its first download, so nothing else changes. [arch-mact2]
keeps its own SigLevel = Never: those packages are unsigned upstream.
The offline mirror index test compares databases with diff and the
nightly test reads JSON with jq. Neither is in the Arch base image, so
both tests failed in CI while passing on a workstation.
The cache kept the ISO's packages between runs. A package that has gone
missing from the mirror would then still be found on the runner, and the
nightly would build and pass while a fresh machine could not. Every
package is fetched again each night: with the T2 repository asked on
GitHub first, the build is 3 to 4 minutes without a cache.
Which channels a run makes and where they go, and what is laid out and
what latest.json then says, were shell inside the workflow file, tried
only by hand. A green run published a latest.json with the Omarchy
version in it twice, and nothing noticed.
Both are scripts now, test/vm-image/nightly-plan and nightly-layout,
and test/unit/nightly-test.sh tries them without a runner: a first run,
a run started again, a release already published, a tag that is not a
version; every field of latest.json for each channel, the checksums
against the files, the version a consumer reads from the ISO's name.
nightly-layout also refuses to describe an image it cannot describe
fully, and checks its own latest.json before anything is published.
The same test covers two things the install test decides: which Omarchy
packages an ISO installs, and when an install is over.
On edge, pacman answers for both omarchy and omarchy-dev with the one
installed package, so the version and the commit were each written
twice, with a line break between.
Starting a run again keeps its run number, so a second attempt would
have built new images for the paths the first attempt had already
partly filled, and the upload refuses to replace a file. From the second
attempt on, the attempt is part of the number: 431, then 431.2.
The same runner the nightly uses. Its kernel has no Landlock, so the
test's container steps around pacman's sandbox the way the ISO build
does. It gets no cache volume: this workflow runs for pull requests, and
a cache one of them wrote would be trusted by the next run.
The wait loop printed "installing (Ns)" until SSH answered on the
installed system. A nightly run sat on that line for twelve minutes
while the install had finished after two: the first boot was what hung.
The firmware names what it starts on the serial port, so the loop now
notices the installed disk's Limine starting, logs how long the install
took, and counts the first boot from there.
Next to edge, the nightly now makes a VM image of the last release. The
latest release of omacom/omarchy names the version, iso.omarchy.org has
the ISO, and it is used only if its signature is the release key's, the
one in builder/omarchy.gpg. It gets the same install test, seal and
checks, and is published under stable/ with its own latest.json, whose
url is the ISO's existing place: the ISO is not uploaded again.
A release is done once. A plan job decides what a run makes: edge
always, stable unless a scheduled run finds that version already in
stable/latest.json. Build and publish run once per channel, and one
channel failing does not hold back the other.
The install test's key sits in the user's home, which is its own
subvolume and not part of the factory snapshot, and the seal removes it
there. The check says so of every image, instead of leaving it to the
layout staying as it is.
The seal read /boot/EFI/Linux/omarchy_linux-omarchy.efi. An edge install
has linux-omarchy-bore, so the first nightly run stopped there. It now
asks the running system which kernel package it booted and takes that
UKI, and takes the command line from /proc/cmdline, since the boot
entry, not the UKI, may be what holds it. The nightly's package report
names the kernel the same way.
linux-t2, the kernel every x86 live ISO boots, comes from arch-mact2,
which has two servers in these configs: the maintainer's host in Seoul
and their GitHub release. Both carry the same database. An ISO build on a
runner in Virginia waited on the Seoul host at 0.45 MB/s for a 92 MB
package; the same file from GitHub came at 58 MB/s.
GitHub goes first, which is also the order of arch-mact2's own mirror
list. The Seoul host stays as the fallback.
A nightly downloads the packages of the ISO's offline mirror, a few
gigabytes, and the test container's own, and would download most of them
again the next night. On Namespace both directories live on the runner
profile's cache volume. The build already prunes from its cache what an
ISO no longer needs. The step is skipped, not failed, on a runner
without a cache volume.
repo-add takes one package at a time: it hashes the archive and lists its
files, which means decompressing all of it, for about a thousand
packages one after another.
A database is a tar of one directory per package. The packages are dealt
out to one repo-add per CPU, each writing its own database, and the
pieces are joined. On 160 packages on a 16-core machine the result is
identical to one repo-add's, db and files, in 1.0 s against 6.3 s. Two
versions of one package name are an error: one repo-add would keep the
last, the pieces apart would keep both.
GitHub's hosted runner has 4 cores. The build job moves to
namespace-profile-linux-amd64-16x32; the publish job, which only copies
files and holds the bucket's key, stays where it is.
Those runners' kernel has no Landlock, so pacman 7 fails its first sync
in a container there. The ISO build already steps around that; the
install test's container does the same now. The job also prints what it
is running on: CPU, memory, LSMs and /dev/kvm.
pacman 7 sandboxes its downloads with Landlock and fails every database
sync on a kernel without it, which Namespace's AmpereOne arm64 machines
run: "restricting filesystem access failed because Landlock is not
supported by the kernel!". An ISO build there stops at its first
pacman -Sy.
Where /sys/kernel/security/lsm lists no landlock, or cannot be read, a
wrapper in /usr/local/bin runs every pacman in the build container,
pacstrap's included, with --disable-sandbox, the flag pacman provides for
such kernels. Not DisableSandbox in a config: pacman-offline.conf ships
on the ISO.
The nightly built the stable ISO and kept it as an artifact for a day.
It now builds the edge ISO, runs the install test on it, seals the
installed disk that test leaves behind into a VM image, checks the
image, and publishes ISO and image under a version that is never reused:
nightly.omarchy.org/edge/latest.json
nightly.omarchy.org/edge/<version>/omarchy-edge-<version>.iso
nightly.omarchy.org/edge/<version>/vm/
latest.json gives the version, the ISO's URL and checksum, the
omarchy-iso commit, the Omarchy version and commit inside, and where the
image is. It is written last, after every file has been fetched back
through the CDN and compared.
Publishing is a second job, the only one with the bucket's key, and only
on quattro. There is no pull_request trigger. A manual run on a branch
builds and checks everything and publishes nothing.
test/integration already installs the ISO unattended and saves the
installed disk as test-runs/<iso>-integration/base.qcow2, with its
firmware variables and the key that logs in to it. That folder is what
vm-seal takes, so the separate install script goes.
The sealed image now also carries the firmware variables, which hold the
boot entry the installer registered: a consumer that boots the disk
through UEFI needs them, a direct kernel boot does not.
builder/build-iso.sh puts omarchy-dev and omarchy-settings-dev on an edge
ISO, as it does on a dev or local one. The harness picked its package
names from the ISO's file name and knew dev, local and pr, so for an edge
ISO it asked the installer for the release packages.
As they were tested by hand on the 4.0.4 ISO, before any of it is wired
into a workflow:
vm-install unattended install of an ISO into a qcow2, through cidata
vm-seal makes the disk publishable: SSH takes keys only, no machine
ID, host keys, build key or build logs left, compressed
with zstd, with checksums and the kernel pulled out for
direct boot
vm-verify boots the sealed image once and checks all of that
vm-boot boots a throwaway copy with a key made for that boot and
runs a command in it
On this machine: install 114 s, seal 27 s, a 4.2 GB image, SSH about 5 s
after power-on, 17 of 17 verify checks, and 7 of them failing on an
image sealed without these fixes.
* Carry the installer input-method choice into the target
* Support Archinstall 4.5 sanity checks
* Constrain VM keyboard choices to supported credential layouts
* Cover VM keyboard offsets and reject absent choices
./test/all runs the cidata and orchestrator tests under test/unit/ in a couple of seconds, with no ISO. It runs in an Arch container, where the code it covers runs: the container boots systemd for the tests that ask localectl for its keymaps, and the tests run as an ordinary user. The install test keeps covering everything that needs a VM.
The harness runs as root in its container, so test-runs and the serial
logs QEMU writes there belong to root. The runner's user could then
neither record the pull request number beside them nor read
install-serial.log into the artifact (EACCES at zip time), which failed
the job after a passing install and boot. A chown -R step after the
container step, run even on failure, fixes both. The install loop now
prints a line every 30 s and still screenshots every 2 min.
Review fixes for the comment workflow. Artifacts are downloaded by name
for the exact attempt that finished, and for the baseline run's latest
attempt, since a rerun keeps earlier attempts' artifacts under the same
run. Every lookup prints nothing instead of failing, so no quattro
baseline yet, an expired one, or a missing boot file shows as "–"
rather than ending the script under set -e before any comment. The
install test's own marker step takes ownership of test-runs first,
which the harness created as root inside its container.
The harness now runs in archlinux:latest with /dev/kvm passed through,
so CI uses Arch's current QEMU and OVMF (QEMU 11.1, edk2 202608) at the
paths the harness expects, the same as an Omarchy workstation, instead
of Ubuntu 24.04's QEMU 8.2 with symlinks and shims. Arch splits QEMU's
devices into modules: virtio-vga needs both qemu-hw-display-virtio-gpu
and qemu-hw-display-virtio-vga, and QEMU 11.1 segfaults on
-device virtio-vga when only the latter is installed.
wait_for_ssh counted only its sleeps toward the timeout. Under QEMU user
networking each attempt can also spend its whole ConnectTimeout waiting
for a banner, so a "300 s" wait ran for ten minutes or more without a
line of output. It now uses a wall-clock deadline, polls every 2 s, and
prints a progress line with a console screenshot every 30 s. When the
boot scenario times out it prints the serial console's last lines too.
Review fixes. The first-boot systemd-analyze now waits (bounded) for
startup to finish, since SSH can answer first and systemd-analyze
refuses until it has. In the boot scenario, a non-zero exit from
is-system-running (as for "degraded") no longer ends the script under
set -e before the diagnostics are saved, and a failed query for failed
units writes a marker instead of an empty file that read as "none".
A second workflow, triggered when the install test completes, posts one
comment on the pull request and updates it on every run: install, first
boot and everyday boot next to quattro's latest numbers, with the change,
and the install phases in a collapsed table. Pushes to quattro now run
the install test too, so there is always a baseline.
It runs from quattro's copy with write access, which works for pull
requests from forks, so it never checks out or runs pull request code.
It reads only the finished run's artifact, trusts the pull request
number in it only when that pull request's head is the commit the run
tested, and keeps phase names to plain text.
Emir Beganovic