The question to the channel had no time limit, and a download that
failed gave an empty answer, which the test read as nothing on offer. A
server that held the connection open could keep the test until the job's
limit, and one that was down made it report "Nothing to upgrade to".
The request now has a deadline, a failure anywhere in it counts, and the
test stops with its own message and the last lines of what went wrong.
The early stop compared the offered version with the installed one for
equality. Started from a release newer than the channel holds, the test
went on to the updater with nothing to install. It now asks pacman's
vercmp and stops when the offer is the same or older.
The picture of a reboot that stuck was inside the condition that wants
both the before and the after picture, so it was left out exactly when
there was no after to show. It now has its own condition.
Run against rc, the test stopped at a password prompt and then at a
question, each until its 30 minute limit.
sudo asked for a password although the test had written a rule that
lets it through: sudo takes the last rule that matches and reads its
folder in name order, and the rule's name sorted before the install's
own. The release's updater also runs "sudo -v", which asks unless every
rule for the user lets it through. The rule now sorts last, turns the
question off for the user, and the test checks both before going on.
rc holds what stable holds today. The release's updater, run with
nothing to do, ends at a prompt nobody answers. The test now asks the
channel what it offers before it starts, and with nothing newer on offer
it stops in seconds and says so.
The comment shows the two pictures only when both exist.
A channel that holds the version the machine already has upgrades
nothing, and every other check passes on a machine that was only
rebooted: rc is in that state today. A twelfth check requires the
Omarchy package to have changed.
Two runs on different branches both push pictures to one branch. The
second to push was refused and its comment never written. The push is
now tried again on top of whatever got there first; each run's folder is
its own, so there is nothing to merge.
The workflow no longer runs for pull requests. It is started by hand,
and still posts to the pull request of the branch it was started on.
The fork fallback and the fork guard go with the trigger.
Two checks could pass without the thing they named being true:
The wallpaper check asked only whether the file the link points at
exists, so an update that swapped the user's picture for another would
pass. It now compares before and after: the same picture, whatever
format it is kept in, and present.
The failed-units check read an empty answer as nothing failed, which is
also what a service manager that cannot be asked gives. A query that
fails is now recorded as that, and fails the check.
Every test so far installs fresh. Nothing took a machine on the last
release and updated it, which is what most users of a new release do.
test/upgrade-test takes an installed system, moves it to edge or rc with
omarchy-channel-set in a login shell, as a user would, reboots it
through the firmware so the new kernel is the one that starts, and
checks eleven things: the update's exit status, skipped migrations, that
the reboot comes back by itself, into a new boot and an installed
kernel, the channel, the system's state, failed units, the desktop, the
wallpaper the user had, and the boot menu. It writes a report and takes
a picture of the unlocked desktop before and after. A reboot that only
comes back after a key press is a failure, and is said to be that.
The Upgrade test workflow installs a release from its signed ISO with
the install test, then runs it. It is started by hand, and runs for a
pull request that changes the test. A second job, the only one that can
write, commits the pictures to the upgrade-test-screenshots branch and
posts the report as one comment on the branch's pull request, updated in
place on later runs.
Run by hand from 4.0.4 to today's edge it found two things: the
wallpaper link left pointing at a file the update renamed
(omacom/omarchy#14596), and one reboot in three stuck on a black screen
until a key was pressed (omacom/omarchy#14771).
A unit test run stopped before it began: Docker Hub refused to hand out
archlinux:latest, "You have reached your unauthenticated pull rate
limit". GitHub's runners share addresses, so the limit is not ours to
stay under.
Arch publishes the same image as ghcr.io/archlinux/archlinux, which has
no such limit from a runner. The three workflow containers and the ISO
build's own container pull it from there.
The warning was written for the bulk upload only, and counted lines:
latest.json is uploaded by a second rclone call, which could still pass
on a retry unseen, and a file that failed on two attempts was counted as
two uploads.
Both calls go through test/vm-image/nightly-rclone, which passes
rclone's output and exit status through and names each file once. A
stand-in rclone tests it.
The four jobs on GitHub's own runners move from ubuntu-24.04 and
ubuntu-latest to ubuntu-26.04.
On 26.04 /tmp is held in memory, so the publish job's artifact, 11 GB for
edge, goes to the runner's temp directory on disk instead. nightly-fetch
now checks there is room before it starts and says how much is missing.
Ubuntu's rclone is 1.60.1, from 2022. In the first publish R2 answered
every upload of the first attempt "501 Not Implemented", having stored
the file each time, and the job passed because rclone's second attempt
found everything in place.
Run against a stand-in server, 1.60.1 and the current 1.75.2 make the
same requests but for one header: 1.60.1 sends "x-amz-acl: private"
with each upload and 1.75.2 sends none to Cloudflare. The publish job
now installs 1.75.2, by version and checksum.
Should an upload pass only on a retry again, the run says so in a
warning.
The first publish spent 9 min 42 s in the download action for the 4.6 GB
stable release; the 11.5 GB edge release would not have fitted in the
job's 45 minutes. GitHub's artifact storage limits each connection to a
few megabytes a second, and the action uses few.
An artifact is one zip at a signed address. test/vm-image/nightly-fetch
asks the API for that address and takes it with aria2c over 16
connections, checks the size and unpacks it. The same 4.6 GB artifact
came down in 123 s that way. The publish job gets actions: read for it,
and checks out test/vm-image for the script.
test/unit/nightly-test.sh plays the network with stand-ins: the newest
artifact of a name is the one fetched, a failed download is tried again
with a new address, a wrong size is refused and not unpacked.
actions/checkout v5.0.0 -> v7.0.1
actions/upload-artifact v6.0.0 -> v7.0.2
actions/download-artifact v4.3.0 -> v8.0.2
docker/setup-buildx-action v4.3.0 -> v4.4.1
Each stays pinned to a commit, with the version in its comment.
From the release notes, what the new majors change and why none of it
touches these workflows: download-artifact v5 moved the path of an
artifact fetched by ID, and ours is fetched by name; v8 fails on a
checksum mismatch where v4 said nothing, which is what a publish job
wants. checkout v7 refuses a fork's code under pull_request_target and
workflow_run, and the one workflow_run here checks nothing out. The
Node 24 releases need runner 2.327.1; ours run 2.337.0.
Pinned actions do not update themselves, so .github/dependabot.yml asks
for one grouped pull request a week.
An install test failed 13 seconds in: a public Arch mirror dropped two
signature files while the test container was installing QEMU. The three
test containers now ask mirror.omarchy.org first, with the public
mirrors after it, and try up to three times.
The image was a folder, vm/, holding base.qcow2 and what boots it. It is
now one file next to the ISO, with the ISO's name and its own checksum
file, the way Ubuntu, Debian, Arch, Fedora and CentOS publish theirs:
edge/<version>/omarchy-edge-<version>.qcow2
stable/<version>/<run>/omarchy-<version>.qcow2
The image boots by itself through UEFI. What a direct boot needs, the
kernel, initramfs, kernel arguments and vm-boot, is in an unpacked/
folder beside it, Ubuntu's name for the same thing. The firmware
variables are no longer published: a boot with the firmware's defaults
finds the disk.
latest.json follows: vm.url, vm.sha256 and vm.size are the image, as
url, sha256 and size are the ISO; vm.unpacked and vm.unpacked_files are
the folder. vm-boot takes the image file. The publish job compares every
file it uploaded, whatever the layout. README, tests and the workflow's
header say the same.
The install is a numbered list of steps, each starting with its verb.
Verifying the download and the manual's pages have their own headings.
Long sentences are split, compressed ones written out, and one thing has
one name: "nightly build", "latest release".
The part for users pointed at nightly.omarchy.org for a VM image. Users
are sent to the release ISO only: the commands read the latest release
of omacom/omarchy, fetch that ISO and its checksum from iso.omarchy.org
and check it. The nightly site and the VM images are for testing, and
everything about them is now under "For maintainers".
Download, write to a stick, turn off Secure Boot, boot, answer the
questions: the manual's own words, shortened, then links to its pages on
getting started, dual booting and unattended installs. The detailed
autoinstall reference moves under "For maintainers" and points at the
manual's page for users.
The top is for someone who wants Omarchy: where to get the ISO and how
to check it, how to try a VM image, and autoinstall. Everything about
building, testing and publishing follows under "For maintainers", with
what is published where, how the nightly works, the fields of
latest.json and the files of a VM image.
A read against the code corrected several statements, some of them
older: the installer runs omarchy-provision-user; a defer-provisioning
marker can stand in for the credentials file; an authorized_keys file
fails the install when it holds no keys, not when its keys are bad; the
stock firewall also allows Docker's DNS; built_at is when latest.json
was written; vm-boot's needs are listed in full.
The publish job now fetches the two checksum files back as well, so
"every file of the version is compared" is exact.
One table of the release ISO, the edge ISO and the two VM images with
their addresses, then a part for each: the release ISO and its checksum,
the nightly, every field of latest.json, and the files of a VM image
with the commands that fetch, check and boot one.
The three configs the build downloads the ISO's packages with had
"SigLevel = Optional TrustAll" on [omarchy]. Optional checks a signature
when there is one and accepts a package that has none; TrustAll accepts
any key in the keyring. So a package whose .sig was missing went onto the
ISO unchecked.
Without the line, [omarchy] follows the configs' own default, Required
DatabaseOptional, which is what an installed system uses. Every package
in the repository is signed, and the build adds and signs the release
key before its first download, so nothing else changes. [arch-mact2]
keeps its own SigLevel = Never: those packages are unsigned upstream.
The offline mirror index test compares databases with diff and the
nightly test reads JSON with jq. Neither is in the Arch base image, so
both tests failed in CI while passing on a workstation.
The cache kept the ISO's packages between runs. A package that has gone
missing from the mirror would then still be found on the runner, and the
nightly would build and pass while a fresh machine could not. Every
package is fetched again each night: with the T2 repository asked on
GitHub first, the build is 3 to 4 minutes without a cache.
Which channels a run makes and where they go, and what is laid out and
what latest.json then says, were shell inside the workflow file, tried
only by hand. A green run published a latest.json with the Omarchy
version in it twice, and nothing noticed.
Both are scripts now, test/vm-image/nightly-plan and nightly-layout,
and test/unit/nightly-test.sh tries them without a runner: a first run,
a run started again, a release already published, a tag that is not a
version; every field of latest.json for each channel, the checksums
against the files, the version a consumer reads from the ISO's name.
nightly-layout also refuses to describe an image it cannot describe
fully, and checks its own latest.json before anything is published.
The same test covers two things the install test decides: which Omarchy
packages an ISO installs, and when an install is over.
On edge, pacman answers for both omarchy and omarchy-dev with the one
installed package, so the version and the commit were each written
twice, with a line break between.
Starting a run again keeps its run number, so a second attempt would
have built new images for the paths the first attempt had already
partly filled, and the upload refuses to replace a file. From the second
attempt on, the attempt is part of the number: 431, then 431.2.
The same runner the nightly uses. Its kernel has no Landlock, so the
test's container steps around pacman's sandbox the way the ISO build
does. It gets no cache volume: this workflow runs for pull requests, and
a cache one of them wrote would be trusted by the next run.
The wait loop printed "installing (Ns)" until SSH answered on the
installed system. A nightly run sat on that line for twelve minutes
while the install had finished after two: the first boot was what hung.
The firmware names what it starts on the serial port, so the loop now
notices the installed disk's Limine starting, logs how long the install
took, and counts the first boot from there.
Next to edge, the nightly now makes a VM image of the last release. The
latest release of omacom/omarchy names the version, iso.omarchy.org has
the ISO, and it is used only if its signature is the release key's, the
one in builder/omarchy.gpg. It gets the same install test, seal and
checks, and is published under stable/ with its own latest.json, whose
url is the ISO's existing place: the ISO is not uploaded again.
A release is done once. A plan job decides what a run makes: edge
always, stable unless a scheduled run finds that version already in
stable/latest.json. Build and publish run once per channel, and one
channel failing does not hold back the other.
The install test's key sits in the user's home, which is its own
subvolume and not part of the factory snapshot, and the seal removes it
there. The check says so of every image, instead of leaving it to the
layout staying as it is.
The seal read /boot/EFI/Linux/omarchy_linux-omarchy.efi. An edge install
has linux-omarchy-bore, so the first nightly run stopped there. It now
asks the running system which kernel package it booted and takes that
UKI, and takes the command line from /proc/cmdline, since the boot
entry, not the UKI, may be what holds it. The nightly's package report
names the kernel the same way.
linux-t2, the kernel every x86 live ISO boots, comes from arch-mact2,
which has two servers in these configs: the maintainer's host in Seoul
and their GitHub release. Both carry the same database. An ISO build on a
runner in Virginia waited on the Seoul host at 0.45 MB/s for a 92 MB
package; the same file from GitHub came at 58 MB/s.
GitHub goes first, which is also the order of arch-mact2's own mirror
list. The Seoul host stays as the fallback.
A nightly downloads the packages of the ISO's offline mirror, a few
gigabytes, and the test container's own, and would download most of them
again the next night. On Namespace both directories live on the runner
profile's cache volume. The build already prunes from its cache what an
ISO no longer needs. The step is skipped, not failed, on a runner
without a cache volume.
repo-add takes one package at a time: it hashes the archive and lists its
files, which means decompressing all of it, for about a thousand
packages one after another.
A database is a tar of one directory per package. The packages are dealt
out to one repo-add per CPU, each writing its own database, and the
pieces are joined. On 160 packages on a 16-core machine the result is
identical to one repo-add's, db and files, in 1.0 s against 6.3 s. Two
versions of one package name are an error: one repo-add would keep the
last, the pieces apart would keep both.
GitHub's hosted runner has 4 cores. The build job moves to
namespace-profile-linux-amd64-16x32; the publish job, which only copies
files and holds the bucket's key, stays where it is.
Those runners' kernel has no Landlock, so pacman 7 fails its first sync
in a container there. The ISO build already steps around that; the
install test's container does the same now. The job also prints what it
is running on: CPU, memory, LSMs and /dev/kvm.
pacman 7 sandboxes its downloads with Landlock and fails every database
sync on a kernel without it, which Namespace's AmpereOne arm64 machines
run: "restricting filesystem access failed because Landlock is not
supported by the kernel!". An ISO build there stops at its first
pacman -Sy.
Where /sys/kernel/security/lsm lists no landlock, or cannot be read, a
wrapper in /usr/local/bin runs every pacman in the build container,
pacstrap's included, with --disable-sandbox, the flag pacman provides for
such kernels. Not DisableSandbox in a config: pacman-offline.conf ships
on the ISO.
The nightly built the stable ISO and kept it as an artifact for a day.
It now builds the edge ISO, runs the install test on it, seals the
installed disk that test leaves behind into a VM image, checks the
image, and publishes ISO and image under a version that is never reused:
nightly.omarchy.org/edge/latest.json
nightly.omarchy.org/edge/<version>/omarchy-edge-<version>.iso
nightly.omarchy.org/edge/<version>/vm/
latest.json gives the version, the ISO's URL and checksum, the
omarchy-iso commit, the Omarchy version and commit inside, and where the
image is. It is written last, after every file has been fetched back
through the CDN and compared.
Publishing is a second job, the only one with the bucket's key, and only
on quattro. There is no pull_request trigger. A manual run on a branch
builds and checks everything and publishes nothing.
test/integration already installs the ISO unattended and saves the
installed disk as test-runs/<iso>-integration/base.qcow2, with its
firmware variables and the key that logs in to it. That folder is what
vm-seal takes, so the separate install script goes.
The sealed image now also carries the firmware variables, which hold the
boot entry the installer registered: a consumer that boots the disk
through UEFI needs them, a direct kernel boot does not.
builder/build-iso.sh puts omarchy-dev and omarchy-settings-dev on an edge
ISO, as it does on a dev or local one. The harness picked its package
names from the ISO's file name and knew dev, local and pr, so for an edge
ISO it asked the installer for the release packages.
As they were tested by hand on the 4.0.4 ISO, before any of it is wired
into a workflow:
vm-install unattended install of an ISO into a qcow2, through cidata
vm-seal makes the disk publishable: SSH takes keys only, no machine
ID, host keys, build key or build logs left, compressed
with zstd, with checksums and the kernel pulled out for
direct boot
vm-verify boots the sealed image once and checks all of that
vm-boot boots a throwaway copy with a key made for that boot and
runs a command in it
On this machine: install 114 s, seal 27 s, a 4.2 GB image, SSH about 5 s
after power-on, 17 of 17 verify checks, and 7 of them failing on an
image sealed without these fixes.
Emir Beganovic