The cache kept the ISO's packages between runs. A package that has gone
missing from the mirror would then still be found on the runner, and the
nightly would build and pass while a fresh machine could not. Every
package is fetched again each night: with the T2 repository asked on
GitHub first, the build is 3 to 4 minutes without a cache.
Which channels a run makes and where they go, and what is laid out and
what latest.json then says, were shell inside the workflow file, tried
only by hand. A green run published a latest.json with the Omarchy
version in it twice, and nothing noticed.
Both are scripts now, test/vm-image/nightly-plan and nightly-layout,
and test/unit/nightly-test.sh tries them without a runner: a first run,
a run started again, a release already published, a tag that is not a
version; every field of latest.json for each channel, the checksums
against the files, the version a consumer reads from the ISO's name.
nightly-layout also refuses to describe an image it cannot describe
fully, and checks its own latest.json before anything is published.
The same test covers two things the install test decides: which Omarchy
packages an ISO installs, and when an install is over.
On edge, pacman answers for both omarchy and omarchy-dev with the one
installed package, so the version and the commit were each written
twice, with a line break between.
Starting a run again keeps its run number, so a second attempt would
have built new images for the paths the first attempt had already
partly filled, and the upload refuses to replace a file. From the second
attempt on, the attempt is part of the number: 431, then 431.2.
Next to edge, the nightly now makes a VM image of the last release. The
latest release of omacom/omarchy names the version, iso.omarchy.org has
the ISO, and it is used only if its signature is the release key's, the
one in builder/omarchy.gpg. It gets the same install test, seal and
checks, and is published under stable/ with its own latest.json, whose
url is the ISO's existing place: the ISO is not uploaded again.
A release is done once. A plan job decides what a run makes: edge
always, stable unless a scheduled run finds that version already in
stable/latest.json. Build and publish run once per channel, and one
channel failing does not hold back the other.
The seal read /boot/EFI/Linux/omarchy_linux-omarchy.efi. An edge install
has linux-omarchy-bore, so the first nightly run stopped there. It now
asks the running system which kernel package it booted and takes that
UKI, and takes the command line from /proc/cmdline, since the boot
entry, not the UKI, may be what holds it. The nightly's package report
names the kernel the same way.
A nightly downloads the packages of the ISO's offline mirror, a few
gigabytes, and the test container's own, and would download most of them
again the next night. On Namespace both directories live on the runner
profile's cache volume. The build already prunes from its cache what an
ISO no longer needs. The step is skipped, not failed, on a runner
without a cache volume.
GitHub's hosted runner has 4 cores. The build job moves to
namespace-profile-linux-amd64-16x32; the publish job, which only copies
files and holds the bucket's key, stays where it is.
Those runners' kernel has no Landlock, so pacman 7 fails its first sync
in a container there. The ISO build already steps around that; the
install test's container does the same now. The job also prints what it
is running on: CPU, memory, LSMs and /dev/kvm.
The nightly built the stable ISO and kept it as an artifact for a day.
It now builds the edge ISO, runs the install test on it, seals the
installed disk that test leaves behind into a VM image, checks the
image, and publishes ISO and image under a version that is never reused:
nightly.omarchy.org/edge/latest.json
nightly.omarchy.org/edge/<version>/omarchy-edge-<version>.iso
nightly.omarchy.org/edge/<version>/vm/
latest.json gives the version, the ISO's URL and checksum, the
omarchy-iso commit, the Omarchy version and commit inside, and where the
image is. It is written last, after every file has been fetched back
through the CDN and compared.
Publishing is a second job, the only one with the bucket's key, and only
on quattro. There is no pull_request trigger. A manual run on a branch
builds and checks everything and publishes nothing.
The workflow has been running ./build_iso.sh, which has not existed for a long
time — the build moved to bin/omarchy-iso-make. Every scheduled run in the last
100 has failed.
Run the real build driver instead, with NO_BOOT_OFFER=1 so it skips the
interactive "Boot this ISO?" prompt at the end, and collect from ./release/
where the ISO actually lands. Drop the outer archlinux container: omarchy-iso-make
starts its own privileged container, so the job only needs Docker on the runner.
Add a disk cleanup step, since a ~6.5GB ISO plus a ~4.5GB offline mirror does not
fit in a stock runner's free space.
Taken from #67 by @karthikkalidas, which was opened against main. main is 143
commits behind and is no longer the default branch, and scheduled workflows only
run from the default branch, so merging it there would have fixed nothing.
Co-Authored-By: Karthik Kalidas <karthik.kalidas@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Emir Beganovic