rclone reads every RCLONE_<FLAG> variable as that flag. The step that
installs it set RCLONE_VERSION=v1.75.2 for the download, so its own
"rclone version" ran as "--version=v1.75.2" and failed:
Invalid value when setting --version from environment variable
RCLONE_VERSION="v1.75.2": invalid argument for "--version" flag
Both publish jobs stopped there, before uploading, in every run since the
pin was added. Name the two variables PINNED_RCLONE and
PINNED_RCLONE_SHA256, and have the nightly test refuse any RCLONE_ variable
in a workflow other than the remote's RCLONE_CONFIG_ settings.
A unit test run stopped before it began: Docker Hub refused to hand out
archlinux:latest, "You have reached your unauthenticated pull rate
limit". GitHub's runners share addresses, so the limit is not ours to
stay under.
Arch publishes the same image as ghcr.io/archlinux/archlinux, which has
no such limit from a runner. The three workflow containers and the ISO
build's own container pull it from there.
The warning was written for the bulk upload only, and counted lines:
latest.json is uploaded by a second rclone call, which could still pass
on a retry unseen, and a file that failed on two attempts was counted as
two uploads.
Both calls go through test/vm-image/nightly-rclone, which passes
rclone's output and exit status through and names each file once. A
stand-in rclone tests it.
The four jobs on GitHub's own runners move from ubuntu-24.04 and
ubuntu-latest to ubuntu-26.04.
On 26.04 /tmp is held in memory, so the publish job's artifact, 11 GB for
edge, goes to the runner's temp directory on disk instead. nightly-fetch
now checks there is room before it starts and says how much is missing.
Ubuntu's rclone is 1.60.1, from 2022. In the first publish R2 answered
every upload of the first attempt "501 Not Implemented", having stored
the file each time, and the job passed because rclone's second attempt
found everything in place.
Run against a stand-in server, 1.60.1 and the current 1.75.2 make the
same requests but for one header: 1.60.1 sends "x-amz-acl: private"
with each upload and 1.75.2 sends none to Cloudflare. The publish job
now installs 1.75.2, by version and checksum.
Should an upload pass only on a retry again, the run says so in a
warning.
The first publish spent 9 min 42 s in the download action for the 4.6 GB
stable release; the 11.5 GB edge release would not have fitted in the
job's 45 minutes. GitHub's artifact storage limits each connection to a
few megabytes a second, and the action uses few.
An artifact is one zip at a signed address. test/vm-image/nightly-fetch
asks the API for that address and takes it with aria2c over 16
connections, checks the size and unpacks it. The same 4.6 GB artifact
came down in 123 s that way. The publish job gets actions: read for it,
and checks out test/vm-image for the script.
test/unit/nightly-test.sh plays the network with stand-ins: the newest
artifact of a name is the one fetched, a failed download is tried again
with a new address, a wrong size is refused and not unpacked.
actions/checkout v5.0.0 -> v7.0.1
actions/upload-artifact v6.0.0 -> v7.0.2
actions/download-artifact v4.3.0 -> v8.0.2
docker/setup-buildx-action v4.3.0 -> v4.4.1
Each stays pinned to a commit, with the version in its comment.
From the release notes, what the new majors change and why none of it
touches these workflows: download-artifact v5 moved the path of an
artifact fetched by ID, and ours is fetched by name; v8 fails on a
checksum mismatch where v4 said nothing, which is what a publish job
wants. checkout v7 refuses a fork's code under pull_request_target and
workflow_run, and the one workflow_run here checks nothing out. The
Node 24 releases need runner 2.327.1; ours run 2.337.0.
Pinned actions do not update themselves, so .github/dependabot.yml asks
for one grouped pull request a week.
An install test failed 13 seconds in: a public Arch mirror dropped two
signature files while the test container was installing QEMU. The three
test containers now ask mirror.omarchy.org first, with the public
mirrors after it, and try up to three times.
The image was a folder, vm/, holding base.qcow2 and what boots it. It is
now one file next to the ISO, with the ISO's name and its own checksum
file, the way Ubuntu, Debian, Arch, Fedora and CentOS publish theirs:
edge/<version>/omarchy-edge-<version>.qcow2
stable/<version>/<run>/omarchy-<version>.qcow2
The image boots by itself through UEFI. What a direct boot needs, the
kernel, initramfs, kernel arguments and vm-boot, is in an unpacked/
folder beside it, Ubuntu's name for the same thing. The firmware
variables are no longer published: a boot with the firmware's defaults
finds the disk.
latest.json follows: vm.url, vm.sha256 and vm.size are the image, as
url, sha256 and size are the ISO; vm.unpacked and vm.unpacked_files are
the folder. vm-boot takes the image file. The publish job compares every
file it uploaded, whatever the layout. README, tests and the workflow's
header say the same.
The top is for someone who wants Omarchy: where to get the ISO and how
to check it, how to try a VM image, and autoinstall. Everything about
building, testing and publishing follows under "For maintainers", with
what is published where, how the nightly works, the fields of
latest.json and the files of a VM image.
A read against the code corrected several statements, some of them
older: the installer runs omarchy-provision-user; a defer-provisioning
marker can stand in for the credentials file; an authorized_keys file
fails the install when it holds no keys, not when its keys are bad; the
stock firewall also allows Docker's DNS; built_at is when latest.json
was written; vm-boot's needs are listed in full.
The publish job now fetches the two checksum files back as well, so
"every file of the version is compared" is exact.
The cache kept the ISO's packages between runs. A package that has gone
missing from the mirror would then still be found on the runner, and the
nightly would build and pass while a fresh machine could not. Every
package is fetched again each night: with the T2 repository asked on
GitHub first, the build is 3 to 4 minutes without a cache.
Which channels a run makes and where they go, and what is laid out and
what latest.json then says, were shell inside the workflow file, tried
only by hand. A green run published a latest.json with the Omarchy
version in it twice, and nothing noticed.
Both are scripts now, test/vm-image/nightly-plan and nightly-layout,
and test/unit/nightly-test.sh tries them without a runner: a first run,
a run started again, a release already published, a tag that is not a
version; every field of latest.json for each channel, the checksums
against the files, the version a consumer reads from the ISO's name.
nightly-layout also refuses to describe an image it cannot describe
fully, and checks its own latest.json before anything is published.
The same test covers two things the install test decides: which Omarchy
packages an ISO installs, and when an install is over.
On edge, pacman answers for both omarchy and omarchy-dev with the one
installed package, so the version and the commit were each written
twice, with a line break between.
Starting a run again keeps its run number, so a second attempt would
have built new images for the paths the first attempt had already
partly filled, and the upload refuses to replace a file. From the second
attempt on, the attempt is part of the number: 431, then 431.2.
Next to edge, the nightly now makes a VM image of the last release. The
latest release of omacom/omarchy names the version, iso.omarchy.org has
the ISO, and it is used only if its signature is the release key's, the
one in builder/omarchy.gpg. It gets the same install test, seal and
checks, and is published under stable/ with its own latest.json, whose
url is the ISO's existing place: the ISO is not uploaded again.
A release is done once. A plan job decides what a run makes: edge
always, stable unless a scheduled run finds that version already in
stable/latest.json. Build and publish run once per channel, and one
channel failing does not hold back the other.
The seal read /boot/EFI/Linux/omarchy_linux-omarchy.efi. An edge install
has linux-omarchy-bore, so the first nightly run stopped there. It now
asks the running system which kernel package it booted and takes that
UKI, and takes the command line from /proc/cmdline, since the boot
entry, not the UKI, may be what holds it. The nightly's package report
names the kernel the same way.
A nightly downloads the packages of the ISO's offline mirror, a few
gigabytes, and the test container's own, and would download most of them
again the next night. On Namespace both directories live on the runner
profile's cache volume. The build already prunes from its cache what an
ISO no longer needs. The step is skipped, not failed, on a runner
without a cache volume.
GitHub's hosted runner has 4 cores. The build job moves to
namespace-profile-linux-amd64-16x32; the publish job, which only copies
files and holds the bucket's key, stays where it is.
Those runners' kernel has no Landlock, so pacman 7 fails its first sync
in a container there. The ISO build already steps around that; the
install test's container does the same now. The job also prints what it
is running on: CPU, memory, LSMs and /dev/kvm.
The nightly built the stable ISO and kept it as an artifact for a day.
It now builds the edge ISO, runs the install test on it, seals the
installed disk that test leaves behind into a VM image, checks the
image, and publishes ISO and image under a version that is never reused:
nightly.omarchy.org/edge/latest.json
nightly.omarchy.org/edge/<version>/omarchy-edge-<version>.iso
nightly.omarchy.org/edge/<version>/vm/
latest.json gives the version, the ISO's URL and checksum, the
omarchy-iso commit, the Omarchy version and commit inside, and where the
image is. It is written last, after every file has been fetched back
through the CDN and compared.
Publishing is a second job, the only one with the bucket's key, and only
on quattro. There is no pull_request trigger. A manual run on a branch
builds and checks everything and publishes nothing.
The workflow has been running ./build_iso.sh, which has not existed for a long
time — the build moved to bin/omarchy-iso-make. Every scheduled run in the last
100 has failed.
Run the real build driver instead, with NO_BOOT_OFFER=1 so it skips the
interactive "Boot this ISO?" prompt at the end, and collect from ./release/
where the ISO actually lands. Drop the outer archlinux container: omarchy-iso-make
starts its own privileged container, so the job only needs Docker on the runner.
Add a disk cleanup step, since a ~6.5GB ISO plus a ~4.5GB offline mirror does not
fit in a stock runner's free space.
Taken from #67 by @karthikkalidas, which was opened against main. main is 143
commits behind and is no longer the default branch, and scheduled workflows only
run from the default branch, so merging it there would have fixed nothing.
Co-Authored-By: Karthik Kalidas <karthik.kalidas@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Emir Beganovic