Sections that still described the old two-channel, host-only world:
- command scope: the host-tree commands forward over ssh now, and advance
replaces the deprecated migrate
- global flags: --mirror takes edge|rc|stable, and --local exists
- build examples and directory structure: the rc channel and the release lock
- release section: omarchy-release is the front door; omarchy-pkgs is the pin
engine it drives (and still usable directly)
- management: bin/repo timers, the backoff state files, and clearing a stale
lock, instead of raw systemctl invocations that omitted the rc units
- a leftover reference to the 6-hourly timer as the trigger backstop
Also fixed an anchor that pointed at the section containing the link rather
than the Quick Start section it names.
Matches what is already published to edge: 0.0.34 was built on the server
before the bump reached git, so the checkout read as out of date and queued a
rebuild of the older 0.0.33-2 that promotion then refused. Produced by
bin/sync-upstream, the same path the scheduled workflow uses.
Reporting to BASECAMP_CHATBOT_URL is a working setup — the second variable
exists only for those who want release traffic in its own chat. setup now
states which chat receives reports instead of warning about the common case,
and the README frames the split as optional rather than expected.
Two failures from the first live run:
notify_basecamp declared 'local BASECAMP_CHATBOT_URL' and then called
release_chatbot_url, whose fallback reads that same global — bash locals are
visible to called functions, so the fallback saw the empty local and every
notification silently went nowhere for anyone with only the legacy variable
set. The local is now named 'url'.
check-versions compared PKGBUILD and published versions with !=, so a checkout
BEHIND the channel queued a rebuild of an older version every cycle: the
builder produced it and promotion refused it, because that exact filename is
already published with different bytes. It now skips (with a warning naming
the package) when an artifact for the PKGBUILD's version already exists in the
channel, whichever direction the versions differ.
The rc service fetched and reset /root/omarchy-pkgs-rc in ExecStartPre, but
that worktree does not exist until the first RC creates the rc branch — so on
a freshly set up host the unit failed every five minutes, forever, and showed
up as a failed unit in the timer report.
It now runs bin/auto-release-rc from the main checkout, which always exists:
nothing queued exits silently, no rc branch is a clean no-op, and a missing
worktree is created on demand before handing off to the worktree's own
auto-release.
A bad in-place edit truncated the file to zero: open(p,'w') ran before
open(p).read(), so the read saw an already-empty file. Restored from c0a63dd
and reapplied the two intended changes — the start report now naming queued
packages, and the nothing-to-publish report described as the check/builder
disagreement signal it is.
Restores the no-change report now that it is clear it cannot fire on an idle
timer tick: releases only run when the version check queued work, so a run
that publishes nothing means check-versions and the builder disagree about
what is out of date. The report names the packages that were queued but never
built, so a recurring disagreement is diagnosable rather than invisible.
A release that published nothing is not news, and at a 5-minute cadence those
messages would bury the ones that matter — the log and bin/repo timers still
show the run happened.
Removing it would have left a start report with no follow-up, so the start
report now carries its own answer: check-versions writes the package names it
queued into the state file instead of touching an empty one, and the release
run reads them. 'A build is running' becomes 'your package is in this build',
which is the question the reports exist to answer.
Only failures were reported, so a push could reach the mirror with no way to
know short of querying the database by hand. Release runs now report:
- start: channel, arch, host, and the commit being built
- published: the packages and versions that went out, duration, channel URL
- no-changes: the run found nothing to build
- promoted: what advance moved between channels, including the rc bootstrap
and fast-ring replication
- failed: unchanged, plus the commit context the other reports carry
Release traffic goes to OMARCHY_RELEASE_CHATBOT_URL, falling back to
BASECAMP_CHATBOT_URL, so build reports stop drowning the repository chat the
sync workflows post to. bin/setup reports which destination is configured.
The published list is captured after the build step because promote moves the
files out of build-output, and is capped at 25 entries so a full rebuild does
not produce an unreadable wall of chat.
A push reaching the mirror should take minutes, not up to six hours. All four
units now fire every 5 minutes, staggered a minute apart. Three guards make
that cadence safe:
- Scheduled runs take the release lock NON-BLOCKING (try_release_lock) and
skip the tick when a build is running. Blocking would stack one stalled
process per tick behind a long build and stampede when it finished. Manual
commands still wait, as an operator expects.
- check-versions takes the lock too, and now owns its git pull (--pull, passed
by the unit) instead of an ExecStartPre: at this cadence an unlocked pull
would swap PKGBUILDs out from under a running build.
- A failed release records .build-failed-<channel> and backs off
exponentially (10m, 20m, 40m … capped at 6h) rather than rebuilding the same
broken tree every 5 minutes. Any new commit clears the backoff, since a push
is the most likely fix.
Idle ticks exit without output so the journal keeps showing the runs that
matter, and bin/repo timers reports backoff state — a paused channel is
otherwise indistinguishable from an idle one.
Wraps systemctl list-timers with the things you actually want when checking on
the build host: per-unit enabled state, last run and whether it succeeded,
which channels have builds queued (state files), whether the release lock is
held by a live process, and any failed units. Units are discovered from
systemd/*.timer so the report cannot drift from what setup installs.
It forwards over ssh like the other host commands, so the build box's timer
state is one command away from a workstation (--local to inspect this machine).
--check warned 'rc worktree would be created' whenever the directory was
absent, implying a plain setup run would create it — but with no rc branch in
existence setup skips it, so the warning described something that would not
happen and asked for action that was not possible. It now reports the branch
state: present, would-create (branch exists), or nothing-to-do (no branch
yet — the first RC cut creates the branch and the build trigger creates the
worktree on demand). Branch detection is read-only, as --check must be.
Eligibility used package_moves_to_channel, which excludes packages built
natively in the destination — right for ongoing edge -> rc advances (a native
build must not be raced under the same filename) but wrong for the bootstrap,
whose entire purpose is rc == stable. omarchy and omarchy-settings were
therefore left out, so a machine switched to the rc channel could not install
or update the release pair until the first RC was cut. The bootstrap now
requires only destination membership; nothing is built in rc yet, so there is
no native artifact to conflict with. The dev pair stays edge-only and
fast-ring replication is unchanged.
The builder stage never declared ARG MIRROR, so the keyring [omarchy] repo
pointed at the channel-less legacy pkgs.omarchy.org/$arch path — it works
only because a stale copy of the old layout still answers there, and it would
miss a keyring rotation. Each image now pulls omarchy-keyring from its own
channel (edge/rc/stable), matching the base mirror it already selects.
update-repo and remove-package switch to the edge x86_64 image: repo-add and
repo-remove compile nothing, and using the channel image would deadlock
bootstrap-rc — the rc image can only build once the rc channel it pulls the
keyring from exists remotely.
With a repository host configured (OMARCHY_REPO_HOST / .repo-host), release,
build, sign, promote, update, clean, advance, bootstrap-rc, remove, sync, and
migrate exec on the host over ssh — same code, run where the published tree
lives, after sourcing the host credentials and a --ff-only pull. --local
forces local execution. list/push/deploy/setup never forward. The host itself
has no .repo-host, so ssh'd-in manual use is unchanged. omarchy-release's
advance now rides the same forwarding (one code path), and --host exports
OMARCHY_REPO_HOST so child bin/repo calls follow it.
This closes the gap where bootstrap-rc ran against a workstation's stale
local tree despite .repo-host being set.
The published-db marker also exists on any workstation that once ran a full
local release, so --host / OMARCHY_REPO_HOST / .repo-host are now checked
before on_repo_host everywhere (triggers, advances, doctor). README documents
the detection, the caveat, and the .repo-host tie-breaker.
Release commands now work from anywhere: on_repo_host (the published database
living in this checkout) routes build triggers, advances, and promotion to
local execution; other machines go over ssh to the configured destination.
The host setting is any ssh destination — root@<ip>, root@<hostname>, or an
~/.ssh/config alias — resolved from --host, OMARCHY_REPO_HOST, then the
one-line .repo-host file. doctor reports which mode applies, and the README
documents the format and precedence. All host connections are plain ssh.
The work/mirror clones were HTTPS end to end, so pushes went through git's
credential-helper config — which breaks the moment a stale absolute gh path
is baked into it (as gh auth setup-git once did with /usr/bin/gh). Reads stay
anonymous HTTPS; pushes now use an SSH push URL (derived from the clone URL,
overridable with OMARCHY_UPSTREAM_PUSH_URL), set idempotently on every run so
existing cached clones self-repair.
Changes reach a release branch as backports — cherry-picks with new SHAs — so
the original quattro merge commit is never an ancestor of a patch branch and
the ancestry filter let already-applied PRs through. Candidates are now also
matched against the branch's own commit messages since it left quattro:
'backport of #N' / squash '(#N)' references and 'cherry picked from commit
<sha>' trailers (which pick -x itself writes). Explicitly named PRs/commits
that are already on the branch are skipped with a note instead of re-picked.
Verified against the live v4-0-2 branch: the five backported PRs it carries
filter out; un-backported ones are still offered.
- ship: no interactive override of the untested-commit guard; the tag targets
the pinned commit the artifacts were built from (never the branch head); a
tagged-but-incomplete train is found and resumed instead of vanishing from
open-train detection; a fully shipped train reports as such
- start: a failed edge→rc advance fails the command loudly (both start and
the advance are idempotent) instead of opening a train against stale rc
- rc trigger: bootstraps the server's rc worktree on first use, so a host set
up before the rc branch existed can run its first RC build
- advance-channel: fast-ring packages are excluded from edge→rc (the stable
build replicated by parity is authoritative for rc — same filename, other
bytes); differing destination bytes abort instead of warn; a package whose
signature copy was interrupted gets its .sig restored on resume
- the release lock now also covers direct promote/update/clean/remove/sync
invocations, not just release/advance/upload-prebuilt
A release train has three human moments, each one command: start (release
branch on basecamp/omarchy + notes staging PR + edge→rc advance for
minor/major), rc (pin both PKGBUILDs to the branch head as X.Y.ZrcN on the
pkgs rc branch, trigger the rc channel build, wait for publish, optional RC
ISO), and ship (final pins → promote rc→stable → tag → pins to master → GitHub
release from the staging PR body → final ISO → website bump, each step
skip-if-done so a crashed run resumes).
Bare omarchy-release is the shepherd: it derives the train state from observed
reality (remote branches, rc-branch pins, published channel dbs, tags — no
state files) and offers the correct next step. Versions are inferred from
branch names (v4-0-2 ⇒ 4.0.2rcN ⇒ v4.0.2). ship refuses to promote a commit
no RC was cut from. pick is a multi-select over merged quattro PRs,
cherry-picking merge commits. doctor pre-flights every credential and
connection. self-test wired into CI.
bin/omarchy-pkgs stays as the pin engine, driven with its db URL pointed at
the rc channel and pins committed to the standing rc branch (rebuilt as
master + pins per cut and force-pushed; the server rc worktree follows with
reset --hard).
The rc service builds from the rc branch worktree (/root/omarchy-pkgs-rc,
created by bin/setup) but publishes into the primary checkout's channel tree
via OMARCHY_REPO_ROOT. The timer is a retry backstop: rc builds are normally
triggered immediately over SSH by the release orchestrator.
bin/repo advance --from/--to moves packages forward through the pipeline
(edge → rc → stable), driven by the source channel's database rather than the
raw directory, copying packages AND their detached signatures (fixing the old
migrate bug that left promoted packages unverifiable), refusing to rewrite any
published filename, and requiring a .sig for everything it moves. stable → rc
is allowed only as --fast-ring parity replication or the one-time
--bootstrap seed (bin/repo bootstrap-rc). 'migrate' stays as a deprecated
alias for the transition.
helpers/lock-helpers.sh adds a host-wide flock shared by bin/release,
advance-channel, and upload-prebuilt (reentrant via OMARCHY_RELEASE_LOCK_HELD)
so timers and operators serialize instead of interleaving partial publishes.
bin/release gains a stable-only step 7: replicate fast-ring artifacts to rc so
rc and stable stay in parity between release trains (skipped until rc is
bootstrapped).
A package's .omarchy/package.json may now pin where it lives with
"channels": [...]. With the key present the package builds in each listed
channel except stable (stable is only fed by promotion); without it, today's
defaults hold (build for edge; fast-ring also builds stable directly).
package_moves_to_channel() is the advance/promote eligibility rule: a member
of the destination channel that is not built there natively.
The release pair (omarchy, omarchy-settings) is edge+rc+stable — edge stays
during the client-migration overlap window and drops later. The dev pair is
pinned to edge only.
- helpers/paths.sh: validate_mirror/require_valid_mirror for the edge|rc|stable
set, and REPO_ROOT (OMARCHY_REPO_ROOT override) so a secondary checkout like
the rc branch worktree publishes into the same channel tree as the primary
- validate --mirror everywhere it previously accepted any string (sync-repo,
promote-build, update-repo, clean-repo, remove-package) and widen the
edge|stable checks in build, deploy, push-build, auto-release
- build/Dockerfile: rc builds compile against rc-mirror.omarchy.org
GNU date accepts relative expressions like '2 days ago', which would let a
buggy hook fabricate a release age; the backstop now insists on an ISO 8601
timestamp before date parses it. The provider header now states, rather than
contradicts, the code's behavior for a feed with no stable releases: that is
a loud failure by design, while quarantined releases report no update.
