Reporting to BASECAMP_CHATBOT_URL is a working setup — the second variable
exists only for those who want release traffic in its own chat. setup now
states which chat receives reports instead of warning about the common case,
and the README frames the split as optional rather than expected.
The rc service fetched and reset /root/omarchy-pkgs-rc in ExecStartPre, but
that worktree does not exist until the first RC creates the rc branch — so on
a freshly set up host the unit failed every five minutes, forever, and showed
up as a failed unit in the timer report.
It now runs bin/auto-release-rc from the main checkout, which always exists:
nothing queued exits silently, no rc branch is a clean no-op, and a missing
worktree is created on demand before handing off to the worktree's own
auto-release.
A bad in-place edit truncated the file to zero: open(p,'w') ran before
open(p).read(), so the read saw an already-empty file. Restored from c0a63dd
and reapplied the two intended changes — the start report now naming queued
packages, and the nothing-to-publish report described as the check/builder
disagreement signal it is.
A release that published nothing is not news, and at a 5-minute cadence those
messages would bury the ones that matter — the log and bin/repo timers still
show the run happened.
Removing it would have left a start report with no follow-up, so the start
report now carries its own answer: check-versions writes the package names it
queued into the state file instead of touching an empty one, and the release
run reads them. 'A build is running' becomes 'your package is in this build',
which is the question the reports exist to answer.
Only failures were reported, so a push could reach the mirror with no way to
know short of querying the database by hand. Release runs now report:
- start: channel, arch, host, and the commit being built
- published: the packages and versions that went out, duration, channel URL
- no-changes: the run found nothing to build
- promoted: what advance moved between channels, including the rc bootstrap
and fast-ring replication
- failed: unchanged, plus the commit context the other reports carry
Release traffic goes to OMARCHY_RELEASE_CHATBOT_URL, falling back to
BASECAMP_CHATBOT_URL, so build reports stop drowning the repository chat the
sync workflows post to. bin/setup reports which destination is configured.
The published list is captured after the build step because promote moves the
files out of build-output, and is capped at 25 entries so a full rebuild does
not produce an unreadable wall of chat.
A push reaching the mirror should take minutes, not up to six hours. All four
units now fire every 5 minutes, staggered a minute apart. Three guards make
that cadence safe:
- Scheduled runs take the release lock NON-BLOCKING (try_release_lock) and
skip the tick when a build is running. Blocking would stack one stalled
process per tick behind a long build and stampede when it finished. Manual
commands still wait, as an operator expects.
- check-versions takes the lock too, and now owns its git pull (--pull, passed
by the unit) instead of an ExecStartPre: at this cadence an unlocked pull
would swap PKGBUILDs out from under a running build.
- A failed release records .build-failed-<channel> and backs off
exponentially (10m, 20m, 40m … capped at 6h) rather than rebuilding the same
broken tree every 5 minutes. Any new commit clears the backoff, since a push
is the most likely fix.
Idle ticks exit without output so the journal keeps showing the runs that
matter, and bin/repo timers reports backoff state — a paused channel is
otherwise indistinguishable from an idle one.
Wraps systemctl list-timers with the things you actually want when checking on
the build host: per-unit enabled state, last run and whether it succeeded,
which channels have builds queued (state files), whether the release lock is
held by a live process, and any failed units. Units are discovered from
systemd/*.timer so the report cannot drift from what setup installs.
It forwards over ssh like the other host commands, so the build box's timer
state is one command away from a workstation (--local to inspect this machine).
With a repository host configured (OMARCHY_REPO_HOST / .repo-host), release,
build, sign, promote, update, clean, advance, bootstrap-rc, remove, sync, and
migrate exec on the host over ssh — same code, run where the published tree
lives, after sourcing the host credentials and a --ff-only pull. --local
forces local execution. list/push/deploy/setup never forward. The host itself
has no .repo-host, so ssh'd-in manual use is unchanged. omarchy-release's
advance now rides the same forwarding (one code path), and --host exports
OMARCHY_REPO_HOST so child bin/repo calls follow it.
This closes the gap where bootstrap-rc ran against a workstation's stale
local tree despite .repo-host being set.
The published-db marker also exists on any workstation that once ran a full
local release, so --host / OMARCHY_REPO_HOST / .repo-host are now checked
before on_repo_host everywhere (triggers, advances, doctor). README documents
the detection, the caveat, and the .repo-host tie-breaker.
Release commands now work from anywhere: on_repo_host (the published database
living in this checkout) routes build triggers, advances, and promotion to
local execution; other machines go over ssh to the configured destination.
The host setting is any ssh destination — root@<ip>, root@<hostname>, or an
~/.ssh/config alias — resolved from --host, OMARCHY_REPO_HOST, then the
one-line .repo-host file. doctor reports which mode applies, and the README
documents the format and precedence. All host connections are plain ssh.
A release train has three human moments, each one command: start (release
branch on basecamp/omarchy + notes staging PR + edge→rc advance for
minor/major), rc (pin both PKGBUILDs to the branch head as X.Y.ZrcN on the
pkgs rc branch, trigger the rc channel build, wait for publish, optional RC
ISO), and ship (final pins → promote rc→stable → tag → pins to master → GitHub
release from the staging PR body → final ISO → website bump, each step
skip-if-done so a crashed run resumes).
Bare omarchy-release is the shepherd: it derives the train state from observed
reality (remote branches, rc-branch pins, published channel dbs, tags — no
state files) and offers the correct next step. Versions are inferred from
branch names (v4-0-2 ⇒ 4.0.2rcN ⇒ v4.0.2). ship refuses to promote a commit
no RC was cut from. pick is a multi-select over merged quattro PRs,
cherry-picking merge commits. doctor pre-flights every credential and
connection. self-test wired into CI.
bin/omarchy-pkgs stays as the pin engine, driven with its db URL pointed at
the rc channel and pins committed to the standing rc branch (rebuilt as
master + pins per cut and force-pushed; the server rc worktree follows with
reset --hard).
An empty min_release_age string now maps to unparseable rather than absent,
so "min_release_age": "" fails validation instead of silently running
with a zero-second quarantine. The end-to-end fixtures extend the
checked-in pkgver (.90/.91) so the test keeps working at any future mise
version. A Tests workflow runs bin/sync-upstream self-test and
bin/omarchy-pkgs self-test on every PR in the Arch container, making the
proof machine-checked instead of author-supplied. The README package
metadata field list documents upstream and min_release_age.
bin/sync-upstream self-test swaps the two network fetches in
helpers/upstream-github.sh for fixture readers and runs the production code
paths: fallback past a quarantined release, draft/prerelease filtering, the
deliberate bypass, unchanged-version and all-quarantined no-update paths,
unusable tags/timestamps and missing checksums failing the sync, {tag} and
{pkgver} asset templates with ./ and * manifest prefixes across both
architectures, the min_release_age backstop verdicts (now a testable
release_age_status function), the duration parser, and manifest validation.
Also fixes from the review: the duration parser forces base-10 arithmetic
(leading zeros no longer parse as octal) and bounds values to nine digits so
no suffix can overflow; jq // treating false as absent can no longer let
"min_release_age": false or "upstream": false slip through as unset; the
release feed page grew to the API maximum of 100 with the bounded search
documented; and the README package-metadata section documents the upstream
block, min_release_age, the bypass, and provider-versus-hook exclusivity.
A review at xhigh found several ways this command could report success while delivering nothing, which is the exact failure it exists to prevent.
A trigger named in rebuild_on but missing from rebuilt_against was never examined, because the comparison walked the record rather than the declared list. Adding a dependency to a package already opted in left that dependency untracked forever. The comparison now walks the declared triggers, so a name the record does not carry reads as changed.
That also retires the separate baseline path. Recording a package's triggers without bumping pkgrel certified a build nobody had checked: a package already broken by a release that moved before it opted in would be recorded as current and never rebuilt. Opting in now costs one rebuild, which is much the cheaper mistake.
A bumped version was only checked against the checked-in one. The floor is what users already have, so a checkout that had fallen behind the repository could be bumped to a version pacman orders below the package it means to replace, with the record advancing regardless. The published database is now the floor, and an unreadable one warns rather than blocks.
Metadata that did not parse dropped its package out of an unscoped run without a word, an unreadable rebuild_on being indistinguishable from an absent one. It is now reported and fails the run.
The workflow reads versions from mirror.omarchy.org, the mirror the x86_64 builder itself uses, rather than whichever mirror the container defaulted to. A mirror running ahead of the builder would record a version the build never linked against, and nothing re-fires once the record matches.
aarch64 stays uncovered and is documented as such: those builds resolve from Arch Linux ARM, one record cannot describe two architectures, and only x86_64 is published today.
bin/sync-rebuilds --self-test covers each of these against a throwaway repository root with pacman and curl stubbed.
🤖 Generated by Opus 5 in Claude Code. Reviewed by Codex XHigh.
Co-Authored-By: Codex XHigh <codex@openai.com>
A package that links Qt private API has to be rebuilt whenever qt6-base moves, because Qt_6_PRIVATE_API symbols are not covered by the soname and pacman upgrades Qt out from under the installed binary while the dependency stays unversioned. Nothing here noticed. Both version gates ask whether the package's own source moved, and for a VCS package pinned to a commit that answer stays no through every Qt release.
Unlocking the build gate would not have been enough on its own. A rebuild that reuses the published version string produces a package pacman never offers anyone, so the trigger has to edit git and bump pkgrel, which is why it sits beside sync-aur and sync-upstream rather than inside check-versions or the builder. Once pkgrel moves, both existing gates already do the right thing untouched.
Packages opt in with rebuild_on in .omarchy/package.json. bin/sync-rebuilds records what each was last bumped for in rebuilt_against and compares that to core, extra and multilib, ignoring testing and kde-unstable because those are not what the builder links against. A package with no record yet is only recorded, never bumped: what its published build linked against is not knowable from here, so the first run establishes the baseline. For an AUR-synced package the bump is written as the dotted Omarchy pkgrel suffix in the metadata as well, since the next sync replaces the PKGBUILD wholesale and would otherwise drop it.
🤖 Generated by Opus 5 in Claude Code.
Some vendors publish a release feed of their own that is faster and more
precise than anyone's packaging of it. A package opts in with an
.omarchy/upstream.sh hook that reports the newest release as JSON, and the
driver rewrites pkgver, the checksum arrays the hook names, and pkgrel.
Writes are guarded on both ends: every assignment the update will touch is
verified to exist before anything is written, so a hook naming an array the
PKGBUILD lacks fails with the file untouched rather than half rewritten; and
pkgver is held to pacman's character set, because it lands in a file makepkg
sources as shell.
Ordering is vercmp's, not sort -V's -- they disagree about whether 1.0a
precedes 1.0, and pacman is what decides if a published package is an upgrade.
That is also why the workflow runs in an Arch container rather than straight on
the runner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first version checked for pacman and refused anything else, so it would
have declined to run on the actual repository host. Nothing about that host
needs to be Arch: makepkg, repo-add and signing all happen inside containers.
Setup now detects apt or pacman and installs the right names for each --
bsdtar is libarchive-tools on Debian and libarchive on Arch. The requirement
list drops gnupg and the Arch build tools, which the host never runs directly,
leaving Docker, rclone, bsdtar, jq, git and rsync.
Docker is checked before being installed. A host may be running a version from
Docker's own repository, and replacing that underneath a working builder would
be a poor trade for consistency; setup starts it if stopped and otherwise
leaves it alone.
Verified both paths: apt installs the five dependencies and docker.io on a
bare Ubuntu 24.04 container, and an Arch host with Docker already running is
left untouched. A missing systemctl now reports that a container cannot be a
repository host instead of failing on an unknown command.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sync guard could not read the repository database because bsdtar was not
installed on the host, and the first fix was to parse around its absence. The
better answer is for the host to have what the tooling needs: libarchive ships
the library pacman links against without necessarily installing the binary, so
bsdtar being present was an assumption, not a fact.
bin/setup installs the dependencies, enables Docker, creates the state
directory, and installs and enables the release timers -- the steps the README
previously listed by hand. It is idempotent and takes --check to report without
changing anything. Signing credentials and the rclone remote hold secrets, so
it reports on those rather than creating them.
sync-repo goes back to reading the database with bsdtar alone, and says to run
bin/setup when it is missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--package meant a pkgbase to bin/build and a literal package name to
bin/push-build, so deploy --package nvidia-580xx-utils built three packages and
published one, leaving nvidia-580xx-dkms and opencl-nvidia-580xx behind with no
indication anything was missing. Hit while deploying exactly that package.
Selection now matches on the pkgbase recorded in .PKGINFO as well as on the
package name, so a pkgbase ships all of its outputs and an individual name
still selects just that one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bin/build asks the local repository database which packages are already built.
A build machine has no such database, so every package looks out of date: an
unscoped 'bin/repo deploy' on this laptop would have built all 108 packages and
published them. Verified with a dry run.
deploy now refuses to run unscoped when that database is absent, and push
refuses the same combination under --yes, where nobody would see the list it
prints before publishing. Both are allowed on the repository host, which has
the database that makes the comparison meaningful.
Also states the split in the README: build, push and deploy are the three
commands that may run off the repository host; everything else works on the
published tree directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The warning dated from when .build-host was the release trigger's own setting
and push was a second consumer of it. One setting now names one machine for all
three commands, so there is nothing to keep separate -- and the advice was
wrong regardless: OMARCHY_REPO_HOST is read by the same resolver, so it arms
the trigger exactly as the file does. Only --host is per-invocation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OMARCHY_BUILD_HOST and .build-host were carried as fallbacks through the
rename. Nothing in this checkout ever set either one, so they were a second
name for a setting that has only one real name.
Resolution is now --host, OMARCHY_REPO_HOST, .repo-host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Builds now happen wherever the operator likes, so naming the destination after
building described the old arrangement rather than the current one. What the
push and deploy commands reach is the machine that serves pkgs.omarchy.org and
holds the signing key: the repository host. It also runs the scheduled builds,
which is why the trigger in omarchy-pkgs release points at the same place.
Resolution moves into helpers/host-helpers.sh, which all three commands now
share instead of repeating: --host, then OMARCHY_REPO_HOST, then .repo-host.
OMARCHY_BUILD_HOST and .build-host keep working as fallbacks, so existing
environments and checkouts are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The worst was fatal: push passed --skip-prod-check to upload-prebuilt, which
forwards every argument to sign, promote and update as well, and sign rejects
unknown options. Every non-dry-run push and deploy would have uploaded and
verified its artifacts and then failed before signing. upload-prebuilt now
routes publishing flags to sync alone.
The partial-tree guard was weaker than it looked:
- it counted archive files locally against package names in the remote
database, and this tree keeps two versions per package, so a checkout with
a spare version of half the repository could pass while still hiding
hundreds of packages. It now compares package-name sets and lists what
would be hidden.
- it treated any unreadable remote as an empty one, so an auth failure or a
corrupt database disabled it. Only rclone's "directory not found" now
counts as a fresh mirror; every other failure aborts.
Also:
- sync had no set -e, so a failed package upload fell through to publishing
the database, advertising packages that were never uploaded. Each transfer
is now checked before the next step.
- --package with no names silently meant "every package", which under --yes
could publish everything from one unset variable in a script.
- push now refuses to run when the host has packages staged from an earlier
failure, since publishing would sign and promote those too.
- epoch versions contain a colon, which rsync reads as host:path, so no
package with an epoch could be transferred. Sources are ./-prefixed.
- remote paths are quoted for the remote shell.
- sync spun forever on a missing option value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
deploy runs build then push, which is the whole workflow on a local build
machine. It resolves the build host before building so a missing --host fails
in a second rather than after a long compile.
--host now overrides $OMARCHY_BUILD_HOST and .build-host on deploy, push, and
the build trigger in omarchy-pkgs release, so a server can be named per
invocation without arming the release auto-trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Heavy packages build faster on a local machine, but there was no way to get
the artifacts to the server: bin/upload-prebuilt publishes to the rclone
remote from whatever tree it runs in, so the local -> host hop was manual.
bin/repo push rsyncs build-output artifacts to the host, verifies checksums,
and runs upload-prebuilt over ssh. Signing stays on the host, which is the
only machine with the key and the only one holding a complete repository.
Publishing from a local checkout was worse than merely unsupported. sync ran
rclone sync --delete-after against a tree that pkgs.omarchy.org/ gitignores,
so on any machine that had not run a full release it would have deleted the
production repository -- guarded only by a y/N prompt that --skip-prod-check
turns off. Package uploads are now additive, deletion moves behind --prune,
and sync refuses to publish a database built from a tree holding fewer
packages than the remote already lists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One-command releases for the omarchy + omarchy-settings pair:
bin/omarchy-pkgs release v4.0.0 | latest | rc [--commit sha] [--base X.Y.Z]
Rewrites both PKGBUILDs in lockstep (same _tag/_commit/pkgver/sha256sums,
pkgrel reset to 1), normalizes upstream tag forms to the vercmp-safe
attached rcN convention, refuses downgrades against the published edge DB,
regenerates and verifies checksums from a cached mirror clone, commits and
pushes to master, and triggers the build host when OMARCHY_BUILD_HOST is
configured. RCs stay on edge; finals are promoted with bin/repo migrate.
Includes a self-test covering tag normalization and pacman ordering, and a
README runbook.
Ryan Hughes