Harden multi-architecture release pipeline

This commit is contained in:
Ryan Hughes committed 2026-09-04 23:40:49 -04:00
1 parent b677f50358
commit 76687fcc82
23 files changed
+557 -271

No files matched your search

+1
View File
@@ -27,6 +27,7 @@ jobs:
set -euo pipefail
pacman -Syu --noconfirm jq
./bin/sync-upstream self-test
./bin/sync-rebuilds --self-test
./bin/omarchy-pkgs self-test
./bin/omarchy-release self-test
'
+46 -20
View File
@@ -57,14 +57,25 @@ file per channel and architecture (`.sync-needed-<channel>-<arch>`);
`auto-release <channel>` works through the queues one architecture at a
time, each with its own backoff (`.build-failed-<channel>-<arch>`), so a
failing build on one architecture never holds up the other; and the release
train advances channels with `--arch all`, so a channel never moves for one
architecture and not another. The first entry is the reference architecture
the release train observes channels through.
train advances channels with `--arch all`: it takes one host-wide lock and
verifies every architecture's source database before moving any of them. The
first entry is the reference architecture the release train observes channels
through. A remote sync failure can still leave a promotion temporarily partial;
rerunning the same advance completes it safely.
Adding an architecture is therefore one change to that list (or
`OMARCHY_ARCHES` in the host's build credentials): the next `check-versions`
tick queues everything the new architecture lacks, and the next
`auto-release` tick starts building it. The builder image bootstraps
Adding an architecture to the scheduled pipeline is therefore one checked-in
change to that list: the next `check-versions` tick queues everything the new
architecture lacks, and the next `auto-release` tick starts building it. A
checked-in list also means the rebuild workflow and release host cannot drift
onto different architecture sets. For a one-off run, override it directly:
```bash
OMARCHY_ARCHES=x86_64 bin/check-versions
OMARCHY_ARCHES=aarch64 bin/check-versions
OMARCHY_ARCHES="x86_64 aarch64" bin/check-versions
```
The builder image bootstraps
`omarchy-keyring` from the x86_64 tree for every architecture, so the first
build of a new architecture does not depend on a repository that only it can
create.
@@ -450,7 +461,12 @@ A package names those dependencies in `.omarchy/package.json`:
{ "source": "aur", "sync": false, "rebuild_on": ["qt6-base", "qt6-declarative", "qt6-wayland"] }
```
`bin/sync-rebuilds` reads each named package's version from the official repositories and compares it to `rebuilt_against`, the record of what the checked-in pkgrel was last bumped for. pkgrel is bumped unless every name in `rebuild_on` is recorded and still matches, so a name the record does not carry reads as changed rather than going unexamined forever. Opting a package in therefore buys one rebuild: what its published build actually linked against is not knowable from here, and a record written without a rebuild would certify a build nobody checked.
`bin/sync-rebuilds` reads each named package's version from the official
repositories for every published architecture the package supports and compares
it to `rebuilt_against`. Records are kept per architecture because Arch and
Arch Linux ARM can carry different dependency versions. pkgrel is bumped once
when any recorded version moves; that one source revision is then rebuilt by
each architecture's normal queue.
The bump is the point of the command, and it has to land in git rather than in the builder. A rebuild that reuses the published version string produces a package pacman will never offer anyone, so merely unlocking the build gate would ship nothing. Bumping pkgrel needs no other change: `bin/check-versions` and the builder both already rebuild when pkgrel moves.
@@ -458,9 +474,12 @@ For an AUR-synced package the bump is expressed as the dotted Omarchy pkgrel suf
The bumped version is checked against the published one as well as the checked-in one, and refused when pacman would not order it higher. The checked-in version is not the floor; what a user already has is, and a checkout that has fallen behind the repository can otherwise be bumped to something that loses to the package it means to replace. That check is skipped with a warning when the published database cannot be read.
Versions are read from the local pacman database, so this runs on Arch or in an Arch container against a synced database. Only `core`, `extra` and `multilib` count: a Qt release sitting in testing or kde-unstable is not what the builder will link against, and rebuilding for it would ship a package built against the wrong ABI. The workflow points that database at `mirror.omarchy.org`, the mirror the x86_64 builder itself uses, because a mirror running ahead of the builder would record a version the build never linked against and nothing re-fires once the record matches.
aarch64 is not covered. Those builds resolve Qt from Arch Linux ARM, which can lag Arch, so one record cannot describe both architectures. Only x86_64 is published today, so nothing currently ships from the untracked side; if ARM publishing starts, `rebuilt_against` has to become per-architecture before this can be trusted there.
x86_64 versions are read from the local pacman database, so the workflow runs
in an Arch container pointed at `mirror.omarchy.org`, the same mirror as the
x86_64 builder. aarch64 versions are read directly from the live Arch Linux ARM
repository database, which is also what the ARM builder uses. Testing and
staging repositories do not count. A legacy flat `rebuilt_against` record is
read as x86_64 and is migrated naturally the next time a rebuild is needed.
### Other
@@ -668,7 +687,7 @@ Fields:
- `skip_build`: optional boolean; defaults to `false`. Set `true` to exclude a package from scheduled version checks and unscoped builds. The package can still be built explicitly with `bin/repo release --package <name>`.
- `pkgrel`: optional Omarchy pkgrel suffix for a version-pinned rebuild bump. This emits `<aur pkgrel>.<suffix>` instead of replacing AUR's pkgrel. `offset` can be used only when preserving monotonic upgrades from old absolute pkgrel bumps. The metadata is removed automatically when AUR sync changes `pkgver`; the current package version is read from the checked-in PKGBUILD, so the version is not duplicated in JSON.
- `rebuild_on`: optional array of package names this package links against closely enough that it must be rebuilt when they change, independent of its own source. Read by `bin/sync-rebuilds`.
- `rebuilt_against`: written by `bin/sync-rebuilds`. Records the version of each `rebuild_on` package that the current pkgrel was bumped for.
- `rebuilt_against`: written by `bin/sync-rebuilds`. Maps each published architecture to the versions of its `rebuild_on` packages that the current pkgrel was bumped for.
- `upstream_commit`: set by `bin/sync-aur` for AUR packages. Used by `bin/package-worktree` to recreate the exact raw AUR package that Omarchy last synced.
### Build Matrix
@@ -764,8 +783,10 @@ bin/repo release --package my-package
### aarch64
- Built on the repository host like x86_64; under QEMU when the host is x86_64
- Uses Arch Linux ARM repositories (one mirrorlist for every channel — Arch
Linux ARM publishes no dated snapshots to pin a channel's base to)
- On an ARM host, package builds and the signing/database utility containers
run natively; only an explicitly requested x86_64 package build is emulated
- Uses Arch Linux ARM repositories through the same HTTPS mirror for every
channel (Arch Linux ARM publishes no dated snapshots to pin a channel's base)
- Additional repos: `[alarm]`, `[aur]`
- Same workflow, just add `--arch aarch64`; the scheduled pipeline runs it
automatically once `aarch64` is in `PUBLISHED_ARCHES`
@@ -835,10 +856,11 @@ That cadence is only safe because of three guards:
an operator expects. `check-versions` takes it too — its `git pull` would
otherwise swap PKGBUILDs out from under a running build.
- **Backoff on failure.** A failed release records the attempt in
`.build-failed-<channel>` and backs off exponentially — 10m, 20m, 40m, up to
`.build-failed-<channel>-<arch>` and backs off exponentially — 10m, 20m, 40m, up to
a 6h ceiling — instead of rebuilding the same broken tree every 5 minutes.
**Any new commit clears the backoff immediately**, since a push is the most
likely fix. Clear it by hand with `rm /root/.state/.build-failed-<channel>`.
likely fix. Clear it by hand with
`rm /root/.state/.build-failed-<channel>-<arch>`.
- **Quiet when idle.** With nothing queued a tick exits without output, so the
journal shows the runs that mattered rather than 288 no-ops a day.
@@ -891,10 +913,14 @@ bin/repo timers --local # inspect this machine instead
```
State files are stored in `/root/.state/`:
- `.sync-needed-<channel>` — the packages queued for that channel, one per
line; the release run reads them to name what it is building
- `.build-failed-<channel>` — consecutive failure count, timestamp, and the
commit it failed on (drives the backoff; removing it forces a retry)
- `.sync-needed-<channel>-<arch>` — the packages queued for that channel and
architecture, one per line; the release run reads them to name what it is
building
- `.build-failed-<channel>-<arch>` — consecutive failure count, timestamp, and
the commit it failed on (drives the backoff; removing it forces a retry)
Legacy files without the architecture suffix are consumed once as x86_64
state, so upgrading the host does not lose an in-flight build.
### Schedule (America/New_York)
+40 -33
View File
@@ -25,31 +25,21 @@ SKIP_PROD_CHECK=false
FAST_RING_ONLY=false
BOOTSTRAP=false
PACKAGES=""
ALL_ARCHES=false
# Kept for --arch all, which re-invokes this script per architecture with the
# other arguments unchanged (minus the --arch all pair itself).
ORIGINAL_ARGS=()
skip_next=false
prev=""
arch_option=false
for arg in "$@"; do
if [[ "$skip_next" == true ]]; then
skip_next=false
prev=""
continue
if [[ "$arch_option" == true ]]; then
[[ "$arg" != "all" ]] && ORIGINAL_ARGS+=("--arch" "$arg")
arch_option=false
elif [[ "$arg" == "--arch" ]]; then
arch_option=true
else
ORIGINAL_ARGS+=("$arg")
fi
if [[ "$arg" == "--arch" ]]; then
prev="--arch"
continue
fi
if [[ "$prev" == "--arch" ]]; then
prev=""
if [[ "$arg" == "all" ]]; then
continue
fi
ORIGINAL_ARGS+=("--arch" "$arg")
continue
fi
ORIGINAL_ARGS+=("$arg")
done
usage() {
@@ -93,21 +83,13 @@ while [[ $# -gt 0 ]]; do
;;
--arch)
if [[ "$2" == "all" ]]; then
# A release train advances every published architecture together; a
# channel that moved for one and not another would serve mismatched
# versions to users on the two. Re-run this script once per
# architecture with the same arguments, stopping at the first failure
# so the operator sees which architecture is stuck (the advance is
# idempotent, so re-running with --arch all resumes where it stopped).
shift 2
for arch in $(published_arches); do
"$0" --arch "$arch" "${ORIGINAL_ARGS[@]}" || exit $?
done
exit 0
ALL_ARCHES=true
ARCH=all
else
require_valid_arch "$2"
ARCH="$2"
update_arch_paths
fi
require_valid_arch "$2"
ARCH="$2"
update_arch_paths
shift 2
;;
--package)
@@ -175,6 +157,31 @@ case "$FROM->$TO" in
;;
esac
if [[ "$ALL_ARCHES" == true ]]; then
# Hold one lock across the whole operation so a timer cannot mutate a
# channel between architectures. Check every source database before moving
# the first one; a failed sync can still require an idempotent retry, but a
# missing architecture never creates a knowingly partial advance.
if [[ "$DRY_RUN" != true ]]; then
acquire_release_lock || exit 1
fi
ARCHES=$(published_arches)
for arch in $ARCHES; do
source_db="$REPO_ROOT/$FROM/$arch/omarchy.db.tar.zst"
if [[ ! -f "$source_db" ]]; then
print_error "Source database not found: $source_db"
echo "Nothing has been published to the $FROM channel for $arch."
exit 1
fi
done
for arch in $ARCHES; do
"$0" --arch "$arch" "${ORIGINAL_ARGS[@]}" || exit $?
done
exit 0
fi
SOURCE_DIR="$REPO_ROOT/$FROM/$ARCH"
TARGET_DIR="$REPO_ROOT/$TO/$ARCH"
SOURCE_DB="$SOURCE_DIR/omarchy.db.tar.zst"
+13 -2
View File
@@ -131,17 +131,28 @@ release_arch() {
if "$BUILD_ROOT/bin/repo" release --mirror "$MIRROR" --arch "$arch" --skip-prod-check; then
print_success "Release completed successfully for $MIRROR ($arch)"
rm -f "$state_file" "$fail_file"
local completed_state=("$state_file" "$fail_file")
if [[ "$arch" == "x86_64" ]]; then
completed_state+=("$(legacy_sync_queue_file "$MIRROR")" "$(legacy_sync_fail_file "$MIRROR")")
fi
if ! rm -f "${completed_state[@]}"; then
print_error "Release succeeded, but its queue state could not be cleared"
return 1
fi
print_success "State file removed: $state_file"
return 0
fi
fail_count=$((fail_count + 1))
cat >"$fail_file" <<EOF
if ! cat >"$fail_file" <<EOF
FAILURE_COUNT=$fail_count
FAILURE_AT=$(date +%s)
FAILURE_FINGERPRINT=$(current_fingerprint)
EOF
then
print_error "Could not record failure state: $fail_file"
return 1
fi
local next
next=$(backoff_seconds "$fail_count")
print_error "Release failed for $MIRROR ($arch) (attempt $fail_count)"
+5 -4
View File
@@ -102,6 +102,8 @@ while [[ $# -gt 0 ]]; do
esac
done
require_valid_arch "$ARCH"
if [[ $DEFER_RUNTIME_DEPS != "false" && $DEFER_RUNTIME_DEPS != "true" ]]; then
print_error "OMARCHY_DEFER_RUNTIME_DEPS must be true or false"
exit 1
@@ -170,8 +172,7 @@ check_docker
HOST_ARCH=$(uname -m)
[[ "$HOST_ARCH" == "arm64" ]] && HOST_ARCH=aarch64
if [[ "$HOST_ARCH" != "$ARCH" ]]; then
# shellcheck disable=SC2086
if ! docker run --rm $(get_platform_arg "$ARCH") alpine:3.21 /bin/true >/dev/null 2>&1; then
if ! docker run --rm "$(get_platform_arg "$ARCH")" alpine:3.21 /bin/true >/dev/null 2>&1; then
print_info "Setting up QEMU for $ARCH emulation on this $HOST_ARCH host..."
setup_qemu
fi
@@ -183,7 +184,7 @@ if [[ $KEEP_BUILD_WORKSPACE == "1" ]]; then
print_info "Keeping existing build workspace..."
else
print_info "Cleaning build workspace..."
rm -rf "$BUILD_OUTPUT_DIR"/*
rm -rf "${BUILD_OUTPUT_DIR:?}"/*
fi
mkdir -p "$BUILD_OUTPUT_DIR"
@@ -237,7 +238,7 @@ DOCKER_ARGS=(
# Run the builder with assembled args
PLATFORM_ARG=$(get_platform_arg "$ARCH")
docker run $PLATFORM_ARG "${DOCKER_ARGS[@]}" "$IMAGE_TAG" /build/build.sh
docker run "$PLATFORM_ARG" "${DOCKER_ARGS[@]}" "$IMAGE_TAG" /build/build.sh
BUILD_RESULT=$?
+1 -1
View File
@@ -180,7 +180,7 @@ check_mirror() {
needs_build=true
packages+=("$pkg")
fi
done < <(packages_for_unscoped_build "$mirror")
done < <(packages_for_unscoped_build "$mirror" "$ARCH")
echo ""
+7 -6
View File
@@ -31,12 +31,13 @@ clean_packages() {
# Skip signature files
[[ "$pkg" == *.sig ]] && continue
# Extract package name (remove version and architecture)
# Format: name-version-release-arch.pkg.tar.*
# The version starts at the first "-<digit>" and neither pkgver nor pkgrel
# can contain a hyphen, so the architecture is whatever sits between the
# last hyphen and .pkg.tar — any, x86_64, aarch64, and whatever comes next.
local pkgname=$(echo "$pkg" | sed -E 's/-[0-9]+.*-[^-]+\.pkg\.tar\..*//')
# Format: name-version-release-arch.pkg.tar.*. Work from the right because
# package names can themselves contain version-like pieces (qt6-5compat,
# nvidia-580xx-utils), while pkgver and pkgrel cannot contain hyphens.
local stem="${pkg%%.pkg.tar.*}"
stem="${stem%-*}" # architecture
stem="${stem%-*}" # pkgrel
local pkgname="${stem%-*}" # pkgver
# Add to array
if [[ -n "${packages[$pkgname]}" ]]; then
+12 -6
View File
@@ -23,7 +23,7 @@ source "$BUILD_ROOT/helpers/paths.sh"
source "$BUILD_ROOT/helpers/host-helpers.sh"
UPSTREAM_URL="${OMARCHY_UPSTREAM_URL:-https://github.com/basecamp/omarchy.git}"
EDGE_DB_URL="${OMARCHY_EDGE_DB_URL:-https://pkgs.omarchy.org/edge/x86_64/omarchy.db.tar.zst}"
EDGE_DB_URL="${OMARCHY_EDGE_DB_URL:-https://pkgs.omarchy.org/edge/$(reference_arch)/omarchy.db.tar.zst}"
RELEASE_PACKAGES=(omarchy omarchy-settings)
DEFAULT_RC_REF="quattro"
SRCDEST_DIR="$BUILD_ROOT/.srcdest"
@@ -327,13 +327,19 @@ regenerate_checksums() {
trigger_build_host() {
local host
queue_edge_builds() {
mkdir -p "$STATE_DIR" || return 1
local arch
for arch in $(published_arches); do
touch "$(sync_queue_file edge "$arch")" || return 1
done
}
# Explicit host configuration outranks the local-host inference (a
# workstation that ran a full local release carries the db marker too).
if ! resolve_repo_host "${REPO_HOST_OVERRIDE:-}" >/dev/null && on_repo_host; then
print_info "Triggering edge build locally (this is the build host)..."
if mkdir -p "${OMARCHY_STATE_DIR:-/root/.state}" &&
touch "${OMARCHY_STATE_DIR:-/root/.state}/.sync-needed-edge" &&
systemctl start --no-block omarchy-auto-release-edge.service; then
if queue_edge_builds && systemctl start --no-block omarchy-auto-release-edge.service; then
print_success "Edge build triggered"
else
print_warning "Could not start the edge release service — the 6-hourly timer will pick it up"
@@ -343,11 +349,11 @@ trigger_build_host() {
if ! host=$(resolve_repo_host "${REPO_HOST_OVERRIDE:-}"); then
print_info "No repository host configured (set OMARCHY_REPO_HOST or $BUILD_ROOT/.repo-host — any ssh destination, e.g. root@<host> or an ssh-config alias)."
print_info "The 6-hourly auto-release timer will pick this up, or trigger manually:"
echo " ssh <build-host> 'git -C /root/omarchy-pkgs pull --ff-only && touch /root/.state/.sync-needed-edge && systemctl start omarchy-auto-release-edge.service'"
echo " ssh <build-host> 'git -C /root/omarchy-pkgs pull --ff-only && systemctl start omarchy-check-versions.service omarchy-auto-release-edge.service'"
return 0
fi
print_info "Triggering edge build on $host..."
if ssh "$host" 'git -C /root/omarchy-pkgs pull --ff-only && mkdir -p /root/.state && touch /root/.state/.sync-needed-edge && systemctl start --no-block omarchy-auto-release-edge.service'; then
if ssh "$host" 'git -C /root/omarchy-pkgs pull --ff-only && cd /root/omarchy-pkgs && export BUILD_ROOT=/root/omarchy-pkgs && source helpers/paths.sh && mkdir -p "$STATE_DIR" && for arch in $(published_arches); do touch "$(sync_queue_file edge "$arch")"; done && systemctl start --no-block omarchy-auto-release-edge.service'; then
print_success "Edge build triggered on $host"
else
print_warning "Could not trigger $host — the 6-hourly timer will pick it up"
+75 -50
View File
@@ -28,11 +28,9 @@ ISO_REPO="${OMARCHY_ISO_REPO:-omacom-io/omarchy-iso}"
DEV_BRANCH="${OMARCHY_DEV_BRANCH:-quattro}"
PKGS_DB_BASE="${OMARCHY_PKGS_DB_BASE:-https://pkgs.omarchy.org}"
# The train observes channels through the reference architecture (the first
# published one). The pair is pinned to one version for every architecture and
# built for all of them in one rc trigger, so one architecture's database is
# the state of the release; per-architecture drift is a build failure the
# scheduled pipeline reports, not something the train has to poll for.
# The first published architecture supplies the version-ordering floor when
# cutting pins. Readiness checks below still verify every published
# architecture before an RC or final release can move forward.
OBSERVED_ARCH=$(reference_arch)
RC_DB_URL="$PKGS_DB_BASE/rc/$OBSERVED_ARCH/omarchy.db.tar.zst"
@@ -187,12 +185,12 @@ ensure_work_clone() {
# Prints omarchy's published version in a channel; empty when absent, rc 2 when
# the database cannot be read (callers must not mistake an outage for absence).
published_version() {
local channel="$1" tmp descs
local channel="$1" arch="${2:-$OBSERVED_ARCH}" tmp descs
tmp=$(mktemp) || return 2
# A unique query string busts the CDN cache: right after a sync the plain
# URL can keep serving the previous db for a while, which reads as "not
# published yet" to status, the wait loop, and ship's pre-checks.
if ! curl -sf "$PKGS_DB_BASE/$channel/$OBSERVED_ARCH/omarchy.db.tar.zst?$(date +%s%N)" -o "$tmp"; then
if ! curl -sf "$PKGS_DB_BASE/$channel/$arch/omarchy.db.tar.zst?$(date +%s%N)" -o "$tmp"; then
rm -f "$tmp"
return 2
fi
@@ -213,6 +211,14 @@ published_version() {
' <<<"$descs"
}
all_arches_at_version() { # all_arches_at_version <channel> <pkgver>
local channel="$1" want="$2" arch got
for arch in $(published_arches); do
got=$(published_version "$channel" "$arch" 2>/dev/null) || return 1
[[ "${got%-*}" == "$want" ]] || return 1
done
}
# The rc branch of THIS repo carries the current pins. Read them without
# touching the working tree.
rc_branch_pin() { # prints "pkgver commit", empty when no rc branch
@@ -297,12 +303,16 @@ host_advance() { # host_advance <from> <to> [extra args...]
}
wait_for_published() { # wait_for_published <channel> <version> [timeout-seconds]
local channel="$1" want="$2" timeout="${3:-3600}" waited=0 got
print_info "Waiting for $want to appear in the $channel channel (up to $((timeout / 60))m)..."
local channel="$1" want="$2" timeout="${3:-3600}" waited=0 arch got pending
print_info "Waiting for $want in $channel for: $(published_arches | tr '\n' ' ')"
while ((waited < timeout)); do
got=$(published_version "$channel" 2>/dev/null) || got=""
if [[ "${got%-*}" == "$want" ]]; then
print_success "$channel now serves omarchy $got"
pending=""
for arch in $(published_arches); do
got=$(published_version "$channel" "$arch" 2>/dev/null) || got=""
[[ "${got%-*}" == "$want" ]] || pending+=" $arch=${got:-unreachable}"
done
if [[ -z "$pending" ]]; then
print_success "$channel now serves omarchy $want on every published architecture"
return 0
fi
sleep 60
@@ -310,7 +320,7 @@ wait_for_published() { # wait_for_published <channel> <version> [timeout-seconds
printf '.' >&2
done
echo "" >&2
print_warning "Timed out waiting for $want in $channel (currently: ${got:-unknown})"
print_warning "Timed out waiting for $want in $channel (pending:$pending)"
print_info "The build may still be running — re-run this command to resume."
return 1
}
@@ -674,15 +684,13 @@ cmd_rc() {
if [[ -n "$pin" ]]; then
local pin_ver="${pin%% *}" pin_commit="${pin##* }"
if [[ "$pin_commit" == "$head" && "$pin_ver" =~ ^${version//./\\.}rc[0-9]+$ ]]; then
local pub
pub=$(published_version rc 2>/dev/null) || pub=""
if [[ "${pub%-*}" == "$pin_ver" ]]; then
print_success "$pin_ver is already cut from this head and published to rc"
if all_arches_at_version rc "$pin_ver"; then
print_success "$pin_ver is already cut from this head and published to rc on every architecture"
maybe_iso "$pin_ver" rc "$iso_mode"
return 0
fi
print_info "$pin_ver is pinned from this head but not published yet — re-triggering the build"
trigger_rc_build || true
trigger_rc_build || return 1
[[ "$wait" == true ]] && wait_for_published rc "$pin_ver"
maybe_iso "$pin_ver" rc "$iso_mode"
return 0
@@ -698,7 +706,7 @@ cmd_rc() {
new_ver="${new_pin%% *}"
print_success "Pinned $new_ver (rc branch pushed)"
trigger_rc_build || true
trigger_rc_build || return 1
if [[ "$wait" == true ]]; then
wait_for_published rc "$new_ver" || return 1
fi
@@ -720,9 +728,7 @@ cmd_ship() {
exit 1
fi
version=$(branch_to_version "$branch")
local stable_now
stable_now=$(published_version stable 2>/dev/null) || stable_now=""
if [[ "${stable_now%-*}" == "$version" ]] &&
if all_arches_at_version stable "$version" &&
gh release view "v$version" --repo "$UPSTREAM_REPO" >/dev/null 2>&1; then
print_success "Nothing to ship — $version is tagged, released, and live on stable"
exit 0
@@ -764,10 +770,8 @@ cmd_ship() {
echo " omarchy-release rc"
exit 1
fi
local pub
pub=$(published_version rc 2>/dev/null) || pub=""
if [[ "${pub%-*}" != "$pin_ver" ]]; then
print_error "$pin_ver is pinned but rc serves '${pub:-nothing}' — the candidate build hasn't published"
if ! all_arches_at_version rc "$pin_ver"; then
print_error "$pin_ver is pinned but is not published for every architecture"
echo "Wait for it (or re-run: omarchy-release rc), then ship."
exit 1
fi
@@ -811,10 +815,9 @@ cmd_ship() {
# 2. Final pins into rc. Resolve the tag we just established so the final
# PKGBUILDs record both its provenance and its exact commit.
local rc_pub stable_pub
rc_pub=$(published_version rc 2>/dev/null) || rc_pub=""
if [[ "${rc_pub%-*}" == "$version" ]]; then
print_success "2/7 Final $version already published to rc"
local stable_pub
if all_arches_at_version rc "$version"; then
print_success "2/7 Final $version already published to rc on every architecture"
else
if [[ "$pin_ver" != "$version" ]]; then
print_info "2/7 Pinning final $version from tag v$version..."
@@ -822,22 +825,21 @@ cmd_ship() {
else
print_info "2/7 Final $version pinned — re-triggering build"
fi
trigger_rc_build || true
trigger_rc_build || exit 1
wait_for_published rc "$version" || exit 1
fi
# 3. Promote rc -> stable
stable_pub=$(published_version stable 2>/dev/null) || stable_pub=""
if [[ "${stable_pub%-*}" == "$version" ]]; then
print_success "3/7 Stable already serves $version"
if all_arches_at_version stable "$version"; then
print_success "3/7 Stable already serves $version on every architecture"
else
host_advance rc stable || exit 1
stable_pub=$(published_version stable 2>/dev/null) || stable_pub=""
if [[ "${stable_pub%-*}" != "$version" ]]; then
print_error "Promotion ran but stable serves '${stable_pub:-nothing}' — investigate before continuing"
if ! all_arches_at_version stable "$version"; then
print_error "Promotion ran but stable does not serve $version on every architecture"
exit 1
fi
print_success "3/7 Promoted to stable: omarchy $stable_pub"
stable_pub=$(published_version stable 2>/dev/null) || stable_pub="$version"
print_success "3/7 Promoted to stable: omarchy $stable_pub on every architecture"
fi
# 4. Final pins onto master (keeps edge overlap publishing and the repo record)
@@ -920,7 +922,7 @@ next_step() { # prints "<command>|<description>"
last=$(newest_release_branch 2>/dev/null) || last=""
if [[ -n "$last" && "$STABLE_VER" != "<unreachable>" ]]; then
last_ver=$(branch_to_version "$last")
if [[ "${STABLE_VER%-*}" != "$last_ver" ]] ||
if ! all_arches_at_version stable "$last_ver" ||
{ command -v gh >/dev/null && ! gh release view "v$last_ver" --repo "$UPSTREAM_REPO" >/dev/null 2>&1; }; then
echo "ship|$last_ver is tagged but not fully shipped — resume ship"
return
@@ -936,7 +938,7 @@ next_step() { # prints "<command>|<description>"
echo "ship|Final $TRAIN_VER is pinned — finish shipping (re-runs are safe)"
elif [[ "$pin_commit" != "$TRAIN_HEAD" ]]; then
echo "rc|$TRAIN has commits newer than $pin_ver — cut the next candidate"
elif [[ "${RC_VER%-*}" != "$pin_ver" ]]; then
elif ! all_arches_at_version rc "$pin_ver"; then
echo "rc|$pin_ver is pinned but not published — re-run rc to re-trigger/wait"
else
echo "ship|$pin_ver is published to rc — test it, then ship"
@@ -1006,20 +1008,24 @@ cmd_doctor() {
check "makepkg available (checksums)" command -v makepkg
check "curl available" command -v curl
check "upstream reachable ($UPSTREAM_URL)" git ls-remote "$UPSTREAM_URL" HEAD
local ch
local ch arch
for ch in edge stable; do
if published_version "$ch" >/dev/null 2>&1; then
print_success "$ch channel db readable"
for arch in $(published_arches); do
if published_version "$ch" "$arch" >/dev/null 2>&1; then
print_success "$ch/$arch channel db readable"
else
print_error "$ch/$arch channel db readable"
failures=$((failures + 1))
fi
done
done
for arch in $(published_arches); do
if published_version rc "$arch" >/dev/null 2>&1; then
print_success "rc/$arch channel db readable"
else
print_error "$ch channel db readable"
failures=$((failures + 1))
print_warning "rc/$arch channel db not readable — bootstrap it first: bin/repo bootstrap-rc (on the build host)"
fi
done
if published_version rc >/dev/null 2>&1; then
print_success "rc channel db readable"
else
print_warning "rc channel db not readable — bootstrap it first: bin/repo bootstrap-rc (on the build host)"
fi
local host
if host=$(repo_host); then
check "build host ssh ($host)" ssh -o ConnectTimeout=10 "$host" true
@@ -1066,6 +1072,25 @@ cmd_self_test() {
expect "version_is_patch 5.0.0" "$(version_is_patch 5.0.0 && echo yes || echo no)" "no"
expect "previous_patch_tag 4.0.2" "$(previous_patch_tag 4.0.2)" "v4.0.1"
expect "previous_patch_tag 4.0.10" "$(previous_patch_tag 4.0.10)" "v4.0.9"
# Keep release readiness fail-closed when only one architecture has reached
# the requested version. This replaces the network reader for this process;
# self-test exits immediately afterwards.
published_version() {
case "$2" in
x86_64) echo "${TEST_X86_VERSION:-4.0.2-1}" ;;
aarch64) echo "${TEST_ARM_VERSION:-4.0.2-1}" ;;
esac
}
PUBLISHED_ARCHES=x86_64
expect "x86-only readiness" "$(all_arches_at_version rc 4.0.2 && echo ready || echo blocked)" "ready"
PUBLISHED_ARCHES=aarch64
expect "ARM-only readiness" "$(all_arches_at_version rc 4.0.2 && echo ready || echo blocked)" "ready"
PUBLISHED_ARCHES="x86_64 aarch64"
TEST_ARM_VERSION=4.0.1-1
expect "mixed versions block release" "$(all_arches_at_version rc 4.0.2 && echo ready || echo blocked)" "blocked"
TEST_ARM_VERSION=4.0.2-1
expect "both architectures ready" "$(all_arches_at_version rc 4.0.2 && echo ready || echo blocked)" "ready"
echo ""
if ((failures == 0)); then
print_success "Self-test passed"
+9 -6
View File
@@ -88,9 +88,12 @@ if [[ ! $REPLY =~ ^[Yy]$ ]]; then
exit 0
fi
# Build/update the Docker image (always use x86_64 for removal - it's architecture independent)
# repo-remove is mirror-independent — always use the edge x86_64 image
build_docker_image "$BUILD_DIR" "x86_64" "edge"
# repo-remove is architecture-independent, so use a host-native edge image.
TOOL_ARCH=$(docker_native_arch) || {
print_error "Unsupported host architecture: $(uname -m)"
exit 1
}
build_docker_image "$BUILD_DIR" "$TOOL_ARCH" "edge"
acquire_release_lock || exit 1
@@ -99,13 +102,13 @@ print_info "Removing package..."
# Ensure directory is writable by container user
make_dir_writable "$REPO_DIR"
# Run the removal script in Docker (always use x86_64 image)
docker run --rm --platform linux/amd64 \
# Run the removal script in the host-native image.
docker run --rm "$(get_platform_arg "$TOOL_ARCH")" \
-e ARCH="$ARCH" \
-e MIRROR="$MIRROR" \
-v "$REPO_ROOT:/pkgs.omarchy.org" \
-v "$BUILD_DIR:/build:ro" \
omarchy-pkg-builder:latest-x86_64-edge /build/remove-package.sh "$PACKAGE_NAME"
"omarchy-pkg-builder:latest-$TOOL_ARCH-edge" /build/remove-package.sh "$PACKAGE_NAME"
RESULT=$?
+2 -1
View File
@@ -15,6 +15,7 @@ set -e
BUILD_ROOT=$(realpath "${BASH_SOURCE[0]%/*}/..")
source "$BUILD_ROOT/helpers/message-helpers.sh"
source "$BUILD_ROOT/helpers/paths.sh"
CHECK_ONLY=false
SKIP_TIMERS=false
@@ -238,7 +239,7 @@ echo ""
# --- release timers ----------------------------------------------------------
print_info "Published architectures: $(published_arches | tr '\n' ' ')"
echo " (PUBLISHED_ARCHES in helpers/paths.sh, or OMARCHY_ARCHES in $CREDENTIALS)"
echo " (PUBLISHED_ARCHES in helpers/paths.sh; OMARCHY_ARCHES overrides a one-off command)"
echo ""
TIMERS=(omarchy-check-versions omarchy-auto-release-edge omarchy-auto-release-rc omarchy-auto-release-stable)
+10 -5
View File
@@ -66,23 +66,28 @@ if [[ -z "$GPG_PASSPHRASE" ]]; then
exit 1
fi
# Build/update the Docker image (always use x86_64 for signing - it's architecture independent)
build_docker_image "$BUILD_DIR" "x86_64" "$MIRROR"
# Signing is architecture-independent, so run its utility container natively
# on either an x86_64 or ARM host.
TOOL_ARCH=$(docker_native_arch) || {
print_error "Unsupported host architecture: $(uname -m)"
exit 1
}
build_docker_image "$BUILD_DIR" "$TOOL_ARCH" "$MIRROR"
print_info "Running package signing..."
# Ensure output directory is writable by container user
make_dir_writable "$BUILD_OUTPUT_DIR"
# Run the signing script in Docker (always use x86_64 image)
docker run --rm --platform linux/amd64 \
# Run the signing script in the host-native image.
docker run --rm "$(get_platform_arg "$TOOL_ARCH")" \
-e ARCH="$ARCH" \
-e MIRROR="$MIRROR" \
-e GPG_PRIVATE_KEY="$GPG_PRIVATE_KEY" \
-e GPG_PASSPHRASE="$GPG_PASSPHRASE" \
-v "$BUILD_ROOT/build-output:/build-output" \
-v "$BUILD_DIR:/build:ro" \
omarchy-pkg-builder:latest-x86_64-$MIRROR /build/sign.sh
"omarchy-pkg-builder:latest-$TOOL_ARCH-$MIRROR" /build/sign.sh
SIGN_RESULT=$?
+228 -59
View File
@@ -17,11 +17,14 @@ SELF_TEST=false
# rebuilding for it would ship a package built against the wrong ABI.
OFFICIAL_REPOS=" core extra multilib core-debug extra-debug "
# The published repository, used as the floor a bumped pkgrel has to clear.
# Only x86_64 is published today; aarch64 has no repository to compare against.
# The published repositories are the floor a bumped pkgrel has to clear.
PUBLISHED_BASE_URL="${OMARCHY_PUBLISHED_BASE_URL:-https://pkgs.omarchy.org}"
PUBLISHED_MIRRORS=(edge stable)
PUBLISHED_ARCH=x86_64
# Arch Linux ARM has no dated snapshots. This is the same live repository the
# aarch64 builder resolves; override it only when the builder mirror changes.
ALARM_BASE_URL="${OMARCHY_ALARM_BASE_URL:-https://fl.us.mirror.archlinuxarm.org/aarch64}"
ALARM_REPOS=(core extra alarm aur)
usage() {
cat <<EOF
@@ -34,10 +37,13 @@ A package opts in by naming those dependencies in .omarchy/package.json:
{ "source": "aur", "sync": false, "rebuild_on": ["qt6-base"] }
The versions the current pkgrel was bumped for are recorded alongside, in
rebuilt_against, and written by this command:
The versions the current pkgrel was bumped for are recorded per published
architecture in rebuilt_against, and written by this command:
{ "rebuild_on": ["qt6-base"], "rebuilt_against": { "qt6-base": "6.11.2-2" } }
{ "rebuild_on": ["qt6-base"], "rebuilt_against": {
"x86_64": { "qt6-base": "6.11.2-3" },
"aarch64": { "qt6-base": "6.11.2-2" }
} }
pkgrel is bumped unless every package named in rebuild_on is recorded and still
matches. A name that is missing from the record counts as changed, so opting a
@@ -60,8 +66,9 @@ Examples:
$0 # Update every package that declares rebuild_on
$0 quickshell-git # Update specific packages
Trigger versions are read from the local pacman database, so sync it first
(pacman -Sy) or this reports whatever that database last saw.
x86_64 trigger versions come from the local pacman database; aarch64 versions
come from the live Arch Linux ARM repository database. Only architectures in
PUBLISHED_ARCHES that the package supports are considered.
EOF
}
@@ -94,7 +101,7 @@ SPECIFIC_MODE=false
# The version of a trigger package as the build container would resolve it.
# A name pacman does not know reports nothing rather than failing, so the caller
# gets to say which package was left alone instead of the run dying here.
repo_version() {
native_repo_version() {
local package="$1"
local info
@@ -108,9 +115,59 @@ repo_version() {
' <<<"$info"
}
load_alarm_repo() {
local repo="$1" db index
index="$TEMP_DIR/alarm-$repo.index"
[[ -f "$index" ]] && return 0
db="$TEMP_DIR/alarm-$repo.db"
if ! curl -fsSL --max-time 120 -o "$db" "$ALARM_BASE_URL/$repo/$repo.db" 2>/dev/null; then
print_error "Could not read the aarch64 $repo repository database"
return 1
fi
if ! tar -tf "$db" >/dev/null 2>&1; then
print_error "Unreadable aarch64 $repo repository database"
return 1
fi
tar -xOf "$db" --wildcards '*/desc' 2>/dev/null | awk '
function emit() {
if (name != "" && version != "") {
print name "\t" version
if (base != "" && base != name) print base "\t" version
}
name=""; base=""; version=""
}
$0 == "%FILENAME%" { emit(); next }
$0 == "%NAME%" { if (name != "" && version != "") emit(); getline; name=$0; next }
$0 == "%BASE%" { getline; base=$0; next }
$0 == "%VERSION%" { getline; version=$0; next }
END { emit() }
' >"$index"
}
alarm_repo_version() {
local package="$1" repo version
for repo in "${ALARM_REPOS[@]}"; do
load_alarm_repo "$repo" || return 1
version=$(awk -F '\t' -v package="$package" '$1 == package { print $2; exit }' "$TEMP_DIR/alarm-$repo.index")
if [[ -n "$version" ]]; then
echo "$version"
return 0
fi
done
}
repo_version() { # repo_version <arch> <package>
case "$1" in
x86_64) native_repo_version "$2" ;;
aarch64) alarm_repo_version "$2" ;;
*) return 1 ;;
esac
}
declare -A PUBLISHED_VERSION=()
PUBLISHED_LOADED=false
PUBLISHED_AVAILABLE=true
remember_published() {
local name="$1"
@@ -131,34 +188,35 @@ load_published_versions() {
[[ "$PUBLISHED_LOADED" == true ]] && return 0
PUBLISHED_LOADED=true
local mirror db name base version
local arch mirror db name base version
for mirror in "${PUBLISHED_MIRRORS[@]}"; do
db="$TEMP_DIR/published-$mirror.db.tar.zst"
for arch in $(published_arches); do
for mirror in "${PUBLISHED_MIRRORS[@]}"; do
db="$TEMP_DIR/published-$mirror-$arch.db.tar.zst"
if ! curl -fsSL --max-time 120 -o "$db" \
"$PUBLISHED_BASE_URL/$mirror/$PUBLISHED_ARCH/omarchy.db.tar.zst" 2>/dev/null; then
print_warning "Could not read the published $mirror database; bumps are not checked against it this run"
PUBLISHED_AVAILABLE=false
continue
fi
if ! curl -fsSL --max-time 120 -o "$db" \
"$PUBLISHED_BASE_URL/$mirror/$arch/omarchy.db.tar.zst" 2>/dev/null; then
print_warning "Could not read the published $mirror/$arch database; bumps are not checked against it this run"
continue
fi
while IFS=$'\t' read -r name base version; do
[[ -n "$name" && -n "$version" ]] && remember_published "$name" "$version"
[[ -n "$base" && -n "$version" ]] && remember_published "$base" "$version"
done < <(
tar -xOf "$db" --wildcards '*/desc' 2>/dev/null | awk '
function emit() {
if (name != "" && version != "") print name "\t" base "\t" version
name=""; base=""; version=""
}
$0 == "%FILENAME%" { emit(); next }
$0 == "%NAME%" { if (name != "" && version != "") emit(); getline; name=$0; next }
$0 == "%BASE%" { getline; base=$0; next }
$0 == "%VERSION%" { getline; version=$0; next }
END { emit() }
'
)
while IFS=$'\t' read -r name base version; do
[[ -n "$name" && -n "$version" ]] && remember_published "$name" "$version"
[[ -n "$base" && -n "$version" ]] && remember_published "$base" "$version"
done < <(
tar -xOf "$db" --wildcards '*/desc' 2>/dev/null | awk '
function emit() {
if (name != "" && version != "") print name "\t" base "\t" version
name=""; base=""; version=""
}
$0 == "%FILENAME%" { emit(); next }
$0 == "%NAME%" { if (name != "" && version != "") emit(); getline; name=$0; next }
$0 == "%BASE%" { getline; base=$0; next }
$0 == "%VERSION%" { getline; version=$0; next }
END { emit() }
'
)
done
done
}
@@ -252,7 +310,17 @@ record_triggers() {
local package_dir="$1"
local current="$2"
write_metadata "$package_dir" '.rebuilt_against = $current' --argjson current "$current"
# A flat record is the legacy x86_64 shape. Preserve records for
# architectures outside this run, then replace the ones just rebuilt.
write_metadata "$package_dir" '
(.rebuilt_against // {}) as $old |
(if ($old | length) == 0 then {}
elif ($old | to_entries | all(.value | type == "string"))
then {x86_64: $old}
else $old
end) as $by_arch |
.rebuilt_against = ($by_arch * $current)
' --argjson current "$current"
}
# Metadata that does not parse would otherwise drop its package out of the run
@@ -300,21 +368,41 @@ sync_package() {
print_info "Checking $package against ${triggers[*]}..."
local current="{}" trigger version
for trigger in "${triggers[@]}"; do
version=$(repo_version "$trigger")
if [[ -z "$version" ]]; then
print_error " $trigger is in no official repository; leaving $package alone"
((++FAILED))
return 0
fi
if ! current=$(jq -c --arg name "$trigger" --arg version "$version" '.[$name] = $version' <<<"$current"); then
print_error " Could not record $trigger $version for $package"
local current="{}" arch arch_current trigger version considered=0
for arch in $(published_arches); do
package_supports_arch "$package_dir" "$arch" || continue
considered=$((considered + 1))
arch_current="{}"
for trigger in "${triggers[@]}"; do
if ! version=$(repo_version "$arch" "$trigger"); then
print_error " Could not read $arch repository versions; leaving $package alone"
((++FAILED))
return 0
fi
if [[ -z "$version" ]]; then
print_error " $trigger is in no $arch repository; leaving $package alone"
((++FAILED))
return 0
fi
if ! arch_current=$(jq -c --arg name "$trigger" --arg version "$version" '.[$name] = $version' <<<"$arch_current"); then
print_error " Could not record $arch/$trigger $version for $package"
((++FAILED))
return 0
fi
done
if ! current=$(jq -c --arg arch "$arch" --argjson versions "$arch_current" '.[$arch] = $versions' <<<"$current"); then
print_error " Could not record $arch trigger versions for $package"
((++FAILED))
return 0
fi
done
if ((considered == 0)); then
print_info " Skipping: not built for any published architecture"
((++SKIPPED))
return 0
fi
local recorded
if ! recorded=$(package_metadata_value "$package_dir" '.rebuilt_against' ""); then
print_error " Could not read .omarchy/package.json for $package"
@@ -323,13 +411,20 @@ sync_package() {
fi
[[ -n "$recorded" && "$recorded" != "null" ]] || recorded="{}"
# Before architecture-specific records existed, rebuilt_against described
# x86_64. Read it that way without forcing a metadata-only migration.
if jq -e 'to_entries | all(.value | type == "string")' >/dev/null <<<"$recorded"; then
recorded=$(jq -c '{x86_64: .}' <<<"$recorded")
fi
# Walk the declared triggers rather than the record, so a name the record does
# not carry reads as changed instead of going unexamined forever.
local moved
if ! moved=$(jq -r --argjson recorded "$recorded" '
to_entries
| map(select($recorded[.key] != .value)
| "\(.key) \($recorded[.key] // "unrecorded") -> \(.value)")
[to_entries[] as $arch
| $arch.value | to_entries[] as $trigger
| select($recorded[$arch.key][$trigger.key] != $trigger.value)
| "\($arch.key)/\($trigger.key) \($recorded[$arch.key][$trigger.key] // "unrecorded") -> \($trigger.value)"]
| join(", ")
' <<<"$current"); then
print_error " Could not compare recorded trigger versions for $package"
@@ -338,7 +433,7 @@ sync_package() {
fi
if [[ -z "$moved" ]]; then
print_info " Already rebuilt against $(jq -r 'to_entries | map("\(.key) \(.value)") | join(", ")' <<<"$current")"
print_info " Already rebuilt against every published architecture"
((++SKIPPED))
return 0
fi
@@ -466,11 +561,11 @@ selftest_root() {
}
selftest_package() {
local root="$1" name="$2" pkgrel="$3" metadata="$4" pkgver="${5:-1.0}"
local root="$1" name="$2" pkgrel="$3" metadata="$4" pkgver="${5:-1.0}" arches="${6:-x86_64}"
local dir="$root/pkgbuilds/$name"
mkdir -p "$dir/.omarchy"
printf 'pkgname=%s\npkgver=%s\npkgrel=%s\narch=(x86_64)\n' "$name" "$pkgver" "$pkgrel" > "$dir/PKGBUILD"
printf 'pkgname=%s\npkgver=%s\npkgrel=%s\narch=(%s)\n' "$name" "$pkgver" "$pkgrel" "$arches" > "$dir/PKGBUILD"
printf '%s\n' "$metadata" > "$dir/.omarchy/package.json"
}
@@ -514,19 +609,51 @@ selftest_published() {
cat > "$root/stub/curl" <<'STUB'
#!/bin/bash
out=""
url=""
while [[ $# -gt 0 ]]; do
case "$1" in
-o) out="$2"; shift 2 ;;
*) shift ;;
*) url="$1"; shift ;;
esac
done
db="$(dirname "$0")/omarchy.db.tar.zst"
if [[ "$url" == */aarch64/* ]]; then
repo="${url%/*}"
repo="${repo##*/}"
db="$(dirname "$0")/alarm-$repo.db"
else
db="$(dirname "$0")/omarchy.db.tar.zst"
fi
[[ -f "$db" && -n "$out" ]] || exit 22
cp "$db" "$out"
STUB
chmod +x "$root/stub/curl"
}
selftest_alarm() {
local root="$1"
shift
local repo staging="$root/stub/alarm-db" entry name version
for repo in core extra; do
rm -rf "$staging"
mkdir -p "$staging"
if [[ "$repo" == "core" ]]; then
mkdir -p "$staging/unrelated-1-1"
printf '%%FILENAME%%\nunrelated-1-1-aarch64.pkg.tar.zst\n\n%%NAME%%\nunrelated\n\n%%BASE%%\nunrelated\n\n%%VERSION%%\n1-1\n' \
> "$staging/unrelated-1-1/desc"
else
for entry in "$@"; do
name="${entry%=*}"
version="${entry#*=}"
mkdir -p "$staging/$name-$version"
printf '%%FILENAME%%\n%s-%s-aarch64.pkg.tar.zst\n\n%%NAME%%\n%s\n\n%%BASE%%\n%s\n\n%%VERSION%%\n%s\n' \
"$name" "$version" "$name" "$name" "$version" > "$staging/$name-$version/desc"
done
fi
tar -czf "$root/stub/alarm-$repo.db" -C "$staging" .
done
}
cmd_self_test() {
local failures=0
local root
@@ -545,7 +672,8 @@ cmd_self_test() {
local root="$1"
shift
local status=0
PATH="$root/stub:$PATH" "$root/bin/sync-rebuilds" "$@" > "$root/output" 2>&1 || status=$?
OMARCHY_ARCHES="${SELFTEST_ARCHES:-x86_64}" \
PATH="$root/stub:$PATH" "$root/bin/sync-rebuilds" "$@" > "$root/output" 2>&1 || status=$?
echo "$status"
}
@@ -563,7 +691,7 @@ cmd_self_test() {
check "run succeeds" 0 "$(run_case "$root")"
check "pkgrel bumped" 2 "$(pkgrel_of "$root/pkgbuilds/t-partial")"
check "unrecorded trigger now recorded" "2-2" \
"$(jq -r '.rebuilt_against["dep-b"]' "$root/pkgbuilds/t-partial/.omarchy/package.json")"
"$(jq -r '.rebuilt_against.x86_64["dep-b"]' "$root/pkgbuilds/t-partial/.omarchy/package.json")"
echo "Opting a package in buys a rebuild rather than a bare record:"
root=$(selftest_root fresh)
@@ -573,7 +701,7 @@ cmd_self_test() {
check "run succeeds" 0 "$(run_case "$root")"
check "pkgrel bumped" 2 "$(pkgrel_of "$root/pkgbuilds/t-fresh")"
check "trigger recorded" "1-1" \
"$(jq -r '.rebuilt_against["dep-a"]' "$root/pkgbuilds/t-fresh/.omarchy/package.json")"
"$(jq -r '.rebuilt_against.x86_64["dep-a"]' "$root/pkgbuilds/t-fresh/.omarchy/package.json")"
echo "An unchanged package is left alone:"
root=$(selftest_root current)
@@ -612,6 +740,47 @@ cmd_self_test() {
check "suffix recorded for the next AUR sync" 1 \
"$(jq -r '.pkgrel.suffix' "$root/pkgbuilds/t-aur/.omarchy/package.json")"
echo "A dependency is tracked independently for both published architectures:"
root=$(selftest_root multiarch)
selftest_package "$root" t-multi 1 '{"source":"local","rebuild_on":["dep-a"],"rebuilt_against":{"dep-a":"1-1"}}' 1.0 'x86_64 aarch64'
selftest_pacman "$root" dep-a=1-1
selftest_published "$root"
selftest_alarm "$root" dep-a=2-1
SELFTEST_ARCHES="x86_64 aarch64"
check "run succeeds" 0 "$(run_case "$root")"
unset SELFTEST_ARCHES
check "pkgrel bumped once" 2 "$(pkgrel_of "$root/pkgbuilds/t-multi")"
check "x86_64 trigger recorded" "1-1" \
"$(jq -r '.rebuilt_against.x86_64["dep-a"]' "$root/pkgbuilds/t-multi/.omarchy/package.json")"
check "aarch64 trigger recorded" "2-1" \
"$(jq -r '.rebuilt_against.aarch64["dep-a"]' "$root/pkgbuilds/t-multi/.omarchy/package.json")"
echo "An ARM-only run records only the ARM dependency state:"
root=$(selftest_root arm-only)
selftest_package "$root" t-arm 1 '{"source":"local","rebuild_on":["dep-a"]}' 1.0 'x86_64 aarch64'
selftest_pacman "$root"
selftest_published "$root"
selftest_alarm "$root" dep-a=2-1
SELFTEST_ARCHES=aarch64
check "run succeeds" 0 "$(run_case "$root")"
unset SELFTEST_ARCHES
check "pkgrel bumped" 2 "$(pkgrel_of "$root/pkgbuilds/t-arm")"
check "ARM trigger recorded" "2-1" \
"$(jq -r '.rebuilt_against.aarch64["dep-a"]' "$root/pkgbuilds/t-arm/.omarchy/package.json")"
check "x86_64 was not consulted" "false" \
"$(jq -r '.rebuilt_against | has("x86_64")' "$root/pkgbuilds/t-arm/.omarchy/package.json")"
echo "An x86-only package ignores ARM during a dual-architecture run:"
root=$(selftest_root x86-package)
selftest_package "$root" t-x86 1 '{"source":"local","rebuild_on":["dep-a"],"rebuilt_against":{"dep-a":"1-1"}}'
selftest_pacman "$root" dep-a=1-1
selftest_published "$root"
selftest_alarm "$root" dep-a=2-1
SELFTEST_ARCHES="x86_64 aarch64"
check "run succeeds" 0 "$(run_case "$root")"
unset SELFTEST_ARCHES
check "pkgrel untouched" 1 "$(pkgrel_of "$root/pkgbuilds/t-x86")"
echo ""
if [[ "$failures" -eq 0 ]]; then
print_success "Self-test passed"
@@ -626,9 +795,9 @@ if [[ "$SELF_TEST" == true ]]; then
exit $?
fi
for tool in pacman vercmp jq curl; do
for tool in pacman vercmp jq curl tar; do
if ! command -v "$tool" >/dev/null 2>&1; then
print_error "$tool not found: reading trigger versions and ordering pkgrels both need pacman"
print_error "$tool not found: rebuild trigger sync cannot run"
exit 1
fi
done
+26 -26
View File
@@ -105,32 +105,32 @@ echo ""
paused=false
for channel in edge rc stable; do
for arch in $(published_arches); do
fail_file=$(sync_fail_file "$channel" "$arch")
[[ "$arch" == "x86_64" && ! -f "$fail_file" ]] && fail_file=$(legacy_sync_fail_file "$channel")
[[ -f "$fail_file" ]] || continue
if [[ "$paused" == false ]]; then
print_error "Failing builds (backoff active)"
paused=true
fi
FAILURE_COUNT=0 FAILURE_AT=0 FAILURE_FINGERPRINT=""
# shellcheck disable=SC1090
source "$fail_file" 2>/dev/null || true
delay=600
for ((i = 1; i < FAILURE_COUNT; i++)); do
delay=$((delay * 2))
((delay >= 21600)) && { delay=21600; break; }
done
retry_at=$((FAILURE_AT + delay))
now=$(date +%s)
if ((now < retry_at)); then
when="retries at $(date -d "@$retry_at" '+%H:%M:%S' 2>/dev/null || echo "+$((retry_at - now))s")"
else
when="retries on the next tick"
fi
printf ' ✗ %-7s %-8s %s consecutive failure(s), %s\n' "$channel" "$arch" "$FAILURE_COUNT" "$when"
printf ' last attempt %s on commit %s\n' \
"$(date -d "@$FAILURE_AT" '+%Y-%m-%d %H:%M:%S' 2>/dev/null || echo "$FAILURE_AT")" \
"${FAILURE_FINGERPRINT:0:12}"
fail_file=$(sync_fail_file "$channel" "$arch")
[[ "$arch" == "x86_64" && ! -f "$fail_file" ]] && fail_file=$(legacy_sync_fail_file "$channel")
[[ -f "$fail_file" ]] || continue
if [[ "$paused" == false ]]; then
print_error "Failing builds (backoff active)"
paused=true
fi
FAILURE_COUNT=0 FAILURE_AT=0 FAILURE_FINGERPRINT=""
# shellcheck disable=SC1090
source "$fail_file" 2>/dev/null || true
delay=600
for ((i = 1; i < FAILURE_COUNT; i++)); do
delay=$((delay * 2))
((delay >= 21600)) && { delay=21600; break; }
done
retry_at=$((FAILURE_AT + delay))
now=$(date +%s)
if ((now < retry_at)); then
when="retries at $(date -d "@$retry_at" '+%H:%M:%S' 2>/dev/null || echo "+$((retry_at - now))s")"
else
when="retries on the next tick"
fi
printf ' ✗ %-7s %-8s %s consecutive failure(s), %s\n' "$channel" "$arch" "$FAILURE_COUNT" "$when"
printf ' last attempt %s on commit %s\n' \
"$(date -d "@$FAILURE_AT" '+%Y-%m-%d %H:%M:%S' 2>/dev/null || echo "$FAILURE_AT")" \
"${FAILURE_FINGERPRINT:0:12}"
done
done
if [[ "$paused" == true ]]; then
+10 -6
View File
@@ -28,17 +28,21 @@ update_database() {
# Make output directory writable for container
make_dir_writable "$REPO_DIR"
# repo-add is architecture- and mirror-independent, so always use the edge
# x86_64 image. This also lets bootstrap-rc build the rc database before the
# rc channel exists remotely (an rc image can only build after it does).
build_docker_image "$BUILD_DIR" "x86_64" "edge"
# repo-add is architecture- and mirror-independent. Use the host-native edge
# image, which also lets bootstrap-rc run before that channel exists remotely.
local tool_arch
tool_arch=$(docker_native_arch) || {
print_error "Unsupported host architecture: $(uname -m)"
exit 1
}
build_docker_image "$BUILD_DIR" "$tool_arch" "edge"
docker run --rm --platform linux/amd64 \
docker run --rm "$(get_platform_arg "$tool_arch")" \
-e ARCH="$ARCH" \
-e MIRROR="$MIRROR" \
-v "$REPO_ROOT:/output" \
-v "$BUILD_DIR:/build:ro" \
omarchy-pkg-builder:latest-x86_64-edge /build/update-repo.sh
"omarchy-pkg-builder:latest-$tool_arch-edge" /build/update-repo.sh
}
# Main execution
+1 -3
View File
@@ -45,9 +45,7 @@ RUN if [ "${TARGETARCH}" = "amd64" ]; then \
printf 'Server = https://mirror.omarchy.org/$repo/os/$arch\n' > /etc/pacman.d/mirrorlist; \
fi; \
else \
curl -L "https://raw.githubusercontent.com/archlinuxarm/PKGBUILDs/master/core/pacman-mirrorlist/mirrorlist" 2>/dev/null | \
sed -E 's/^\s*#\s*Server\s*=/Server =/g' > /etc/pacman.d/mirrorlist && \
sed -i 's/\$arch/aarch64/g' /etc/pacman.d/mirrorlist; \
printf 'Server = https://fl.us.mirror.archlinuxarm.org/aarch64/$repo\n' > /etc/pacman.d/mirrorlist; \
fi
# Bootstrap keyrings (required before pacstrap can verify packages)
+4 -28
View File
@@ -186,26 +186,9 @@ get_local_version() {
# Returns 0 (success) if should build, 1 if should skip
should_build_for_arch() {
local pkg="$1"
local current_arch="$ARCH"
local pkgdir=$(find_package_dir "$pkg")
local pkgbuild="$pkgdir/PKGBUILD"
[[ ! -f "$pkgbuild" ]] && return 1
# Check PKGBUILD arch=() array
local pkgbuild_archs=$(cd "$pkgdir" && bash -c 'source PKGBUILD 2>/dev/null; echo "${arch[@]}"')
# If arch=('any'), build for all architectures
if [[ "$pkgbuild_archs" == "any" ]]; then
return 0
fi
# Check if current arch is in PKGBUILD arch=()
if echo "$pkgbuild_archs" | grep -qw "$current_arch"; then
return 0 # Build
else
return 1 # Skip
fi
local pkgdir
pkgdir=$(find_package_dir "$pkg")
[[ -n "$pkgdir" ]] && package_supports_arch "$pkgdir" "$ARCH"
}
# For VCS packages, makepkg recalculates pkgver() before the build. If the
@@ -488,7 +471,7 @@ check_needs_build() {
# Collect packages that should be built for the selected mirror
collect_packages() {
packages_for_unscoped_build "$MIRROR"
packages_for_unscoped_build "$MIRROR" "$ARCH"
}
# Main execution
@@ -540,13 +523,6 @@ if [[ -n "$PACKAGES" ]]; then
else
# Build all packages that need updates from the relevant directories
while IFS= read -r pkg; do
# Check if package should be built for this architecture
if ! should_build_for_arch "$pkg"; then
echo " - $pkg - not built for $ARCH"
SKIPPED_PACKAGES="$SKIPPED_PACKAGES $pkg"
continue
fi
if check_needs_build "$pkg"; then
PACKAGES_TO_BUILD+=("$pkg")
else
+14 -6
View File
@@ -13,13 +13,21 @@ check_docker() {
fi
}
docker_native_arch() {
case "$(uname -m)" in
x86_64) echo x86_64 ;;
aarch64 | arm64) echo aarch64 ;;
*) return 1 ;;
esac
}
setup_qemu() {
# Setup QEMU for building ARM64 packages on x86_64 hosts
# Register emulators for builds whose target differs from the host.
if ! docker run --rm --privileged multiarch/qemu-user-static --reset -p yes --credential yes >/dev/null 2>&1; then
print_error "Failed to setup QEMU for ARM64 emulation"
print_error "Failed to set up QEMU emulation"
exit 1
fi
print_success "QEMU ARM64 emulation enabled"
print_success "QEMU emulation enabled"
}
build_docker_image() {
@@ -53,9 +61,9 @@ build_docker_image() {
get_platform_arg() {
local arch="$1"
case "$arch" in
x86_64) echo "--platform linux/amd64" ;;
aarch64) echo "--platform linux/arm64" ;;
*) echo "" ;;
x86_64) echo "--platform=linux/amd64" ;;
aarch64) echo "--platform=linux/arm64" ;;
*) return 1 ;;
esac
}
+48 -4
View File
@@ -143,6 +143,31 @@ package_has_pkgbuild() {
[[ -f "$pkgdir/PKGBUILD" ]]
}
# The architectures declared by a PKGBUILD. Set CARCH while reading it so a
# conditional arch=() assignment is evaluated for the architecture we are
# actually checking, even when the repository host is a different one.
package_arches() {
local pkgdir="$1"
local arch="${2:-${ARCH:-x86_64}}"
(cd "$pkgdir" && env -u OMARCHY_SRC CARCH="$arch" bash -c '
source PKGBUILD >/dev/null 2>&1
printf "%s\n" "${arch[*]}"
')
}
package_supports_arch() {
local pkgdir="$1"
local target="${2:-${ARCH:-x86_64}}"
local arches
arches=$(package_arches "$pkgdir" "$target") || return 1
case " $arches " in
*" any "* | *" $target "*) return 0 ;;
*) return 1 ;;
esac
}
# Channel membership: where a package may be published. Packages without a
# `channels` key are members of every channel (they flow edge -> rc -> stable).
package_has_channels() {
@@ -301,9 +326,12 @@ packages_for_mirror() {
packages_for_unscoped_build() {
local mirror="$1"
local arch="${2:-${ARCH:-x86_64}}"
package_dirs | while IFS= read -r pkgdir; do
if package_builds_for_mirror "$pkgdir" "$mirror" && ! package_build_skipped "$pkgdir"; then
if package_builds_for_mirror "$pkgdir" "$mirror" &&
! package_build_skipped "$pkgdir" &&
package_supports_arch "$pkgdir" "$arch"; then
basename "$pkgdir"
fi
done
@@ -502,12 +530,28 @@ validate_package_metadata() {
return 1
fi
if ! jq -e '(.rebuilt_against // {}) | type == "object" and (to_entries | all(.value | type == "string" and length > 0))' "$metadata" >/dev/null; then
echo "invalid rebuilt_against for $(basename "$pkgdir"): must be an object mapping package names to versions"
if ! jq -e '
def version_map:
type == "object" and (to_entries | all(.value | type == "string" and length > 0));
(.rebuilt_against // {}) as $record |
($record | version_map) or
(($record | type) == "object"
and ((($record | keys) - ["x86_64", "aarch64"]) | length == 0)
and ($record | to_entries | all(.value | version_map)))
' "$metadata" >/dev/null; then
echo "invalid rebuilt_against for $(basename "$pkgdir"): must map architectures to package-version maps"
return 1
fi
if ! jq -e '((.rebuilt_against // {}) | keys) - (.rebuild_on // []) | length == 0' "$metadata" >/dev/null; then
if ! jq -e '
(.rebuild_on // []) as $triggers |
(.rebuilt_against // {}) as $record |
if ($record | to_entries | all(.value | type == "string")) then
((($record | keys) - $triggers) | length == 0)
else
($record | to_entries | all((((.value | keys) - $triggers) | length) == 0))
end
' "$metadata" >/dev/null; then
echo "invalid rebuilt_against for $(basename "$pkgdir"): records a package that rebuild_on does not name"
return 1
fi
+2 -2
View File
@@ -13,8 +13,8 @@ VALID_ARCHES="x86_64 aarch64"
# in this order; the first entry is the reference architecture that the
# release train observes channels through. Adding an architecture here is the
# enablement step: the next check-versions tick queues its packages and the
# next auto-release tick builds them. OMARCHY_ARCHES overrides it for a host
# or a one-off run.
# next auto-release tick builds them. OMARCHY_ARCHES overrides it for a
# one-off run.
PUBLISHED_ARCHES="${OMARCHY_ARCHES:-x86_64}"
validate_arch() {
+1 -1
View File
@@ -7,7 +7,7 @@ Wants=network-online.target
Type=oneshot
ExecStart=/bin/bash -c 'source /root/.omarchy/build-credentials && /root/omarchy-pkgs/bin/auto-release edge'
Environment=OMARCHY_STATE_DIR=/root/.state
TimeoutStartSec=7200
TimeoutStartSec=43200
[Install]
WantedBy=multi-user.target
+1 -1
View File
@@ -12,7 +12,7 @@ Type=oneshot
# branch's worktree with OMARCHY_RC_PINS=1.
ExecStart=/bin/bash -c 'source /root/.omarchy/build-credentials && /root/omarchy-pkgs/bin/auto-release rc'
Environment=OMARCHY_STATE_DIR=/root/.state
TimeoutStartSec=7200
TimeoutStartSec=43200
[Install]
WantedBy=multi-user.target
+1 -1
View File
@@ -7,7 +7,7 @@ Wants=network-online.target
Type=oneshot
ExecStart=/bin/bash -c 'source /root/.omarchy/build-credentials && /root/omarchy-pkgs/bin/auto-release stable'
Environment=OMARCHY_STATE_DIR=/root/.state
TimeoutStartSec=7200
TimeoutStartSec=43200
[Install]
WantedBy=multi-user.target