From 76687fcc82eed3ee9ca59748e301d0b785d8f918 Mon Sep 17 00:00:00 2001 From: Ryan Hughes Date: Thu, 3 Sep 2026 23:46:57 -0400 Subject: [PATCH] Harden multi-architecture release pipeline --- .github/workflows/test.yml | 1 + README.md | 66 +++-- bin/advance-channel | 73 ++--- bin/auto-release | 15 +- bin/build | 9 +- bin/check-versions | 2 +- bin/clean-repo | 13 +- bin/omarchy-pkgs | 18 +- bin/omarchy-release | 125 +++++---- bin/remove-package | 15 +- bin/setup | 3 +- bin/sign | 15 +- bin/sync-rebuilds | 287 ++++++++++++++++---- bin/timers | 52 ++-- bin/update-repo | 16 +- build/Dockerfile | 4 +- build/build.sh | 32 +-- helpers/docker-helpers.sh | 20 +- helpers/package-metadata.sh | 52 +++- helpers/paths.sh | 4 +- systemd/omarchy-auto-release-edge.service | 2 +- systemd/omarchy-auto-release-rc.service | 2 +- systemd/omarchy-auto-release-stable.service | 2 +- 23 files changed, 557 insertions(+), 271 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 4aaea2b..b4b78b9 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -27,6 +27,7 @@ jobs: set -euo pipefail pacman -Syu --noconfirm jq ./bin/sync-upstream self-test + ./bin/sync-rebuilds --self-test ./bin/omarchy-pkgs self-test ./bin/omarchy-release self-test ' diff --git a/README.md b/README.md index ee876ba..c5338a0 100644 --- a/README.md +++ b/README.md @@ -57,14 +57,25 @@ file per channel and architecture (`.sync-needed--`); `auto-release ` works through the queues one architecture at a time, each with its own backoff (`.build-failed--`), so a failing build on one architecture never holds up the other; and the release -train advances channels with `--arch all`, so a channel never moves for one -architecture and not another. The first entry is the reference architecture -the release train observes channels through. +train advances channels with `--arch all`: it takes one host-wide lock and +verifies every architecture's source database before moving any of them. The +first entry is the reference architecture the release train observes channels +through. A remote sync failure can still leave a promotion temporarily partial; +rerunning the same advance completes it safely. -Adding an architecture is therefore one change to that list (or -`OMARCHY_ARCHES` in the host's build credentials): the next `check-versions` -tick queues everything the new architecture lacks, and the next -`auto-release` tick starts building it. The builder image bootstraps +Adding an architecture to the scheduled pipeline is therefore one checked-in +change to that list: the next `check-versions` tick queues everything the new +architecture lacks, and the next `auto-release` tick starts building it. A +checked-in list also means the rebuild workflow and release host cannot drift +onto different architecture sets. For a one-off run, override it directly: + +```bash +OMARCHY_ARCHES=x86_64 bin/check-versions +OMARCHY_ARCHES=aarch64 bin/check-versions +OMARCHY_ARCHES="x86_64 aarch64" bin/check-versions +``` + +The builder image bootstraps `omarchy-keyring` from the x86_64 tree for every architecture, so the first build of a new architecture does not depend on a repository that only it can create. @@ -450,7 +461,12 @@ A package names those dependencies in `.omarchy/package.json`: { "source": "aur", "sync": false, "rebuild_on": ["qt6-base", "qt6-declarative", "qt6-wayland"] } ``` -`bin/sync-rebuilds` reads each named package's version from the official repositories and compares it to `rebuilt_against`, the record of what the checked-in pkgrel was last bumped for. pkgrel is bumped unless every name in `rebuild_on` is recorded and still matches, so a name the record does not carry reads as changed rather than going unexamined forever. Opting a package in therefore buys one rebuild: what its published build actually linked against is not knowable from here, and a record written without a rebuild would certify a build nobody checked. +`bin/sync-rebuilds` reads each named package's version from the official +repositories for every published architecture the package supports and compares +it to `rebuilt_against`. Records are kept per architecture because Arch and +Arch Linux ARM can carry different dependency versions. pkgrel is bumped once +when any recorded version moves; that one source revision is then rebuilt by +each architecture's normal queue. The bump is the point of the command, and it has to land in git rather than in the builder. A rebuild that reuses the published version string produces a package pacman will never offer anyone, so merely unlocking the build gate would ship nothing. Bumping pkgrel needs no other change: `bin/check-versions` and the builder both already rebuild when pkgrel moves. @@ -458,9 +474,12 @@ For an AUR-synced package the bump is expressed as the dotted Omarchy pkgrel suf The bumped version is checked against the published one as well as the checked-in one, and refused when pacman would not order it higher. The checked-in version is not the floor; what a user already has is, and a checkout that has fallen behind the repository can otherwise be bumped to something that loses to the package it means to replace. That check is skipped with a warning when the published database cannot be read. -Versions are read from the local pacman database, so this runs on Arch or in an Arch container against a synced database. Only `core`, `extra` and `multilib` count: a Qt release sitting in testing or kde-unstable is not what the builder will link against, and rebuilding for it would ship a package built against the wrong ABI. The workflow points that database at `mirror.omarchy.org`, the mirror the x86_64 builder itself uses, because a mirror running ahead of the builder would record a version the build never linked against and nothing re-fires once the record matches. - -aarch64 is not covered. Those builds resolve Qt from Arch Linux ARM, which can lag Arch, so one record cannot describe both architectures. Only x86_64 is published today, so nothing currently ships from the untracked side; if ARM publishing starts, `rebuilt_against` has to become per-architecture before this can be trusted there. +x86_64 versions are read from the local pacman database, so the workflow runs +in an Arch container pointed at `mirror.omarchy.org`, the same mirror as the +x86_64 builder. aarch64 versions are read directly from the live Arch Linux ARM +repository database, which is also what the ARM builder uses. Testing and +staging repositories do not count. A legacy flat `rebuilt_against` record is +read as x86_64 and is migrated naturally the next time a rebuild is needed. ### Other @@ -668,7 +687,7 @@ Fields: - `skip_build`: optional boolean; defaults to `false`. Set `true` to exclude a package from scheduled version checks and unscoped builds. The package can still be built explicitly with `bin/repo release --package `. - `pkgrel`: optional Omarchy pkgrel suffix for a version-pinned rebuild bump. This emits `.` instead of replacing AUR's pkgrel. `offset` can be used only when preserving monotonic upgrades from old absolute pkgrel bumps. The metadata is removed automatically when AUR sync changes `pkgver`; the current package version is read from the checked-in PKGBUILD, so the version is not duplicated in JSON. - `rebuild_on`: optional array of package names this package links against closely enough that it must be rebuilt when they change, independent of its own source. Read by `bin/sync-rebuilds`. -- `rebuilt_against`: written by `bin/sync-rebuilds`. Records the version of each `rebuild_on` package that the current pkgrel was bumped for. +- `rebuilt_against`: written by `bin/sync-rebuilds`. Maps each published architecture to the versions of its `rebuild_on` packages that the current pkgrel was bumped for. - `upstream_commit`: set by `bin/sync-aur` for AUR packages. Used by `bin/package-worktree` to recreate the exact raw AUR package that Omarchy last synced. ### Build Matrix @@ -764,8 +783,10 @@ bin/repo release --package my-package ### aarch64 - Built on the repository host like x86_64; under QEMU when the host is x86_64 -- Uses Arch Linux ARM repositories (one mirrorlist for every channel — Arch - Linux ARM publishes no dated snapshots to pin a channel's base to) +- On an ARM host, package builds and the signing/database utility containers + run natively; only an explicitly requested x86_64 package build is emulated +- Uses Arch Linux ARM repositories through the same HTTPS mirror for every + channel (Arch Linux ARM publishes no dated snapshots to pin a channel's base) - Additional repos: `[alarm]`, `[aur]` - Same workflow, just add `--arch aarch64`; the scheduled pipeline runs it automatically once `aarch64` is in `PUBLISHED_ARCHES` @@ -835,10 +856,11 @@ That cadence is only safe because of three guards: an operator expects. `check-versions` takes it too — its `git pull` would otherwise swap PKGBUILDs out from under a running build. - **Backoff on failure.** A failed release records the attempt in - `.build-failed-` and backs off exponentially — 10m, 20m, 40m, up to + `.build-failed--` and backs off exponentially — 10m, 20m, 40m, up to a 6h ceiling — instead of rebuilding the same broken tree every 5 minutes. **Any new commit clears the backoff immediately**, since a push is the most - likely fix. Clear it by hand with `rm /root/.state/.build-failed-`. + likely fix. Clear it by hand with + `rm /root/.state/.build-failed--`. - **Quiet when idle.** With nothing queued a tick exits without output, so the journal shows the runs that mattered rather than 288 no-ops a day. @@ -891,10 +913,14 @@ bin/repo timers --local # inspect this machine instead ``` State files are stored in `/root/.state/`: -- `.sync-needed-` — the packages queued for that channel, one per - line; the release run reads them to name what it is building -- `.build-failed-` — consecutive failure count, timestamp, and the - commit it failed on (drives the backoff; removing it forces a retry) +- `.sync-needed--` — the packages queued for that channel and + architecture, one per line; the release run reads them to name what it is + building +- `.build-failed--` — consecutive failure count, timestamp, and + the commit it failed on (drives the backoff; removing it forces a retry) + +Legacy files without the architecture suffix are consumed once as x86_64 +state, so upgrading the host does not lose an in-flight build. ### Schedule (America/New_York) diff --git a/bin/advance-channel b/bin/advance-channel index da320da..1effbc6 100755 --- a/bin/advance-channel +++ b/bin/advance-channel @@ -25,31 +25,21 @@ SKIP_PROD_CHECK=false FAST_RING_ONLY=false BOOTSTRAP=false PACKAGES="" +ALL_ARCHES=false # Kept for --arch all, which re-invokes this script per architecture with the # other arguments unchanged (minus the --arch all pair itself). ORIGINAL_ARGS=() -skip_next=false -prev="" +arch_option=false for arg in "$@"; do - if [[ "$skip_next" == true ]]; then - skip_next=false - prev="" - continue + if [[ "$arch_option" == true ]]; then + [[ "$arg" != "all" ]] && ORIGINAL_ARGS+=("--arch" "$arg") + arch_option=false + elif [[ "$arg" == "--arch" ]]; then + arch_option=true + else + ORIGINAL_ARGS+=("$arg") fi - if [[ "$arg" == "--arch" ]]; then - prev="--arch" - continue - fi - if [[ "$prev" == "--arch" ]]; then - prev="" - if [[ "$arg" == "all" ]]; then - continue - fi - ORIGINAL_ARGS+=("--arch" "$arg") - continue - fi - ORIGINAL_ARGS+=("$arg") done usage() { @@ -93,21 +83,13 @@ while [[ $# -gt 0 ]]; do ;; --arch) if [[ "$2" == "all" ]]; then - # A release train advances every published architecture together; a - # channel that moved for one and not another would serve mismatched - # versions to users on the two. Re-run this script once per - # architecture with the same arguments, stopping at the first failure - # so the operator sees which architecture is stuck (the advance is - # idempotent, so re-running with --arch all resumes where it stopped). - shift 2 - for arch in $(published_arches); do - "$0" --arch "$arch" "${ORIGINAL_ARGS[@]}" || exit $? - done - exit 0 + ALL_ARCHES=true + ARCH=all + else + require_valid_arch "$2" + ARCH="$2" + update_arch_paths fi - require_valid_arch "$2" - ARCH="$2" - update_arch_paths shift 2 ;; --package) @@ -175,6 +157,31 @@ case "$FROM->$TO" in ;; esac +if [[ "$ALL_ARCHES" == true ]]; then + # Hold one lock across the whole operation so a timer cannot mutate a + # channel between architectures. Check every source database before moving + # the first one; a failed sync can still require an idempotent retry, but a + # missing architecture never creates a knowingly partial advance. + if [[ "$DRY_RUN" != true ]]; then + acquire_release_lock || exit 1 + fi + + ARCHES=$(published_arches) + for arch in $ARCHES; do + source_db="$REPO_ROOT/$FROM/$arch/omarchy.db.tar.zst" + if [[ ! -f "$source_db" ]]; then + print_error "Source database not found: $source_db" + echo "Nothing has been published to the $FROM channel for $arch." + exit 1 + fi + done + + for arch in $ARCHES; do + "$0" --arch "$arch" "${ORIGINAL_ARGS[@]}" || exit $? + done + exit 0 +fi + SOURCE_DIR="$REPO_ROOT/$FROM/$ARCH" TARGET_DIR="$REPO_ROOT/$TO/$ARCH" SOURCE_DB="$SOURCE_DIR/omarchy.db.tar.zst" diff --git a/bin/auto-release b/bin/auto-release index 10a8380..b7094bb 100755 --- a/bin/auto-release +++ b/bin/auto-release @@ -131,17 +131,28 @@ release_arch() { if "$BUILD_ROOT/bin/repo" release --mirror "$MIRROR" --arch "$arch" --skip-prod-check; then print_success "Release completed successfully for $MIRROR ($arch)" - rm -f "$state_file" "$fail_file" + local completed_state=("$state_file" "$fail_file") + if [[ "$arch" == "x86_64" ]]; then + completed_state+=("$(legacy_sync_queue_file "$MIRROR")" "$(legacy_sync_fail_file "$MIRROR")") + fi + if ! rm -f "${completed_state[@]}"; then + print_error "Release succeeded, but its queue state could not be cleared" + return 1 + fi print_success "State file removed: $state_file" return 0 fi fail_count=$((fail_count + 1)) - cat >"$fail_file" <"$fail_file" </dev/null 2>&1; then + if ! docker run --rm "$(get_platform_arg "$ARCH")" alpine:3.21 /bin/true >/dev/null 2>&1; then print_info "Setting up QEMU for $ARCH emulation on this $HOST_ARCH host..." setup_qemu fi @@ -183,7 +184,7 @@ if [[ $KEEP_BUILD_WORKSPACE == "1" ]]; then print_info "Keeping existing build workspace..." else print_info "Cleaning build workspace..." - rm -rf "$BUILD_OUTPUT_DIR"/* + rm -rf "${BUILD_OUTPUT_DIR:?}"/* fi mkdir -p "$BUILD_OUTPUT_DIR" @@ -237,7 +238,7 @@ DOCKER_ARGS=( # Run the builder with assembled args PLATFORM_ARG=$(get_platform_arg "$ARCH") -docker run $PLATFORM_ARG "${DOCKER_ARGS[@]}" "$IMAGE_TAG" /build/build.sh +docker run "$PLATFORM_ARG" "${DOCKER_ARGS[@]}" "$IMAGE_TAG" /build/build.sh BUILD_RESULT=$? diff --git a/bin/check-versions b/bin/check-versions index fda47d7..9849701 100755 --- a/bin/check-versions +++ b/bin/check-versions @@ -180,7 +180,7 @@ check_mirror() { needs_build=true packages+=("$pkg") fi - done < <(packages_for_unscoped_build "$mirror") + done < <(packages_for_unscoped_build "$mirror" "$ARCH") echo "" diff --git a/bin/clean-repo b/bin/clean-repo index 5b020af..8543a8a 100755 --- a/bin/clean-repo +++ b/bin/clean-repo @@ -31,12 +31,13 @@ clean_packages() { # Skip signature files [[ "$pkg" == *.sig ]] && continue - # Extract package name (remove version and architecture) - # Format: name-version-release-arch.pkg.tar.* - # The version starts at the first "-" and neither pkgver nor pkgrel - # can contain a hyphen, so the architecture is whatever sits between the - # last hyphen and .pkg.tar — any, x86_64, aarch64, and whatever comes next. - local pkgname=$(echo "$pkg" | sed -E 's/-[0-9]+.*-[^-]+\.pkg\.tar\..*//') + # Format: name-version-release-arch.pkg.tar.*. Work from the right because + # package names can themselves contain version-like pieces (qt6-5compat, + # nvidia-580xx-utils), while pkgver and pkgrel cannot contain hyphens. + local stem="${pkg%%.pkg.tar.*}" + stem="${stem%-*}" # architecture + stem="${stem%-*}" # pkgrel + local pkgname="${stem%-*}" # pkgver # Add to array if [[ -n "${packages[$pkgname]}" ]]; then diff --git a/bin/omarchy-pkgs b/bin/omarchy-pkgs index 58ff646..0ca607b 100755 --- a/bin/omarchy-pkgs +++ b/bin/omarchy-pkgs @@ -23,7 +23,7 @@ source "$BUILD_ROOT/helpers/paths.sh" source "$BUILD_ROOT/helpers/host-helpers.sh" UPSTREAM_URL="${OMARCHY_UPSTREAM_URL:-https://github.com/basecamp/omarchy.git}" -EDGE_DB_URL="${OMARCHY_EDGE_DB_URL:-https://pkgs.omarchy.org/edge/x86_64/omarchy.db.tar.zst}" +EDGE_DB_URL="${OMARCHY_EDGE_DB_URL:-https://pkgs.omarchy.org/edge/$(reference_arch)/omarchy.db.tar.zst}" RELEASE_PACKAGES=(omarchy omarchy-settings) DEFAULT_RC_REF="quattro" SRCDEST_DIR="$BUILD_ROOT/.srcdest" @@ -327,13 +327,19 @@ regenerate_checksums() { trigger_build_host() { local host + queue_edge_builds() { + mkdir -p "$STATE_DIR" || return 1 + local arch + for arch in $(published_arches); do + touch "$(sync_queue_file edge "$arch")" || return 1 + done + } + # Explicit host configuration outranks the local-host inference (a # workstation that ran a full local release carries the db marker too). if ! resolve_repo_host "${REPO_HOST_OVERRIDE:-}" >/dev/null && on_repo_host; then print_info "Triggering edge build locally (this is the build host)..." - if mkdir -p "${OMARCHY_STATE_DIR:-/root/.state}" && - touch "${OMARCHY_STATE_DIR:-/root/.state}/.sync-needed-edge" && - systemctl start --no-block omarchy-auto-release-edge.service; then + if queue_edge_builds && systemctl start --no-block omarchy-auto-release-edge.service; then print_success "Edge build triggered" else print_warning "Could not start the edge release service — the 6-hourly timer will pick it up" @@ -343,11 +349,11 @@ trigger_build_host() { if ! host=$(resolve_repo_host "${REPO_HOST_OVERRIDE:-}"); then print_info "No repository host configured (set OMARCHY_REPO_HOST or $BUILD_ROOT/.repo-host — any ssh destination, e.g. root@ or an ssh-config alias)." print_info "The 6-hourly auto-release timer will pick this up, or trigger manually:" - echo " ssh 'git -C /root/omarchy-pkgs pull --ff-only && touch /root/.state/.sync-needed-edge && systemctl start omarchy-auto-release-edge.service'" + echo " ssh 'git -C /root/omarchy-pkgs pull --ff-only && systemctl start omarchy-check-versions.service omarchy-auto-release-edge.service'" return 0 fi print_info "Triggering edge build on $host..." - if ssh "$host" 'git -C /root/omarchy-pkgs pull --ff-only && mkdir -p /root/.state && touch /root/.state/.sync-needed-edge && systemctl start --no-block omarchy-auto-release-edge.service'; then + if ssh "$host" 'git -C /root/omarchy-pkgs pull --ff-only && cd /root/omarchy-pkgs && export BUILD_ROOT=/root/omarchy-pkgs && source helpers/paths.sh && mkdir -p "$STATE_DIR" && for arch in $(published_arches); do touch "$(sync_queue_file edge "$arch")"; done && systemctl start --no-block omarchy-auto-release-edge.service'; then print_success "Edge build triggered on $host" else print_warning "Could not trigger $host — the 6-hourly timer will pick it up" diff --git a/bin/omarchy-release b/bin/omarchy-release index 3fa88a3..762bd80 100755 --- a/bin/omarchy-release +++ b/bin/omarchy-release @@ -28,11 +28,9 @@ ISO_REPO="${OMARCHY_ISO_REPO:-omacom-io/omarchy-iso}" DEV_BRANCH="${OMARCHY_DEV_BRANCH:-quattro}" PKGS_DB_BASE="${OMARCHY_PKGS_DB_BASE:-https://pkgs.omarchy.org}" -# The train observes channels through the reference architecture (the first -# published one). The pair is pinned to one version for every architecture and -# built for all of them in one rc trigger, so one architecture's database is -# the state of the release; per-architecture drift is a build failure the -# scheduled pipeline reports, not something the train has to poll for. +# The first published architecture supplies the version-ordering floor when +# cutting pins. Readiness checks below still verify every published +# architecture before an RC or final release can move forward. OBSERVED_ARCH=$(reference_arch) RC_DB_URL="$PKGS_DB_BASE/rc/$OBSERVED_ARCH/omarchy.db.tar.zst" @@ -187,12 +185,12 @@ ensure_work_clone() { # Prints omarchy's published version in a channel; empty when absent, rc 2 when # the database cannot be read (callers must not mistake an outage for absence). published_version() { - local channel="$1" tmp descs + local channel="$1" arch="${2:-$OBSERVED_ARCH}" tmp descs tmp=$(mktemp) || return 2 # A unique query string busts the CDN cache: right after a sync the plain # URL can keep serving the previous db for a while, which reads as "not # published yet" to status, the wait loop, and ship's pre-checks. - if ! curl -sf "$PKGS_DB_BASE/$channel/$OBSERVED_ARCH/omarchy.db.tar.zst?$(date +%s%N)" -o "$tmp"; then + if ! curl -sf "$PKGS_DB_BASE/$channel/$arch/omarchy.db.tar.zst?$(date +%s%N)" -o "$tmp"; then rm -f "$tmp" return 2 fi @@ -213,6 +211,14 @@ published_version() { ' <<<"$descs" } +all_arches_at_version() { # all_arches_at_version + local channel="$1" want="$2" arch got + for arch in $(published_arches); do + got=$(published_version "$channel" "$arch" 2>/dev/null) || return 1 + [[ "${got%-*}" == "$want" ]] || return 1 + done +} + # The rc branch of THIS repo carries the current pins. Read them without # touching the working tree. rc_branch_pin() { # prints "pkgver commit", empty when no rc branch @@ -297,12 +303,16 @@ host_advance() { # host_advance [extra args...] } wait_for_published() { # wait_for_published [timeout-seconds] - local channel="$1" want="$2" timeout="${3:-3600}" waited=0 got - print_info "Waiting for $want to appear in the $channel channel (up to $((timeout / 60))m)..." + local channel="$1" want="$2" timeout="${3:-3600}" waited=0 arch got pending + print_info "Waiting for $want in $channel for: $(published_arches | tr '\n' ' ')" while ((waited < timeout)); do - got=$(published_version "$channel" 2>/dev/null) || got="" - if [[ "${got%-*}" == "$want" ]]; then - print_success "$channel now serves omarchy $got" + pending="" + for arch in $(published_arches); do + got=$(published_version "$channel" "$arch" 2>/dev/null) || got="" + [[ "${got%-*}" == "$want" ]] || pending+=" $arch=${got:-unreachable}" + done + if [[ -z "$pending" ]]; then + print_success "$channel now serves omarchy $want on every published architecture" return 0 fi sleep 60 @@ -310,7 +320,7 @@ wait_for_published() { # wait_for_published [timeout-seconds printf '.' >&2 done echo "" >&2 - print_warning "Timed out waiting for $want in $channel (currently: ${got:-unknown})" + print_warning "Timed out waiting for $want in $channel (pending:$pending)" print_info "The build may still be running — re-run this command to resume." return 1 } @@ -674,15 +684,13 @@ cmd_rc() { if [[ -n "$pin" ]]; then local pin_ver="${pin%% *}" pin_commit="${pin##* }" if [[ "$pin_commit" == "$head" && "$pin_ver" =~ ^${version//./\\.}rc[0-9]+$ ]]; then - local pub - pub=$(published_version rc 2>/dev/null) || pub="" - if [[ "${pub%-*}" == "$pin_ver" ]]; then - print_success "$pin_ver is already cut from this head and published to rc" + if all_arches_at_version rc "$pin_ver"; then + print_success "$pin_ver is already cut from this head and published to rc on every architecture" maybe_iso "$pin_ver" rc "$iso_mode" return 0 fi print_info "$pin_ver is pinned from this head but not published yet — re-triggering the build" - trigger_rc_build || true + trigger_rc_build || return 1 [[ "$wait" == true ]] && wait_for_published rc "$pin_ver" maybe_iso "$pin_ver" rc "$iso_mode" return 0 @@ -698,7 +706,7 @@ cmd_rc() { new_ver="${new_pin%% *}" print_success "Pinned $new_ver (rc branch pushed)" - trigger_rc_build || true + trigger_rc_build || return 1 if [[ "$wait" == true ]]; then wait_for_published rc "$new_ver" || return 1 fi @@ -720,9 +728,7 @@ cmd_ship() { exit 1 fi version=$(branch_to_version "$branch") - local stable_now - stable_now=$(published_version stable 2>/dev/null) || stable_now="" - if [[ "${stable_now%-*}" == "$version" ]] && + if all_arches_at_version stable "$version" && gh release view "v$version" --repo "$UPSTREAM_REPO" >/dev/null 2>&1; then print_success "Nothing to ship — $version is tagged, released, and live on stable" exit 0 @@ -764,10 +770,8 @@ cmd_ship() { echo " omarchy-release rc" exit 1 fi - local pub - pub=$(published_version rc 2>/dev/null) || pub="" - if [[ "${pub%-*}" != "$pin_ver" ]]; then - print_error "$pin_ver is pinned but rc serves '${pub:-nothing}' — the candidate build hasn't published" + if ! all_arches_at_version rc "$pin_ver"; then + print_error "$pin_ver is pinned but is not published for every architecture" echo "Wait for it (or re-run: omarchy-release rc), then ship." exit 1 fi @@ -811,10 +815,9 @@ cmd_ship() { # 2. Final pins into rc. Resolve the tag we just established so the final # PKGBUILDs record both its provenance and its exact commit. - local rc_pub stable_pub - rc_pub=$(published_version rc 2>/dev/null) || rc_pub="" - if [[ "${rc_pub%-*}" == "$version" ]]; then - print_success "2/7 Final $version already published to rc" + local stable_pub + if all_arches_at_version rc "$version"; then + print_success "2/7 Final $version already published to rc on every architecture" else if [[ "$pin_ver" != "$version" ]]; then print_info "2/7 Pinning final $version from tag v$version..." @@ -822,22 +825,21 @@ cmd_ship() { else print_info "2/7 Final $version pinned — re-triggering build" fi - trigger_rc_build || true + trigger_rc_build || exit 1 wait_for_published rc "$version" || exit 1 fi # 3. Promote rc -> stable - stable_pub=$(published_version stable 2>/dev/null) || stable_pub="" - if [[ "${stable_pub%-*}" == "$version" ]]; then - print_success "3/7 Stable already serves $version" + if all_arches_at_version stable "$version"; then + print_success "3/7 Stable already serves $version on every architecture" else host_advance rc stable || exit 1 - stable_pub=$(published_version stable 2>/dev/null) || stable_pub="" - if [[ "${stable_pub%-*}" != "$version" ]]; then - print_error "Promotion ran but stable serves '${stable_pub:-nothing}' — investigate before continuing" + if ! all_arches_at_version stable "$version"; then + print_error "Promotion ran but stable does not serve $version on every architecture" exit 1 fi - print_success "3/7 Promoted to stable: omarchy $stable_pub" + stable_pub=$(published_version stable 2>/dev/null) || stable_pub="$version" + print_success "3/7 Promoted to stable: omarchy $stable_pub on every architecture" fi # 4. Final pins onto master (keeps edge overlap publishing and the repo record) @@ -920,7 +922,7 @@ next_step() { # prints "|" last=$(newest_release_branch 2>/dev/null) || last="" if [[ -n "$last" && "$STABLE_VER" != "" ]]; then last_ver=$(branch_to_version "$last") - if [[ "${STABLE_VER%-*}" != "$last_ver" ]] || + if ! all_arches_at_version stable "$last_ver" || { command -v gh >/dev/null && ! gh release view "v$last_ver" --repo "$UPSTREAM_REPO" >/dev/null 2>&1; }; then echo "ship|$last_ver is tagged but not fully shipped — resume ship" return @@ -936,7 +938,7 @@ next_step() { # prints "|" echo "ship|Final $TRAIN_VER is pinned — finish shipping (re-runs are safe)" elif [[ "$pin_commit" != "$TRAIN_HEAD" ]]; then echo "rc|$TRAIN has commits newer than $pin_ver — cut the next candidate" - elif [[ "${RC_VER%-*}" != "$pin_ver" ]]; then + elif ! all_arches_at_version rc "$pin_ver"; then echo "rc|$pin_ver is pinned but not published — re-run rc to re-trigger/wait" else echo "ship|$pin_ver is published to rc — test it, then ship" @@ -1006,20 +1008,24 @@ cmd_doctor() { check "makepkg available (checksums)" command -v makepkg check "curl available" command -v curl check "upstream reachable ($UPSTREAM_URL)" git ls-remote "$UPSTREAM_URL" HEAD - local ch + local ch arch for ch in edge stable; do - if published_version "$ch" >/dev/null 2>&1; then - print_success "$ch channel db readable" + for arch in $(published_arches); do + if published_version "$ch" "$arch" >/dev/null 2>&1; then + print_success "$ch/$arch channel db readable" + else + print_error "$ch/$arch channel db readable" + failures=$((failures + 1)) + fi + done + done + for arch in $(published_arches); do + if published_version rc "$arch" >/dev/null 2>&1; then + print_success "rc/$arch channel db readable" else - print_error "$ch channel db readable" - failures=$((failures + 1)) + print_warning "rc/$arch channel db not readable — bootstrap it first: bin/repo bootstrap-rc (on the build host)" fi done - if published_version rc >/dev/null 2>&1; then - print_success "rc channel db readable" - else - print_warning "rc channel db not readable — bootstrap it first: bin/repo bootstrap-rc (on the build host)" - fi local host if host=$(repo_host); then check "build host ssh ($host)" ssh -o ConnectTimeout=10 "$host" true @@ -1066,6 +1072,25 @@ cmd_self_test() { expect "version_is_patch 5.0.0" "$(version_is_patch 5.0.0 && echo yes || echo no)" "no" expect "previous_patch_tag 4.0.2" "$(previous_patch_tag 4.0.2)" "v4.0.1" expect "previous_patch_tag 4.0.10" "$(previous_patch_tag 4.0.10)" "v4.0.9" + + # Keep release readiness fail-closed when only one architecture has reached + # the requested version. This replaces the network reader for this process; + # self-test exits immediately afterwards. + published_version() { + case "$2" in + x86_64) echo "${TEST_X86_VERSION:-4.0.2-1}" ;; + aarch64) echo "${TEST_ARM_VERSION:-4.0.2-1}" ;; + esac + } + PUBLISHED_ARCHES=x86_64 + expect "x86-only readiness" "$(all_arches_at_version rc 4.0.2 && echo ready || echo blocked)" "ready" + PUBLISHED_ARCHES=aarch64 + expect "ARM-only readiness" "$(all_arches_at_version rc 4.0.2 && echo ready || echo blocked)" "ready" + PUBLISHED_ARCHES="x86_64 aarch64" + TEST_ARM_VERSION=4.0.1-1 + expect "mixed versions block release" "$(all_arches_at_version rc 4.0.2 && echo ready || echo blocked)" "blocked" + TEST_ARM_VERSION=4.0.2-1 + expect "both architectures ready" "$(all_arches_at_version rc 4.0.2 && echo ready || echo blocked)" "ready" echo "" if ((failures == 0)); then print_success "Self-test passed" diff --git a/bin/remove-package b/bin/remove-package index 6c79837..eedcfb0 100755 --- a/bin/remove-package +++ b/bin/remove-package @@ -88,9 +88,12 @@ if [[ ! $REPLY =~ ^[Yy]$ ]]; then exit 0 fi -# Build/update the Docker image (always use x86_64 for removal - it's architecture independent) -# repo-remove is mirror-independent — always use the edge x86_64 image -build_docker_image "$BUILD_DIR" "x86_64" "edge" +# repo-remove is architecture-independent, so use a host-native edge image. +TOOL_ARCH=$(docker_native_arch) || { + print_error "Unsupported host architecture: $(uname -m)" + exit 1 +} +build_docker_image "$BUILD_DIR" "$TOOL_ARCH" "edge" acquire_release_lock || exit 1 @@ -99,13 +102,13 @@ print_info "Removing package..." # Ensure directory is writable by container user make_dir_writable "$REPO_DIR" -# Run the removal script in Docker (always use x86_64 image) -docker run --rm --platform linux/amd64 \ +# Run the removal script in the host-native image. +docker run --rm "$(get_platform_arg "$TOOL_ARCH")" \ -e ARCH="$ARCH" \ -e MIRROR="$MIRROR" \ -v "$REPO_ROOT:/pkgs.omarchy.org" \ -v "$BUILD_DIR:/build:ro" \ - omarchy-pkg-builder:latest-x86_64-edge /build/remove-package.sh "$PACKAGE_NAME" + "omarchy-pkg-builder:latest-$TOOL_ARCH-edge" /build/remove-package.sh "$PACKAGE_NAME" RESULT=$? diff --git a/bin/setup b/bin/setup index 014dd2f..6727d96 100755 --- a/bin/setup +++ b/bin/setup @@ -15,6 +15,7 @@ set -e BUILD_ROOT=$(realpath "${BASH_SOURCE[0]%/*}/..") source "$BUILD_ROOT/helpers/message-helpers.sh" +source "$BUILD_ROOT/helpers/paths.sh" CHECK_ONLY=false SKIP_TIMERS=false @@ -238,7 +239,7 @@ echo "" # --- release timers ---------------------------------------------------------- print_info "Published architectures: $(published_arches | tr '\n' ' ')" -echo " (PUBLISHED_ARCHES in helpers/paths.sh, or OMARCHY_ARCHES in $CREDENTIALS)" +echo " (PUBLISHED_ARCHES in helpers/paths.sh; OMARCHY_ARCHES overrides a one-off command)" echo "" TIMERS=(omarchy-check-versions omarchy-auto-release-edge omarchy-auto-release-rc omarchy-auto-release-stable) diff --git a/bin/sign b/bin/sign index 70779e8..c517b9b 100755 --- a/bin/sign +++ b/bin/sign @@ -66,23 +66,28 @@ if [[ -z "$GPG_PASSPHRASE" ]]; then exit 1 fi -# Build/update the Docker image (always use x86_64 for signing - it's architecture independent) -build_docker_image "$BUILD_DIR" "x86_64" "$MIRROR" +# Signing is architecture-independent, so run its utility container natively +# on either an x86_64 or ARM host. +TOOL_ARCH=$(docker_native_arch) || { + print_error "Unsupported host architecture: $(uname -m)" + exit 1 +} +build_docker_image "$BUILD_DIR" "$TOOL_ARCH" "$MIRROR" print_info "Running package signing..." # Ensure output directory is writable by container user make_dir_writable "$BUILD_OUTPUT_DIR" -# Run the signing script in Docker (always use x86_64 image) -docker run --rm --platform linux/amd64 \ +# Run the signing script in the host-native image. +docker run --rm "$(get_platform_arg "$TOOL_ARCH")" \ -e ARCH="$ARCH" \ -e MIRROR="$MIRROR" \ -e GPG_PRIVATE_KEY="$GPG_PRIVATE_KEY" \ -e GPG_PASSPHRASE="$GPG_PASSPHRASE" \ -v "$BUILD_ROOT/build-output:/build-output" \ -v "$BUILD_DIR:/build:ro" \ - omarchy-pkg-builder:latest-x86_64-$MIRROR /build/sign.sh + "omarchy-pkg-builder:latest-$TOOL_ARCH-$MIRROR" /build/sign.sh SIGN_RESULT=$? diff --git a/bin/sync-rebuilds b/bin/sync-rebuilds index 7a0336b..b9aa81d 100755 --- a/bin/sync-rebuilds +++ b/bin/sync-rebuilds @@ -17,11 +17,14 @@ SELF_TEST=false # rebuilding for it would ship a package built against the wrong ABI. OFFICIAL_REPOS=" core extra multilib core-debug extra-debug " -# The published repository, used as the floor a bumped pkgrel has to clear. -# Only x86_64 is published today; aarch64 has no repository to compare against. +# The published repositories are the floor a bumped pkgrel has to clear. PUBLISHED_BASE_URL="${OMARCHY_PUBLISHED_BASE_URL:-https://pkgs.omarchy.org}" PUBLISHED_MIRRORS=(edge stable) -PUBLISHED_ARCH=x86_64 + +# Arch Linux ARM has no dated snapshots. This is the same live repository the +# aarch64 builder resolves; override it only when the builder mirror changes. +ALARM_BASE_URL="${OMARCHY_ALARM_BASE_URL:-https://fl.us.mirror.archlinuxarm.org/aarch64}" +ALARM_REPOS=(core extra alarm aur) usage() { cat </dev/null; then + print_error "Could not read the aarch64 $repo repository database" + return 1 + fi + if ! tar -tf "$db" >/dev/null 2>&1; then + print_error "Unreadable aarch64 $repo repository database" + return 1 + fi + + tar -xOf "$db" --wildcards '*/desc' 2>/dev/null | awk ' + function emit() { + if (name != "" && version != "") { + print name "\t" version + if (base != "" && base != name) print base "\t" version + } + name=""; base=""; version="" + } + $0 == "%FILENAME%" { emit(); next } + $0 == "%NAME%" { if (name != "" && version != "") emit(); getline; name=$0; next } + $0 == "%BASE%" { getline; base=$0; next } + $0 == "%VERSION%" { getline; version=$0; next } + END { emit() } + ' >"$index" +} + +alarm_repo_version() { + local package="$1" repo version + for repo in "${ALARM_REPOS[@]}"; do + load_alarm_repo "$repo" || return 1 + version=$(awk -F '\t' -v package="$package" '$1 == package { print $2; exit }' "$TEMP_DIR/alarm-$repo.index") + if [[ -n "$version" ]]; then + echo "$version" + return 0 + fi + done +} + +repo_version() { # repo_version + case "$1" in + x86_64) native_repo_version "$2" ;; + aarch64) alarm_repo_version "$2" ;; + *) return 1 ;; + esac +} + declare -A PUBLISHED_VERSION=() PUBLISHED_LOADED=false -PUBLISHED_AVAILABLE=true remember_published() { local name="$1" @@ -131,34 +188,35 @@ load_published_versions() { [[ "$PUBLISHED_LOADED" == true ]] && return 0 PUBLISHED_LOADED=true - local mirror db name base version + local arch mirror db name base version - for mirror in "${PUBLISHED_MIRRORS[@]}"; do - db="$TEMP_DIR/published-$mirror.db.tar.zst" + for arch in $(published_arches); do + for mirror in "${PUBLISHED_MIRRORS[@]}"; do + db="$TEMP_DIR/published-$mirror-$arch.db.tar.zst" - if ! curl -fsSL --max-time 120 -o "$db" \ - "$PUBLISHED_BASE_URL/$mirror/$PUBLISHED_ARCH/omarchy.db.tar.zst" 2>/dev/null; then - print_warning "Could not read the published $mirror database; bumps are not checked against it this run" - PUBLISHED_AVAILABLE=false - continue - fi + if ! curl -fsSL --max-time 120 -o "$db" \ + "$PUBLISHED_BASE_URL/$mirror/$arch/omarchy.db.tar.zst" 2>/dev/null; then + print_warning "Could not read the published $mirror/$arch database; bumps are not checked against it this run" + continue + fi - while IFS=$'\t' read -r name base version; do - [[ -n "$name" && -n "$version" ]] && remember_published "$name" "$version" - [[ -n "$base" && -n "$version" ]] && remember_published "$base" "$version" - done < <( - tar -xOf "$db" --wildcards '*/desc' 2>/dev/null | awk ' - function emit() { - if (name != "" && version != "") print name "\t" base "\t" version - name=""; base=""; version="" - } - $0 == "%FILENAME%" { emit(); next } - $0 == "%NAME%" { if (name != "" && version != "") emit(); getline; name=$0; next } - $0 == "%BASE%" { getline; base=$0; next } - $0 == "%VERSION%" { getline; version=$0; next } - END { emit() } - ' - ) + while IFS=$'\t' read -r name base version; do + [[ -n "$name" && -n "$version" ]] && remember_published "$name" "$version" + [[ -n "$base" && -n "$version" ]] && remember_published "$base" "$version" + done < <( + tar -xOf "$db" --wildcards '*/desc' 2>/dev/null | awk ' + function emit() { + if (name != "" && version != "") print name "\t" base "\t" version + name=""; base=""; version="" + } + $0 == "%FILENAME%" { emit(); next } + $0 == "%NAME%" { if (name != "" && version != "") emit(); getline; name=$0; next } + $0 == "%BASE%" { getline; base=$0; next } + $0 == "%VERSION%" { getline; version=$0; next } + END { emit() } + ' + ) + done done } @@ -252,7 +310,17 @@ record_triggers() { local package_dir="$1" local current="$2" - write_metadata "$package_dir" '.rebuilt_against = $current' --argjson current "$current" + # A flat record is the legacy x86_64 shape. Preserve records for + # architectures outside this run, then replace the ones just rebuilt. + write_metadata "$package_dir" ' + (.rebuilt_against // {}) as $old | + (if ($old | length) == 0 then {} + elif ($old | to_entries | all(.value | type == "string")) + then {x86_64: $old} + else $old + end) as $by_arch | + .rebuilt_against = ($by_arch * $current) + ' --argjson current "$current" } # Metadata that does not parse would otherwise drop its package out of the run @@ -300,21 +368,41 @@ sync_package() { print_info "Checking $package against ${triggers[*]}..." - local current="{}" trigger version - for trigger in "${triggers[@]}"; do - version=$(repo_version "$trigger") - if [[ -z "$version" ]]; then - print_error " $trigger is in no official repository; leaving $package alone" - ((++FAILED)) - return 0 - fi - if ! current=$(jq -c --arg name "$trigger" --arg version "$version" '.[$name] = $version' <<<"$current"); then - print_error " Could not record $trigger $version for $package" + local current="{}" arch arch_current trigger version considered=0 + for arch in $(published_arches); do + package_supports_arch "$package_dir" "$arch" || continue + considered=$((considered + 1)) + arch_current="{}" + for trigger in "${triggers[@]}"; do + if ! version=$(repo_version "$arch" "$trigger"); then + print_error " Could not read $arch repository versions; leaving $package alone" + ((++FAILED)) + return 0 + fi + if [[ -z "$version" ]]; then + print_error " $trigger is in no $arch repository; leaving $package alone" + ((++FAILED)) + return 0 + fi + if ! arch_current=$(jq -c --arg name "$trigger" --arg version "$version" '.[$name] = $version' <<<"$arch_current"); then + print_error " Could not record $arch/$trigger $version for $package" + ((++FAILED)) + return 0 + fi + done + if ! current=$(jq -c --arg arch "$arch" --argjson versions "$arch_current" '.[$arch] = $versions' <<<"$current"); then + print_error " Could not record $arch trigger versions for $package" ((++FAILED)) return 0 fi done + if ((considered == 0)); then + print_info " Skipping: not built for any published architecture" + ((++SKIPPED)) + return 0 + fi + local recorded if ! recorded=$(package_metadata_value "$package_dir" '.rebuilt_against' ""); then print_error " Could not read .omarchy/package.json for $package" @@ -323,13 +411,20 @@ sync_package() { fi [[ -n "$recorded" && "$recorded" != "null" ]] || recorded="{}" + # Before architecture-specific records existed, rebuilt_against described + # x86_64. Read it that way without forcing a metadata-only migration. + if jq -e 'to_entries | all(.value | type == "string")' >/dev/null <<<"$recorded"; then + recorded=$(jq -c '{x86_64: .}' <<<"$recorded") + fi + # Walk the declared triggers rather than the record, so a name the record does # not carry reads as changed instead of going unexamined forever. local moved if ! moved=$(jq -r --argjson recorded "$recorded" ' - to_entries - | map(select($recorded[.key] != .value) - | "\(.key) \($recorded[.key] // "unrecorded") -> \(.value)") + [to_entries[] as $arch + | $arch.value | to_entries[] as $trigger + | select($recorded[$arch.key][$trigger.key] != $trigger.value) + | "\($arch.key)/\($trigger.key) \($recorded[$arch.key][$trigger.key] // "unrecorded") -> \($trigger.value)"] | join(", ") ' <<<"$current"); then print_error " Could not compare recorded trigger versions for $package" @@ -338,7 +433,7 @@ sync_package() { fi if [[ -z "$moved" ]]; then - print_info " Already rebuilt against $(jq -r 'to_entries | map("\(.key) \(.value)") | join(", ")' <<<"$current")" + print_info " Already rebuilt against every published architecture" ((++SKIPPED)) return 0 fi @@ -466,11 +561,11 @@ selftest_root() { } selftest_package() { - local root="$1" name="$2" pkgrel="$3" metadata="$4" pkgver="${5:-1.0}" + local root="$1" name="$2" pkgrel="$3" metadata="$4" pkgver="${5:-1.0}" arches="${6:-x86_64}" local dir="$root/pkgbuilds/$name" mkdir -p "$dir/.omarchy" - printf 'pkgname=%s\npkgver=%s\npkgrel=%s\narch=(x86_64)\n' "$name" "$pkgver" "$pkgrel" > "$dir/PKGBUILD" + printf 'pkgname=%s\npkgver=%s\npkgrel=%s\narch=(%s)\n' "$name" "$pkgver" "$pkgrel" "$arches" > "$dir/PKGBUILD" printf '%s\n' "$metadata" > "$dir/.omarchy/package.json" } @@ -514,19 +609,51 @@ selftest_published() { cat > "$root/stub/curl" <<'STUB' #!/bin/bash out="" +url="" while [[ $# -gt 0 ]]; do case "$1" in -o) out="$2"; shift 2 ;; - *) shift ;; + *) url="$1"; shift ;; esac done -db="$(dirname "$0")/omarchy.db.tar.zst" +if [[ "$url" == */aarch64/* ]]; then + repo="${url%/*}" + repo="${repo##*/}" + db="$(dirname "$0")/alarm-$repo.db" +else + db="$(dirname "$0")/omarchy.db.tar.zst" +fi [[ -f "$db" && -n "$out" ]] || exit 22 cp "$db" "$out" STUB chmod +x "$root/stub/curl" } +selftest_alarm() { + local root="$1" + shift + local repo staging="$root/stub/alarm-db" entry name version + + for repo in core extra; do + rm -rf "$staging" + mkdir -p "$staging" + if [[ "$repo" == "core" ]]; then + mkdir -p "$staging/unrelated-1-1" + printf '%%FILENAME%%\nunrelated-1-1-aarch64.pkg.tar.zst\n\n%%NAME%%\nunrelated\n\n%%BASE%%\nunrelated\n\n%%VERSION%%\n1-1\n' \ + > "$staging/unrelated-1-1/desc" + else + for entry in "$@"; do + name="${entry%=*}" + version="${entry#*=}" + mkdir -p "$staging/$name-$version" + printf '%%FILENAME%%\n%s-%s-aarch64.pkg.tar.zst\n\n%%NAME%%\n%s\n\n%%BASE%%\n%s\n\n%%VERSION%%\n%s\n' \ + "$name" "$version" "$name" "$name" "$version" > "$staging/$name-$version/desc" + done + fi + tar -czf "$root/stub/alarm-$repo.db" -C "$staging" . + done +} + cmd_self_test() { local failures=0 local root @@ -545,7 +672,8 @@ cmd_self_test() { local root="$1" shift local status=0 - PATH="$root/stub:$PATH" "$root/bin/sync-rebuilds" "$@" > "$root/output" 2>&1 || status=$? + OMARCHY_ARCHES="${SELFTEST_ARCHES:-x86_64}" \ + PATH="$root/stub:$PATH" "$root/bin/sync-rebuilds" "$@" > "$root/output" 2>&1 || status=$? echo "$status" } @@ -563,7 +691,7 @@ cmd_self_test() { check "run succeeds" 0 "$(run_case "$root")" check "pkgrel bumped" 2 "$(pkgrel_of "$root/pkgbuilds/t-partial")" check "unrecorded trigger now recorded" "2-2" \ - "$(jq -r '.rebuilt_against["dep-b"]' "$root/pkgbuilds/t-partial/.omarchy/package.json")" + "$(jq -r '.rebuilt_against.x86_64["dep-b"]' "$root/pkgbuilds/t-partial/.omarchy/package.json")" echo "Opting a package in buys a rebuild rather than a bare record:" root=$(selftest_root fresh) @@ -573,7 +701,7 @@ cmd_self_test() { check "run succeeds" 0 "$(run_case "$root")" check "pkgrel bumped" 2 "$(pkgrel_of "$root/pkgbuilds/t-fresh")" check "trigger recorded" "1-1" \ - "$(jq -r '.rebuilt_against["dep-a"]' "$root/pkgbuilds/t-fresh/.omarchy/package.json")" + "$(jq -r '.rebuilt_against.x86_64["dep-a"]' "$root/pkgbuilds/t-fresh/.omarchy/package.json")" echo "An unchanged package is left alone:" root=$(selftest_root current) @@ -612,6 +740,47 @@ cmd_self_test() { check "suffix recorded for the next AUR sync" 1 \ "$(jq -r '.pkgrel.suffix' "$root/pkgbuilds/t-aur/.omarchy/package.json")" + echo "A dependency is tracked independently for both published architectures:" + root=$(selftest_root multiarch) + selftest_package "$root" t-multi 1 '{"source":"local","rebuild_on":["dep-a"],"rebuilt_against":{"dep-a":"1-1"}}' 1.0 'x86_64 aarch64' + selftest_pacman "$root" dep-a=1-1 + selftest_published "$root" + selftest_alarm "$root" dep-a=2-1 + SELFTEST_ARCHES="x86_64 aarch64" + check "run succeeds" 0 "$(run_case "$root")" + unset SELFTEST_ARCHES + check "pkgrel bumped once" 2 "$(pkgrel_of "$root/pkgbuilds/t-multi")" + check "x86_64 trigger recorded" "1-1" \ + "$(jq -r '.rebuilt_against.x86_64["dep-a"]' "$root/pkgbuilds/t-multi/.omarchy/package.json")" + check "aarch64 trigger recorded" "2-1" \ + "$(jq -r '.rebuilt_against.aarch64["dep-a"]' "$root/pkgbuilds/t-multi/.omarchy/package.json")" + + echo "An ARM-only run records only the ARM dependency state:" + root=$(selftest_root arm-only) + selftest_package "$root" t-arm 1 '{"source":"local","rebuild_on":["dep-a"]}' 1.0 'x86_64 aarch64' + selftest_pacman "$root" + selftest_published "$root" + selftest_alarm "$root" dep-a=2-1 + SELFTEST_ARCHES=aarch64 + check "run succeeds" 0 "$(run_case "$root")" + unset SELFTEST_ARCHES + check "pkgrel bumped" 2 "$(pkgrel_of "$root/pkgbuilds/t-arm")" + check "ARM trigger recorded" "2-1" \ + "$(jq -r '.rebuilt_against.aarch64["dep-a"]' "$root/pkgbuilds/t-arm/.omarchy/package.json")" + check "x86_64 was not consulted" "false" \ + "$(jq -r '.rebuilt_against | has("x86_64")' "$root/pkgbuilds/t-arm/.omarchy/package.json")" + + echo "An x86-only package ignores ARM during a dual-architecture run:" + root=$(selftest_root x86-package) + selftest_package "$root" t-x86 1 '{"source":"local","rebuild_on":["dep-a"],"rebuilt_against":{"dep-a":"1-1"}}' + selftest_pacman "$root" dep-a=1-1 + selftest_published "$root" + selftest_alarm "$root" dep-a=2-1 + SELFTEST_ARCHES="x86_64 aarch64" + check "run succeeds" 0 "$(run_case "$root")" + unset SELFTEST_ARCHES + check "pkgrel untouched" 1 "$(pkgrel_of "$root/pkgbuilds/t-x86")" + echo "" if [[ "$failures" -eq 0 ]]; then print_success "Self-test passed" @@ -626,9 +795,9 @@ if [[ "$SELF_TEST" == true ]]; then exit $? fi -for tool in pacman vercmp jq curl; do +for tool in pacman vercmp jq curl tar; do if ! command -v "$tool" >/dev/null 2>&1; then - print_error "$tool not found: reading trigger versions and ordering pkgrels both need pacman" + print_error "$tool not found: rebuild trigger sync cannot run" exit 1 fi done diff --git a/bin/timers b/bin/timers index 1cb2f4b..4b39a29 100755 --- a/bin/timers +++ b/bin/timers @@ -105,32 +105,32 @@ echo "" paused=false for channel in edge rc stable; do for arch in $(published_arches); do - fail_file=$(sync_fail_file "$channel" "$arch") - [[ "$arch" == "x86_64" && ! -f "$fail_file" ]] && fail_file=$(legacy_sync_fail_file "$channel") - [[ -f "$fail_file" ]] || continue - if [[ "$paused" == false ]]; then - print_error "Failing builds (backoff active)" - paused=true - fi - FAILURE_COUNT=0 FAILURE_AT=0 FAILURE_FINGERPRINT="" - # shellcheck disable=SC1090 - source "$fail_file" 2>/dev/null || true - delay=600 - for ((i = 1; i < FAILURE_COUNT; i++)); do - delay=$((delay * 2)) - ((delay >= 21600)) && { delay=21600; break; } - done - retry_at=$((FAILURE_AT + delay)) - now=$(date +%s) - if ((now < retry_at)); then - when="retries at $(date -d "@$retry_at" '+%H:%M:%S' 2>/dev/null || echo "+$((retry_at - now))s")" - else - when="retries on the next tick" - fi - printf ' ✗ %-7s %-8s %s consecutive failure(s), %s\n' "$channel" "$arch" "$FAILURE_COUNT" "$when" - printf ' last attempt %s on commit %s\n' \ - "$(date -d "@$FAILURE_AT" '+%Y-%m-%d %H:%M:%S' 2>/dev/null || echo "$FAILURE_AT")" \ - "${FAILURE_FINGERPRINT:0:12}" + fail_file=$(sync_fail_file "$channel" "$arch") + [[ "$arch" == "x86_64" && ! -f "$fail_file" ]] && fail_file=$(legacy_sync_fail_file "$channel") + [[ -f "$fail_file" ]] || continue + if [[ "$paused" == false ]]; then + print_error "Failing builds (backoff active)" + paused=true + fi + FAILURE_COUNT=0 FAILURE_AT=0 FAILURE_FINGERPRINT="" + # shellcheck disable=SC1090 + source "$fail_file" 2>/dev/null || true + delay=600 + for ((i = 1; i < FAILURE_COUNT; i++)); do + delay=$((delay * 2)) + ((delay >= 21600)) && { delay=21600; break; } + done + retry_at=$((FAILURE_AT + delay)) + now=$(date +%s) + if ((now < retry_at)); then + when="retries at $(date -d "@$retry_at" '+%H:%M:%S' 2>/dev/null || echo "+$((retry_at - now))s")" + else + when="retries on the next tick" + fi + printf ' ✗ %-7s %-8s %s consecutive failure(s), %s\n' "$channel" "$arch" "$FAILURE_COUNT" "$when" + printf ' last attempt %s on commit %s\n' \ + "$(date -d "@$FAILURE_AT" '+%Y-%m-%d %H:%M:%S' 2>/dev/null || echo "$FAILURE_AT")" \ + "${FAILURE_FINGERPRINT:0:12}" done done if [[ "$paused" == true ]]; then diff --git a/bin/update-repo b/bin/update-repo index 34d9f51..894e197 100755 --- a/bin/update-repo +++ b/bin/update-repo @@ -28,17 +28,21 @@ update_database() { # Make output directory writable for container make_dir_writable "$REPO_DIR" - # repo-add is architecture- and mirror-independent, so always use the edge - # x86_64 image. This also lets bootstrap-rc build the rc database before the - # rc channel exists remotely (an rc image can only build after it does). - build_docker_image "$BUILD_DIR" "x86_64" "edge" + # repo-add is architecture- and mirror-independent. Use the host-native edge + # image, which also lets bootstrap-rc run before that channel exists remotely. + local tool_arch + tool_arch=$(docker_native_arch) || { + print_error "Unsupported host architecture: $(uname -m)" + exit 1 + } + build_docker_image "$BUILD_DIR" "$tool_arch" "edge" - docker run --rm --platform linux/amd64 \ + docker run --rm "$(get_platform_arg "$tool_arch")" \ -e ARCH="$ARCH" \ -e MIRROR="$MIRROR" \ -v "$REPO_ROOT:/output" \ -v "$BUILD_DIR:/build:ro" \ - omarchy-pkg-builder:latest-x86_64-edge /build/update-repo.sh + "omarchy-pkg-builder:latest-$tool_arch-edge" /build/update-repo.sh } # Main execution diff --git a/build/Dockerfile b/build/Dockerfile index 258fec4..af1bb5e 100644 --- a/build/Dockerfile +++ b/build/Dockerfile @@ -45,9 +45,7 @@ RUN if [ "${TARGETARCH}" = "amd64" ]; then \ printf 'Server = https://mirror.omarchy.org/$repo/os/$arch\n' > /etc/pacman.d/mirrorlist; \ fi; \ else \ - curl -L "https://raw.githubusercontent.com/archlinuxarm/PKGBUILDs/master/core/pacman-mirrorlist/mirrorlist" 2>/dev/null | \ - sed -E 's/^\s*#\s*Server\s*=/Server =/g' > /etc/pacman.d/mirrorlist && \ - sed -i 's/\$arch/aarch64/g' /etc/pacman.d/mirrorlist; \ + printf 'Server = https://fl.us.mirror.archlinuxarm.org/aarch64/$repo\n' > /etc/pacman.d/mirrorlist; \ fi # Bootstrap keyrings (required before pacstrap can verify packages) diff --git a/build/build.sh b/build/build.sh index e0f69ac..8534ee9 100755 --- a/build/build.sh +++ b/build/build.sh @@ -186,26 +186,9 @@ get_local_version() { # Returns 0 (success) if should build, 1 if should skip should_build_for_arch() { local pkg="$1" - local current_arch="$ARCH" - local pkgdir=$(find_package_dir "$pkg") - local pkgbuild="$pkgdir/PKGBUILD" - - [[ ! -f "$pkgbuild" ]] && return 1 - - # Check PKGBUILD arch=() array - local pkgbuild_archs=$(cd "$pkgdir" && bash -c 'source PKGBUILD 2>/dev/null; echo "${arch[@]}"') - - # If arch=('any'), build for all architectures - if [[ "$pkgbuild_archs" == "any" ]]; then - return 0 - fi - - # Check if current arch is in PKGBUILD arch=() - if echo "$pkgbuild_archs" | grep -qw "$current_arch"; then - return 0 # Build - else - return 1 # Skip - fi + local pkgdir + pkgdir=$(find_package_dir "$pkg") + [[ -n "$pkgdir" ]] && package_supports_arch "$pkgdir" "$ARCH" } # For VCS packages, makepkg recalculates pkgver() before the build. If the @@ -488,7 +471,7 @@ check_needs_build() { # Collect packages that should be built for the selected mirror collect_packages() { - packages_for_unscoped_build "$MIRROR" + packages_for_unscoped_build "$MIRROR" "$ARCH" } # Main execution @@ -540,13 +523,6 @@ if [[ -n "$PACKAGES" ]]; then else # Build all packages that need updates from the relevant directories while IFS= read -r pkg; do - # Check if package should be built for this architecture - if ! should_build_for_arch "$pkg"; then - echo " - $pkg - not built for $ARCH" - SKIPPED_PACKAGES="$SKIPPED_PACKAGES $pkg" - continue - fi - if check_needs_build "$pkg"; then PACKAGES_TO_BUILD+=("$pkg") else diff --git a/helpers/docker-helpers.sh b/helpers/docker-helpers.sh index 4a26904..c108340 100644 --- a/helpers/docker-helpers.sh +++ b/helpers/docker-helpers.sh @@ -13,13 +13,21 @@ check_docker() { fi } +docker_native_arch() { + case "$(uname -m)" in + x86_64) echo x86_64 ;; + aarch64 | arm64) echo aarch64 ;; + *) return 1 ;; + esac +} + setup_qemu() { - # Setup QEMU for building ARM64 packages on x86_64 hosts + # Register emulators for builds whose target differs from the host. if ! docker run --rm --privileged multiarch/qemu-user-static --reset -p yes --credential yes >/dev/null 2>&1; then - print_error "Failed to setup QEMU for ARM64 emulation" + print_error "Failed to set up QEMU emulation" exit 1 fi - print_success "QEMU ARM64 emulation enabled" + print_success "QEMU emulation enabled" } build_docker_image() { @@ -53,9 +61,9 @@ build_docker_image() { get_platform_arg() { local arch="$1" case "$arch" in - x86_64) echo "--platform linux/amd64" ;; - aarch64) echo "--platform linux/arm64" ;; - *) echo "" ;; + x86_64) echo "--platform=linux/amd64" ;; + aarch64) echo "--platform=linux/arm64" ;; + *) return 1 ;; esac } diff --git a/helpers/package-metadata.sh b/helpers/package-metadata.sh index 9096103..7382bbf 100644 --- a/helpers/package-metadata.sh +++ b/helpers/package-metadata.sh @@ -143,6 +143,31 @@ package_has_pkgbuild() { [[ -f "$pkgdir/PKGBUILD" ]] } +# The architectures declared by a PKGBUILD. Set CARCH while reading it so a +# conditional arch=() assignment is evaluated for the architecture we are +# actually checking, even when the repository host is a different one. +package_arches() { + local pkgdir="$1" + local arch="${2:-${ARCH:-x86_64}}" + + (cd "$pkgdir" && env -u OMARCHY_SRC CARCH="$arch" bash -c ' + source PKGBUILD >/dev/null 2>&1 + printf "%s\n" "${arch[*]}" + ') +} + +package_supports_arch() { + local pkgdir="$1" + local target="${2:-${ARCH:-x86_64}}" + local arches + + arches=$(package_arches "$pkgdir" "$target") || return 1 + case " $arches " in + *" any "* | *" $target "*) return 0 ;; + *) return 1 ;; + esac +} + # Channel membership: where a package may be published. Packages without a # `channels` key are members of every channel (they flow edge -> rc -> stable). package_has_channels() { @@ -301,9 +326,12 @@ packages_for_mirror() { packages_for_unscoped_build() { local mirror="$1" + local arch="${2:-${ARCH:-x86_64}}" package_dirs | while IFS= read -r pkgdir; do - if package_builds_for_mirror "$pkgdir" "$mirror" && ! package_build_skipped "$pkgdir"; then + if package_builds_for_mirror "$pkgdir" "$mirror" && + ! package_build_skipped "$pkgdir" && + package_supports_arch "$pkgdir" "$arch"; then basename "$pkgdir" fi done @@ -502,12 +530,28 @@ validate_package_metadata() { return 1 fi - if ! jq -e '(.rebuilt_against // {}) | type == "object" and (to_entries | all(.value | type == "string" and length > 0))' "$metadata" >/dev/null; then - echo "invalid rebuilt_against for $(basename "$pkgdir"): must be an object mapping package names to versions" + if ! jq -e ' + def version_map: + type == "object" and (to_entries | all(.value | type == "string" and length > 0)); + (.rebuilt_against // {}) as $record | + ($record | version_map) or + (($record | type) == "object" + and ((($record | keys) - ["x86_64", "aarch64"]) | length == 0) + and ($record | to_entries | all(.value | version_map))) + ' "$metadata" >/dev/null; then + echo "invalid rebuilt_against for $(basename "$pkgdir"): must map architectures to package-version maps" return 1 fi - if ! jq -e '((.rebuilt_against // {}) | keys) - (.rebuild_on // []) | length == 0' "$metadata" >/dev/null; then + if ! jq -e ' + (.rebuild_on // []) as $triggers | + (.rebuilt_against // {}) as $record | + if ($record | to_entries | all(.value | type == "string")) then + ((($record | keys) - $triggers) | length == 0) + else + ($record | to_entries | all((((.value | keys) - $triggers) | length) == 0)) + end + ' "$metadata" >/dev/null; then echo "invalid rebuilt_against for $(basename "$pkgdir"): records a package that rebuild_on does not name" return 1 fi diff --git a/helpers/paths.sh b/helpers/paths.sh index c78db22..5d6191c 100644 --- a/helpers/paths.sh +++ b/helpers/paths.sh @@ -13,8 +13,8 @@ VALID_ARCHES="x86_64 aarch64" # in this order; the first entry is the reference architecture that the # release train observes channels through. Adding an architecture here is the # enablement step: the next check-versions tick queues its packages and the -# next auto-release tick builds them. OMARCHY_ARCHES overrides it for a host -# or a one-off run. +# next auto-release tick builds them. OMARCHY_ARCHES overrides it for a +# one-off run. PUBLISHED_ARCHES="${OMARCHY_ARCHES:-x86_64}" validate_arch() { diff --git a/systemd/omarchy-auto-release-edge.service b/systemd/omarchy-auto-release-edge.service index 7658dfb..3537ed9 100644 --- a/systemd/omarchy-auto-release-edge.service +++ b/systemd/omarchy-auto-release-edge.service @@ -7,7 +7,7 @@ Wants=network-online.target Type=oneshot ExecStart=/bin/bash -c 'source /root/.omarchy/build-credentials && /root/omarchy-pkgs/bin/auto-release edge' Environment=OMARCHY_STATE_DIR=/root/.state -TimeoutStartSec=7200 +TimeoutStartSec=43200 [Install] WantedBy=multi-user.target diff --git a/systemd/omarchy-auto-release-rc.service b/systemd/omarchy-auto-release-rc.service index 0078b3d..c57f5f7 100644 --- a/systemd/omarchy-auto-release-rc.service +++ b/systemd/omarchy-auto-release-rc.service @@ -12,7 +12,7 @@ Type=oneshot # branch's worktree with OMARCHY_RC_PINS=1. ExecStart=/bin/bash -c 'source /root/.omarchy/build-credentials && /root/omarchy-pkgs/bin/auto-release rc' Environment=OMARCHY_STATE_DIR=/root/.state -TimeoutStartSec=7200 +TimeoutStartSec=43200 [Install] WantedBy=multi-user.target diff --git a/systemd/omarchy-auto-release-stable.service b/systemd/omarchy-auto-release-stable.service index e1979e3..110bad3 100644 --- a/systemd/omarchy-auto-release-stable.service +++ b/systemd/omarchy-auto-release-stable.service @@ -7,7 +7,7 @@ Wants=network-online.target Type=oneshot ExecStart=/bin/bash -c 'source /root/.omarchy/build-credentials && /root/omarchy-pkgs/bin/auto-release stable' Environment=OMARCHY_STATE_DIR=/root/.state -TimeoutStartSec=7200 +TimeoutStartSec=43200 [Install] WantedBy=multi-user.target