Run the timers every 5 minutes, with overlap and failure guards

A push reaching the mirror should take minutes, not up to six hours. All four
units now fire every 5 minutes, staggered a minute apart. Three guards make
that cadence safe:

- Scheduled runs take the release lock NON-BLOCKING (try_release_lock) and
  skip the tick when a build is running. Blocking would stack one stalled
  process per tick behind a long build and stampede when it finished. Manual
  commands still wait, as an operator expects.
- check-versions takes the lock too, and now owns its git pull (--pull, passed
  by the unit) instead of an ExecStartPre: at this cadence an unlocked pull
  would swap PKGBUILDs out from under a running build.
- A failed release records .build-failed-<channel> and backs off
  exponentially (10m, 20m, 40m … capped at 6h) rather than rebuilding the same
  broken tree every 5 minutes. Any new commit clears the backoff, since a push
  is the most likely fix.

Idle ticks exit without output so the journal keeps showing the runs that
matter, and bin/repo timers reports backoff state — a paused channel is
otherwise indistinguishable from an idle one.
This commit is contained in:
Ryan Hughes committed 2026-08-27 01:10:33 -04:00
1 parent 9d86123264
commit 71e72e581b
10 files changed
+243 -57

No files matched your search

+19
View File
@@ -8,14 +8,33 @@ BUILD_ROOT=$(realpath "${BASH_SOURCE[0]%/*}/..")
source "$BUILD_ROOT/helpers/message-helpers.sh"
source "$BUILD_ROOT/helpers/paths.sh"
source "$BUILD_ROOT/helpers/package-metadata.sh"
source "$BUILD_ROOT/helpers/lock-helpers.sh"
STATE_DIR="${OMARCHY_STATE_DIR:-/root/.state}"
ARCH="${ARCH:-x86_64}"
PULL=false
[[ "${1:-}" == "--pull" ]] && PULL=true
mkdir -p "$STATE_DIR"
# Runs on a tight schedule, so it must never disturb a build in progress:
# comparing versions reads the PKGBUILDs a running build is reading, and
# --pull would swap them underneath it. Take the lock non-blocking and skip
# the tick when a release owns it — the queue is unchanged, so the next tick
# picks up exactly where this one left off.
if ! try_release_lock; then
exit 0
fi
print_header "Package Version Check"
# The scheduled run pulls; a manual run leaves the operator's tree alone.
if [[ "$PULL" == true ]]; then
print_info "Updating repository..."
git -C "$BUILD_ROOT" pull --ff-only || print_warning "git pull failed — checking the current tree"
fi
get_repo_version() {
local pkg="$1"
local mirror="$2"