Run the timers every 5 minutes, with overlap and failure guards

A push reaching the mirror should take minutes, not up to six hours. All four
units now fire every 5 minutes, staggered a minute apart. Three guards make
that cadence safe:

- Scheduled runs take the release lock NON-BLOCKING (try_release_lock) and
  skip the tick when a build is running. Blocking would stack one stalled
  process per tick behind a long build and stampede when it finished. Manual
  commands still wait, as an operator expects.
- check-versions takes the lock too, and now owns its git pull (--pull, passed
  by the unit) instead of an ExecStartPre: at this cadence an unlocked pull
  would swap PKGBUILDs out from under a running build.
- A failed release records .build-failed-<channel> and backs off
  exponentially (10m, 20m, 40m … capped at 6h) rather than rebuilding the same
  broken tree every 5 minutes. Any new commit clears the backoff, since a push
  is the most likely fix.

Idle ticks exit without output so the journal keeps showing the runs that
matter, and bin/repo timers reports backoff state — a paused channel is
otherwise indistinguishable from an idle one.
This commit is contained in:
Ryan Hughes committed 2026-08-27 01:10:33 -04:00
1 parent 9d86123264
commit 71e72e581b
10 files changed
+243 -57

No files matched your search

+35
View File
@@ -97,6 +97,41 @@ done
[[ "$queued" == false ]] && echo " (nothing queued — all channels up to date)"
echo ""
# A channel in backoff looks identical to an idle one from the outside, so
# say so plainly: it is queued but deliberately not being retried yet.
paused=false
for channel in edge rc stable; do
fail_file="$STATE_DIR/.build-failed-$channel"
[[ -f "$fail_file" ]] || continue
if [[ "$paused" == false ]]; then
print_error "Failing builds (backoff active)"
paused=true
fi
FAILURE_COUNT=0 FAILURE_AT=0 FAILURE_FINGERPRINT=""
# shellcheck disable=SC1090
source "$fail_file" 2>/dev/null || true
delay=600
for ((i = 1; i < FAILURE_COUNT; i++)); do
delay=$((delay * 2))
((delay >= 21600)) && { delay=21600; break; }
done
retry_at=$((FAILURE_AT + delay))
now=$(date +%s)
if ((now < retry_at)); then
when="retries at $(date -d "@$retry_at" '+%H:%M:%S' 2>/dev/null || echo "+$((retry_at - now))s")"
else
when="retries on the next tick"
fi
printf ' ✗ %-7s %s consecutive failure(s), %s\n' "$channel" "$FAILURE_COUNT" "$when"
printf ' last attempt %s on commit %s\n' \
"$(date -d "@$FAILURE_AT" '+%Y-%m-%d %H:%M:%S' 2>/dev/null || echo "$FAILURE_AT")" \
"${FAILURE_FINGERPRINT:0:12}"
done
if [[ "$paused" == true ]]; then
echo " Any new commit clears the backoff; or: rm $STATE_DIR/.build-failed-<channel>"
echo ""
fi
# --- release lock ------------------------------------------------------------
print_info "Release lock"