Run the timers every 5 minutes, with overlap and failure guards
A push reaching the mirror should take minutes, not up to six hours. All four units now fire every 5 minutes, staggered a minute apart. Three guards make that cadence safe: - Scheduled runs take the release lock NON-BLOCKING (try_release_lock) and skip the tick when a build is running. Blocking would stack one stalled process per tick behind a long build and stampede when it finished. Manual commands still wait, as an operator expects. - check-versions takes the lock too, and now owns its git pull (--pull, passed by the unit) instead of an ExecStartPre: at this cadence an unlocked pull would swap PKGBUILDs out from under a running build. - A failed release records .build-failed-<channel> and backs off exponentially (10m, 20m, 40m … capped at 6h) rather than rebuilding the same broken tree every 5 minutes. Any new commit clears the backoff, since a push is the most likely fix. Idle ticks exit without output so the journal keeps showing the runs that matter, and bin/repo timers reports backoff state — a paused channel is otherwise indistinguishable from an idle one.
This commit is contained in:
1 parent
9d86123264
commit
71e72e581b
10 files changed
+243
-57
No files matched your search
+35
@@ -97,6 +97,41 @@ done
|
||||
[[ "$queued" == false ]] && echo " (nothing queued — all channels up to date)"
|
||||
echo ""
|
||||
|
||||
# A channel in backoff looks identical to an idle one from the outside, so
|
||||
# say so plainly: it is queued but deliberately not being retried yet.
|
||||
paused=false
|
||||
for channel in edge rc stable; do
|
||||
fail_file="$STATE_DIR/.build-failed-$channel"
|
||||
[[ -f "$fail_file" ]] || continue
|
||||
if [[ "$paused" == false ]]; then
|
||||
print_error "Failing builds (backoff active)"
|
||||
paused=true
|
||||
fi
|
||||
FAILURE_COUNT=0 FAILURE_AT=0 FAILURE_FINGERPRINT=""
|
||||
# shellcheck disable=SC1090
|
||||
source "$fail_file" 2>/dev/null || true
|
||||
delay=600
|
||||
for ((i = 1; i < FAILURE_COUNT; i++)); do
|
||||
delay=$((delay * 2))
|
||||
((delay >= 21600)) && { delay=21600; break; }
|
||||
done
|
||||
retry_at=$((FAILURE_AT + delay))
|
||||
now=$(date +%s)
|
||||
if ((now < retry_at)); then
|
||||
when="retries at $(date -d "@$retry_at" '+%H:%M:%S' 2>/dev/null || echo "+$((retry_at - now))s")"
|
||||
else
|
||||
when="retries on the next tick"
|
||||
fi
|
||||
printf ' ✗ %-7s %s consecutive failure(s), %s\n' "$channel" "$FAILURE_COUNT" "$when"
|
||||
printf ' last attempt %s on commit %s\n' \
|
||||
"$(date -d "@$FAILURE_AT" '+%Y-%m-%d %H:%M:%S' 2>/dev/null || echo "$FAILURE_AT")" \
|
||||
"${FAILURE_FINGERPRINT:0:12}"
|
||||
done
|
||||
if [[ "$paused" == true ]]; then
|
||||
echo " Any new commit clears the backoff; or: rm $STATE_DIR/.build-failed-<channel>"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# --- release lock ------------------------------------------------------------
|
||||
|
||||
print_info "Release lock"
|
||||
|
||||
Reference in new issue
Block a user