2 Commits
Author SHA1 Message Date
Ryan Hughes 71e72e581b Run the timers every 5 minutes, with overlap and failure guards
A push reaching the mirror should take minutes, not up to six hours. All four
units now fire every 5 minutes, staggered a minute apart. Three guards make
that cadence safe:

- Scheduled runs take the release lock NON-BLOCKING (try_release_lock) and
  skip the tick when a build is running. Blocking would stack one stalled
  process per tick behind a long build and stampede when it finished. Manual
  commands still wait, as an operator expects.
- check-versions takes the lock too, and now owns its git pull (--pull, passed
  by the unit) instead of an ExecStartPre: at this cadence an unlocked pull
  would swap PKGBUILDs out from under a running build.
- A failed release records .build-failed-<channel> and backs off
  exponentially (10m, 20m, 40m … capped at 6h) rather than rebuilding the same
  broken tree every 5 minutes. Any new commit clears the backoff, since a push
  is the most likely fix.

Idle ticks exit without output so the journal keeps showing the runs that
matter, and bin/repo timers reports backoff state — a paused channel is
otherwise indistinguishable from an idle one.
2026-08-27 01:10:33 -04:00
Ryan Hughes 9d86123264 Add bin/repo timers: release timer and queue status at a glance
Wraps systemctl list-timers with the things you actually want when checking on
the build host: per-unit enabled state, last run and whether it succeeded,
which channels have builds queued (state files), whether the release lock is
held by a live process, and any failed units. Units are discovered from
systemd/*.timer so the report cannot drift from what setup installs.

It forwards over ssh like the other host commands, so the build box's timer
state is one command away from a workstation (--local to inspect this machine).
2026-08-27 01:10:33 -04:00