ZacharyZhang-NY/omarchy-pkgs
1
0 Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Publish in a minute: build outside the lock, sign on a hosted runner (#869)

Browse Source 
A one-package merge took 9 to 15 minutes to publish for about 15 seconds of
signing and upload. The run-wide concurrency group made each merge wait for
every earlier run, builds included (#854 waited 13 minutes behind an aarch64
batch), and the publish job waited about 3 minutes for a builder droplet
even when every package had a PR artifact and nothing needed building.

Build x86_64 trees that have no artifact in the rebuild job, one droplet per
entry, as aarch64 already builds there on arm64 runners: in parallel, with
no secrets, and outside any lock. The publish job now only collects
artifacts, signs and uploads, on ubuntu-latest with the GHCR builder image
(#850), and only it holds the publish group.
This commit is contained in:
Ryan Hughes authored and GitHub committed 2026-10-08 15:03:28 -04:00
1 parent 93b1655ca8
commit ebd2d6a766
2 files changed
+49 -50
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/publish.yml
+44 -48
View File
@@ -24,11 +24,10 @@ on:
description: "Space-separated package directories to publish from master" description: "Space-separated package directories to publish from master"
required: true required: true
# Merges serialize. Two publishes into one channel at once would race on # Only the publish job serializes (see its concurrency group): two publishes
# the database; queued is fine, cancelled is not. # into one channel at once would race on the database. Builds run outside the
concurrency: # lock, so a merge waits behind another merge's signing and upload, seconds,
group: publish # never behind its kernel build.
cancel-in-progress: false
jobs: jobs:
changes: changes:
@@ -58,11 +57,10 @@ jobs:
echo "matrix=$matrix" >> "$GITHUB_OUTPUT" echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
echo "count=$(jq '.include | length' <<<"$matrix")" >> "$GITHUB_OUTPUT" echo "count=$(jq '.include | length' <<<"$matrix")" >> "$GITHUB_OUTPUT"
jq -r '.include[] | "\(.package) \(.arch) -> \(.channels)"' <<<"$matrix" jq -r '.include[] | "\(.package) \(.arch) -> \(.channels)"' <<<"$matrix"
# Reuse or rebuild, decided per entry and said out loud. An aarch64 # Reuse or rebuild, decided per entry and said out loud. A tree with
# tree with no build artifact (PR artifacts last 7 days; a dispatch # no build artifact (PR artifacts last 7 days; a dispatch may name
# may name any package) goes to the rebuild job, which builds it # any package) goes to the rebuild job: aarch64 natively on GitHub's
# natively on GitHub's arm64 runner. x86_64 builds inside the # arm64 runner, x86_64 on a builder droplet, one runner per entry.
# publish job on the droplet, as before.
rebuild=() rebuild=()
echo "| package | arch | tree | decision |" >> "$GITHUB_STEP_SUMMARY" echo "| package | arch | tree | decision |" >> "$GITHUB_STEP_SUMMARY"
echo "| --- | --- | --- | --- |" >> "$GITHUB_STEP_SUMMARY" echo "| --- | --- | --- | --- |" >> "$GITHUB_STEP_SUMMARY"
@@ -77,9 +75,10 @@ jobs:
decision="reuse the build artifact ($found)" decision="reuse the build artifact ($found)"
elif [[ $arch == aarch64 ]]; then elif [[ $arch == aarch64 ]]; then
decision="no build artifact: rebuild natively on ubuntu-24.04-arm" decision="no build artifact: rebuild natively on ubuntu-24.04-arm"
rebuild+=("$entry") rebuild+=("$(jq -c '. + {runner: "[\"ubuntu-24.04-arm\"]"}' <<<"$entry")")
else else
decision="no build artifact: build in the publish job on the self-hosted builder" decision="no build artifact: rebuild on a builder droplet"
rebuild+=("$(jq -c '. + {runner: "[\"self-hosted\",\"omarchy-builder\"]"}' <<<"$entry")")
fi fi
echo "==> $label: $decision" echo "==> $label: $decision"
echo "| $package | $arch | ${hash:0:12} | $decision |" >> "$GITHUB_STEP_SUMMARY" echo "| $package | $arch | ${hash:0:12} | $decision |" >> "$GITHUB_STEP_SUMMARY"
@@ -87,16 +86,19 @@ jobs:
echo "rebuild=$(printf '%s\n' "${rebuild[@]}" | jq -sc '{include: .}')" >> "$GITHUB_OUTPUT" echo "rebuild=$(printf '%s\n' "${rebuild[@]}" | jq -sc '{include: .}')" >> "$GITHUB_OUTPUT"
echo "rebuild_count=${#rebuild[@]}" >> "$GITHUB_OUTPUT" echo "rebuild_count=${#rebuild[@]}" >> "$GITHUB_OUTPUT"
# The aarch64 half of "build it now when there is none". It builds exactly # "Build it now when there is none". Each entry builds exactly as
# as build-pr.yml's aarch64 path does (same runner, same builder image, # build-pr.yml builds it (same runner kind, same builder image, same
# same bin/build call) and uploads under the same label, so the publish # bin/build call) and uploads under the same label, so the publish job
# job collects this run's artifact the way it collects a PR's. No secret # collects this run's artifact the way it collects a PR's. No secret
# reaches this runner; signing and upload stay on the self-hosted builder. # reaches these runners, and they hold no lock: entries build in parallel,
# and other merges publish while they do.
rebuild: rebuild:
needs: changes needs: changes
if: needs.changes.outputs.rebuild_count != '0' if: needs.changes.outputs.rebuild_count != '0'
runs-on: ubuntu-24.04-arm runs-on: ${{ fromJSON(matrix.runner) }}
timeout-minutes: 180 # The droplets are x86, so aarch64 never builds there. omarchy-mac-boot
# under QEMU took 2h47m; native arm64 and x86 kernels fit easily.
timeout-minutes: 240
permissions: permissions:
contents: read contents: read
strategy: strategy:
@@ -109,7 +111,7 @@ jobs:
# The same check the publish job makes before building: a re-run for a # The same check the publish job makes before building: a re-run for a
# package the channel already holds at master's version builds # package the channel already holds at master's version builds
# nothing, and uploads nothing that could shadow the published file. # nothing, and uploads nothing that could shadow the published file.
- name: Build ${{ matrix.package }} (${{ matrix.arch }}, native) - name: Build ${{ matrix.package }} (${{ matrix.arch }})
id: build id: build
env: env:
CONTAINER_ENGINE: docker CONTAINER_ENGINE: docker
@@ -140,23 +142,26 @@ jobs:
if-no-files-found: error if-no-files-found: error
retention-days: 7 retention-days: 7
# One job for the whole merge. It collects every PR artifact for the # One job for the whole merge. It collects every artifact for the merged
# merged tree (building only what has none; aarch64 comes from the # tree (PR builds, or the rebuild job above), then walks each channel and
# rebuild job above), then walks each channel and
# architecture slot exactly once: pull that database, add every package # architecture slot exactly once: pull that database, add every package
# that belongs in it, upload. Six slots, six round trips, however many # that belongs in it, upload. Six slots, six round trips, however many
# packages the merge carried. One process is the only writer, so there # packages the merge carried. One process is the only writer, so there
# is no race between packages; the run-level concurrency group above # is no race between packages; the concurrency group keeps one merge's
# keeps one merge from overlapping the next. # publish from overlapping the next. It builds nothing, so it runs on a
# hosted runner in about a minute instead of waiting for a droplet.
# It waits for the rebuild job and runs whatever that job's result: a # It waits for the rebuild job and runs whatever that job's result: a
# failed rebuild leaves its package without an artifact, and the collect # failed rebuild leaves its package without an artifact, and the collect
# step below records that and stops before any publish. # step below records that and stops before any publish.
publish: publish:
needs: [changes, rebuild] needs: [changes, rebuild]
if: ${{ !cancelled() && needs.changes.result == 'success' && needs.changes.outputs.count != '0' }} if: ${{ !cancelled() && needs.changes.result == 'success' && needs.changes.outputs.count != '0' }}
runs-on: [self-hosted, omarchy-builder] runs-on: ubuntu-latest
environment: publish environment: publish
timeout-minutes: 240 timeout-minutes: 30
concurrency:
group: publish
cancel-in-progress: false
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
with: with:
@@ -172,8 +177,8 @@ jobs:
EOF_MATRIX EOF_MATRIX
cat plan.txt cat plan.txt
# Fetch each package's PR artifact into build-output/edge/<arch>/, or # Fetch each package's artifact into build-output/edge/<arch>/: the PR's,
# build it when no artifact exists for exactly this tree. An artifact # or the rebuild job's when the PR's had expired. An artifact
# carries its package files inside packages.tar (see build-pr.yml and # carries its package files inside packages.tar (see build-pr.yml and
# helpers/artifact-helpers.sh: the upload action rejects the colon in # helpers/artifact-helpers.sh: the upload action rejects the colon in
# an epoch filename). # an epoch filename).
@@ -199,8 +204,8 @@ jobs:
mkdir -p "build-output/edge/$arch" mkdir -p "build-output/edge/$arch"
if [[ -n "$found" ]]; then if [[ -n "$found" ]]; then
if [[ $from_run == "${{ github.run_id }}" ]]; then if [[ $from_run == "${{ github.run_id }}" ]]; then
kind=native-rebuild kind=rebuild
echo "==> $label: artifact from this run's native $arch rebuild" echo "==> $label: artifact from this run's $arch rebuild"
else else
kind=pr-artifact kind=pr-artifact
echo "==> $label: reusing the build artifact from run $from_run" echo "==> $label: reusing the build artifact from run $from_run"
@@ -225,20 +230,11 @@ jobs:
jq -nc --arg p "$package" --arg a "$arch" '{package:$p, arch:$a, source:"already-published"}' >> sources.jsonl jq -nc --arg p "$package" --arg a "$arch" '{package:$p, arch:$a, source:"already-published"}' >> sources.jsonl
continue continue
fi fi
# aarch64 never builds here: this droplet is x86 and would # Nothing builds here. No artifact means the rebuild failed (see
# emulate it. No artifact means the native rebuild failed (see # the rebuild job), or an artifact expired between planning and
# the rebuild job), or an artifact expired between planning # now (re-run all jobs).
# and now (re-run all jobs). echo "::error::$label: no artifact from the rebuild job"
if [[ $arch == aarch64 ]]; then jq -nc --arg p "$package" --arg a "$arch" '{package:$p, arch:$a, source:"build-failed"}' >> sources.jsonl; failed=1; break
echo "::error::$label: no artifact from the native rebuild; not building aarch64 under emulation"
jq -nc --arg p "$package" --arg a "$arch" '{package:$p, arch:$a, source:"native-build-failed"}' >> sources.jsonl; failed=1; break
fi
echo "==> $label: no artifact for this tree, building"
if OMARCHY_KEEP_BUILD_WORKSPACE=1 bin/build --mirror edge --arch "$arch" --package "$package"; then
jq -nc --arg p "$package" --arg a "$arch" '{package:$p, arch:$a, source:"built"}' >> sources.jsonl
else
jq -nc --arg p "$package" --arg a "$arch" '{package:$p, arch:$a, source:"build-failed"}' >> sources.jsonl; failed=1; break
fi
fi fi
done < plan.txt done < plan.txt
ls -1 build-output/edge/*/*.pkg.tar.zst 2>/dev/null || true ls -1 build-output/edge/*/*.pkg.tar.zst 2>/dev/null || true
@@ -280,8 +276,8 @@ jobs:
cat publish-record.json cat publish-record.json
exit 0 exit 0
fi fi
# A builder droplet starts with no images, so building this one here # A fresh runner has no images, and building this one here cost
# cost every publish about 100 s (and 20 s more to start a container # every publish about 100 s (and 20 s more to start a container
# from it) for the 14 s of signing and upload it is needed for. # from it) for the 14 s of signing and upload it is needed for.
# builder-images.yml already publishes the tested image for exactly # builder-images.yml already publishes the tested image for exactly
# these build inputs under their key; pull that. Build only when no # these build inputs under their key; pull that. Build only when no
@@ -392,7 +388,7 @@ jobs:
run: | run: |
jq -r --arg outcome "${{ needs.publish.result }}" ' jq -r --arg outcome "${{ needs.publish.result }}" '
def pkgs: [.packages[] | select(test("-debug-")|not)] | join(", "); def pkgs: [.packages[] | select(test("-debug-")|not)] | join(", ");
def src: if .source=="pr-artifact" then "PR artifact" elif .source=="native-rebuild" then "rebuilt natively on arm64" elif .source=="built" then "built here" elif .source=="already-published" then "already published, skipped" else "**" + .source + "**" end; def src: if .source=="pr-artifact" then "PR artifact" elif .source=="rebuild" or .source=="native-rebuild" then "rebuilt at merge" elif .source=="built" then "built here" elif .source=="already-published" then "already published, skipped" else "**" + .source + "**" end;
"### Publish " + (if $outcome=="success" then "succeeded" else "FAILED" end) + "### Publish " + (if $outcome=="success" then "succeeded" else "FAILED" end) +
" → **" + .target + "**" + (if .target!="live" then " (proof prefix, not live)" else "" end), " → **" + .target + "**" + (if .target!="live" then " (proof prefix, not live)" else "" end),
"", "",
ci/README.md
+5 -2
View File
@@ -80,8 +80,11 @@ changing them, on the box:
packages then signatures then the db. packages then signatures then the db.
- aarch64 under QEMU with credential-preserving binfmt. PR builds now run - aarch64 under QEMU with credential-preserving binfmt. PR builds now run
aarch64 natively on `ubuntu-24.04-arm` (QEMU was up to ~15x slower). When a aarch64 natively on `ubuntu-24.04-arm` (QEMU was up to ~15x slower). When a
merged aarch64 tree has no artifact, publish.yml rebuilds it there too, in its merged tree has no artifact, publish.yml rebuilds it in its own job, aarch64
own job, and signs and uploads it on the droplet like a PR artifact. there and x86_64 on a droplet, outside the publish lock.
- Publish itself builds nothing: it signs and uploads on `ubuntu-latest` in the
tested builder image pulled from GHCR, and only that job holds the `publish`
concurrency group, so a merge waits for seconds of signing, not for builds.
- Vouch gate: collaborators, `.github/VOUCHED.td`, or the `build-approved` - Vouch gate: collaborators, `.github/VOUCHED.td`, or the `build-approved`
label; denounced authors cannot be overridden by the label. label; denounced authors cannot be overridden by the label.
- Tests run on PRs only; `result`, `self-tests`, `build-isolation` are the - Tests run on PRs only; `result`, `self-tests`, `build-isolation` are the