Add bin/repo push and stop sync from deleting production
Heavy packages build faster on a local machine, but there was no way to get the artifacts to the server: bin/upload-prebuilt publishes to the rclone remote from whatever tree it runs in, so the local -> host hop was manual. bin/repo push rsyncs build-output artifacts to the host, verifies checksums, and runs upload-prebuilt over ssh. Signing stays on the host, which is the only machine with the key and the only one holding a complete repository. Publishing from a local checkout was worse than merely unsupported. sync ran rclone sync --delete-after against a tree that pkgs.omarchy.org/ gitignores, so on any machine that had not run a full release it would have deleted the production repository -- guarded only by a y/N prompt that --skip-prod-check turns off. Package uploads are now additive, deletion moves behind --prune, and sync refuses to publish a database built from a tree holding fewer packages than the remote already lists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
David Heinemeier Hansson
co-authored by
Claude Opus 5
Claude Opus 5
parent
799a98a456
commit
bf53101bbf
@@ -75,6 +75,20 @@ bin/repo update # Update database
|
||||
bin/repo sync # Sync to remote
|
||||
```
|
||||
|
||||
### Building Heavy Packages Locally
|
||||
|
||||
Large packages build faster on a local machine than on the server. Build them
|
||||
here, then hand the artifacts to the build host, which signs and publishes them:
|
||||
|
||||
```bash
|
||||
bin/repo build --package nvidia-580xx-utils # Build on the fast machine
|
||||
bin/repo push --package nvidia-580xx-utils # Upload + publish on the host
|
||||
```
|
||||
|
||||
`push` uploads to the host's `build-output/`, verifies checksums, and runs
|
||||
`bin/upload-prebuilt` there. Do not publish from a local checkout instead: only
|
||||
the build host holds the complete repository and the signing key.
|
||||
|
||||
## Commands
|
||||
|
||||
### Global Flags
|
||||
@@ -141,10 +155,50 @@ bin/repo sync # Sync current arch/mirror
|
||||
bin/repo sync --mirror stable # Sync stable
|
||||
bin/repo sync --arch aarch64 # Sync ARM64
|
||||
bin/repo sync --skip-prod-check # No confirmation
|
||||
bin/repo sync --prune # Also delete remote packages missing locally
|
||||
```
|
||||
|
||||
Syncs package repositories to the remote server using rclone based on the configured mirror and architecture.
|
||||
|
||||
**Uploads are additive.** A local tree is not authoritative about what belongs on
|
||||
the remote — `pkgs.omarchy.org/` is gitignored, and packages built on another
|
||||
machine exist only there — so sync never deletes by default. Removing packages
|
||||
from the remote requires `--prune`, which only makes sense from a complete tree.
|
||||
|
||||
For the same reason sync refuses to publish a repository database built from a
|
||||
tree holding fewer packages than the remote database already lists. The database
|
||||
is what pacman resolves against, so a partial one hides every package it does not
|
||||
know about even though the files are still on the mirror. Use `bin/repo push` to
|
||||
publish packages built on another machine.
|
||||
|
||||
### Push to the Build Host
|
||||
|
||||
```bash
|
||||
bin/repo push # Push everything in build-output
|
||||
bin/repo push --package nvidia-580xx-utils # Push one package
|
||||
bin/repo push --mirror stable --arch aarch64 # Pick mirror and architecture
|
||||
bin/repo push --host root@example.com # Override the build host
|
||||
bin/repo push --dry-run # Show the plan, transfer nothing
|
||||
```
|
||||
|
||||
Uploads packages from `build-output/` to the build host and publishes them there
|
||||
with `bin/upload-prebuilt` (sign → promote → update → sync). Use it when a package
|
||||
is quicker to build on a local machine than on the server.
|
||||
|
||||
Publishing happens on the host rather than locally for two reasons: the GPG
|
||||
signing key lives there and nowhere else, and only the host holds the complete
|
||||
repository that a correct database and sync require. Local machines therefore
|
||||
need no secrets.
|
||||
|
||||
The host comes from `--host`, `$OMARCHY_BUILD_HOST`, or `.build-host`, in that
|
||||
order. Note that `.build-host` also arms the automatic build trigger in
|
||||
`bin/omarchy-pkgs release`; pass `--host` or set `OMARCHY_BUILD_HOST` to keep the
|
||||
two separate.
|
||||
|
||||
Split packages are selected by their own names, not their pkgbase — pushing
|
||||
`nvidia-580xx-utils` does not carry `nvidia-580xx-dkms` along. Omit `--package` to
|
||||
push everything built.
|
||||
|
||||
### Sync AUR PKGBUILDs
|
||||
|
||||
```bash
|
||||
@@ -161,6 +215,7 @@ bin/repo migrate --arch x86_64 # Promote tested edge artifacts -> stable,
|
||||
bin/repo migrate --package <name> # Promote a single package -> stable
|
||||
bin/repo migrate --dry-run # Preview migration and cleanup
|
||||
bin/repo list # List package metadata
|
||||
bin/repo push # Upload local builds to the host and publish
|
||||
bin/add-package <package> # Add an AUR/local package with metadata
|
||||
bin/package-worktree <package> # Create upstream/patched/current scratch workspace
|
||||
bin/repo remove <package> # Remove package
|
||||
|
||||
Executable
+243
@@ -0,0 +1,243 @@
|
||||
#!/bin/bash
|
||||
# Push locally built packages to the build host and publish them there.
|
||||
#
|
||||
# Heavy packages are quicker to build on a local machine than on the server, but
|
||||
# publishing has to happen where the full repository lives: the signing key is on
|
||||
# the build host, and `bin/repo sync` can only produce a correct remote from a
|
||||
# complete local tree. So this uploads the artifacts and runs the publish steps
|
||||
# over ssh rather than syncing from here.
|
||||
|
||||
set -e
|
||||
|
||||
BUILD_ROOT=$(realpath "${BASH_SOURCE[0]%/*}/..")
|
||||
source "$BUILD_ROOT/helpers/message-helpers.sh"
|
||||
source "$BUILD_ROOT/helpers/paths.sh"
|
||||
|
||||
HOST=""
|
||||
REMOTE_ROOT="/root/omarchy-pkgs"
|
||||
CREDENTIALS="/root/.omarchy/build-credentials"
|
||||
PACKAGES=""
|
||||
DRY_RUN=false
|
||||
ASSUME_YES=false
|
||||
|
||||
print_header "Push Build to Host"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--arch)
|
||||
ARCH="$2"
|
||||
update_arch_paths
|
||||
shift 2
|
||||
;;
|
||||
--mirror)
|
||||
MIRROR="$2"
|
||||
if [[ "$MIRROR" != "edge" && "$MIRROR" != "stable" ]]; then
|
||||
print_error "Invalid mirror: $MIRROR (must be 'edge' or 'stable')"
|
||||
exit 1
|
||||
fi
|
||||
update_arch_paths
|
||||
shift 2
|
||||
;;
|
||||
--package)
|
||||
shift
|
||||
PACKAGES=""
|
||||
while [[ $# -gt 0 && ! "$1" =~ ^-- ]]; do
|
||||
PACKAGES="$PACKAGES $1"
|
||||
shift
|
||||
done
|
||||
PACKAGES="${PACKAGES# }"
|
||||
;;
|
||||
--host)
|
||||
HOST="$2"
|
||||
shift 2
|
||||
;;
|
||||
--remote-root)
|
||||
REMOTE_ROOT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--dry-run)
|
||||
DRY_RUN=true
|
||||
shift
|
||||
;;
|
||||
-y | --yes)
|
||||
ASSUME_YES=true
|
||||
shift
|
||||
;;
|
||||
-h | --help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
echo "Upload packages from build-output/ to the build host, then sign,"
|
||||
echo "promote, update and sync them there."
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --arch <arch> Target architecture (default: x86_64)"
|
||||
echo " --mirror <mirror> Mirror to publish to (edge or stable, default: edge)"
|
||||
echo " --package <names> Only push these packages (space-separated)"
|
||||
echo " --host <host> ssh destination (default: \$OMARCHY_BUILD_HOST or .build-host)"
|
||||
echo " --remote-root <path> Repository path on the host (default: $REMOTE_ROOT)"
|
||||
echo " --dry-run Show what would be pushed, transfer nothing"
|
||||
echo " -y, --yes Do not ask for confirmation"
|
||||
echo " -h, --help Show this help message"
|
||||
echo ""
|
||||
echo "Typical use:"
|
||||
echo " bin/repo build --package nvidia-580xx-utils"
|
||||
echo " bin/repo push --package nvidia-580xx-utils"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
print_error "Unknown option: $1"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# --- host resolution ---------------------------------------------------------
|
||||
|
||||
if [[ -z "$HOST" ]]; then
|
||||
HOST="${OMARCHY_BUILD_HOST:-}"
|
||||
[[ -z "$HOST" && -f "$BUILD_ROOT/.build-host" ]] && HOST=$(<"$BUILD_ROOT/.build-host")
|
||||
fi
|
||||
|
||||
if [[ -z "$HOST" ]]; then
|
||||
print_error "No build host configured"
|
||||
echo ""
|
||||
echo "Pass --host, set OMARCHY_BUILD_HOST, or write the destination to:"
|
||||
echo " $BUILD_ROOT/.build-host"
|
||||
echo ""
|
||||
echo "Note that .build-host also arms the automatic build trigger in"
|
||||
echo "'bin/omarchy-pkgs release'. Use --host or OMARCHY_BUILD_HOST to keep"
|
||||
echo "this command's host separate from that."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- collect artifacts -------------------------------------------------------
|
||||
|
||||
if [[ ! -d "$BUILD_OUTPUT_DIR" ]]; then
|
||||
print_error "Build output directory not found: $BUILD_OUTPUT_DIR"
|
||||
print_warning "Run bin/repo build first"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Package files only. Signatures are produced on the host, and the repo database
|
||||
# is rebuilt there, so neither should ride along.
|
||||
mapfile -t ALL_FILES < <(cd "$BUILD_OUTPUT_DIR" && ls -1 *.pkg.tar.* 2>/dev/null | grep -v '\.sig$' || true)
|
||||
|
||||
FILES=()
|
||||
if [[ -z "$PACKAGES" ]]; then
|
||||
FILES=("${ALL_FILES[@]}")
|
||||
else
|
||||
for file in "${ALL_FILES[@]}"; do
|
||||
# name-version-release-arch.pkg.tar.zst -> name
|
||||
pkgname="${file%-*-*-*.pkg.tar.*}"
|
||||
for wanted in $PACKAGES; do
|
||||
if [[ "$pkgname" == "$wanted" ]]; then
|
||||
FILES+=("$file")
|
||||
break
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
for wanted in $PACKAGES; do
|
||||
found=false
|
||||
for file in "${FILES[@]}"; do
|
||||
[[ "${file%-*-*-*.pkg.tar.*}" == "$wanted" ]] && found=true && break
|
||||
done
|
||||
if [[ "$found" != true ]]; then
|
||||
print_error "No built artifact for '$wanted' in $BUILD_OUTPUT_DIR"
|
||||
print_warning "Split packages are named after their outputs, not their pkgbase"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
if [[ ${#FILES[@]} -eq 0 ]]; then
|
||||
print_error "No packages found in $BUILD_OUTPUT_DIR"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
REMOTE_BUILD_OUTPUT="$REMOTE_ROOT/build-output/$MIRROR/$ARCH"
|
||||
|
||||
print_info "Host: $HOST"
|
||||
print_info "Mirror: $MIRROR"
|
||||
print_info "Architecture: $ARCH"
|
||||
print_info "Local build output: $BUILD_OUTPUT_DIR"
|
||||
print_info "Remote build output: $REMOTE_BUILD_OUTPUT"
|
||||
echo ""
|
||||
|
||||
total=0
|
||||
print_info "${#FILES[@]} package(s) to push:"
|
||||
for file in "${FILES[@]}"; do
|
||||
size=$(stat -c %s "$BUILD_OUTPUT_DIR/$file")
|
||||
total=$((total + size))
|
||||
print_step "$file ($(numfmt --to=iec --format %.1f "$size"))"
|
||||
done
|
||||
echo ""
|
||||
print_info "Total transfer: $(numfmt --to=iec --format %.1f "$total")"
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
print_warning "DRY RUN - nothing transferred"
|
||||
echo ""
|
||||
print_info "Would run on $HOST:"
|
||||
echo " source $CREDENTIALS && cd $REMOTE_ROOT && bin/upload-prebuilt --mirror $MIRROR --arch $ARCH"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Publishing reaches production, so confirm here. The remote publish runs
|
||||
# non-interactively and cannot ask.
|
||||
if [[ "$ASSUME_YES" != true ]]; then
|
||||
print_warning "This publishes to PRODUCTION via $HOST ($MIRROR/$ARCH)"
|
||||
read -p "Continue? (y/N) " -n 1 -r
|
||||
echo
|
||||
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
|
||||
print_info "Push cancelled"
|
||||
exit 0
|
||||
fi
|
||||
echo
|
||||
fi
|
||||
|
||||
# --- transfer ----------------------------------------------------------------
|
||||
|
||||
print_info "Checking host..."
|
||||
if ! ssh "$HOST" "test -d $REMOTE_ROOT"; then
|
||||
print_error "Repository not found on host: $REMOTE_ROOT"
|
||||
print_warning "Pass --remote-root if it lives elsewhere"
|
||||
exit 1
|
||||
fi
|
||||
ssh "$HOST" "mkdir -p $REMOTE_BUILD_OUTPUT"
|
||||
print_success "Host ready"
|
||||
echo ""
|
||||
|
||||
print_info "Uploading packages..."
|
||||
(cd "$BUILD_OUTPUT_DIR" && rsync -a --info=progress2 --partial "${FILES[@]}" "$HOST:$REMOTE_BUILD_OUTPUT/")
|
||||
print_success "Upload complete"
|
||||
echo ""
|
||||
|
||||
print_info "Verifying checksums..."
|
||||
local_sums=$(cd "$BUILD_OUTPUT_DIR" && sha256sum "${FILES[@]}" | sort)
|
||||
remote_sums=$(ssh "$HOST" "cd $REMOTE_BUILD_OUTPUT && sha256sum $(printf '%q ' "${FILES[@]}")" | sort)
|
||||
if [[ "$local_sums" != "$remote_sums" ]]; then
|
||||
print_error "Checksum mismatch after upload"
|
||||
diff <(echo "$local_sums") <(echo "$remote_sums") || true
|
||||
exit 1
|
||||
fi
|
||||
print_success "All ${#FILES[@]} package(s) verified"
|
||||
echo ""
|
||||
|
||||
# --- publish on the host -----------------------------------------------------
|
||||
|
||||
print_info "Publishing on $HOST (sign -> promote -> update -> sync)..."
|
||||
echo ""
|
||||
if ! ssh "$HOST" "source $CREDENTIALS && cd $REMOTE_ROOT && bin/upload-prebuilt --mirror $MIRROR --arch $ARCH --skip-prod-check"; then
|
||||
print_error "Remote publish failed"
|
||||
print_warning "The uploaded packages are still in $REMOTE_BUILD_OUTPUT on $HOST"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
print_info "Published versions:"
|
||||
for file in "${FILES[@]}"; do
|
||||
print_step "${file%-*-*.pkg.tar.*}"
|
||||
done
|
||||
echo ""
|
||||
print_success "Push complete!"
|
||||
@@ -58,6 +58,7 @@ show_usage() {
|
||||
echo " list List source package metadata (use --repo for published repo)"
|
||||
echo " remove Remove a specific package"
|
||||
echo " sync Sync repository to remote"
|
||||
echo " push Upload local builds to the build host and publish them there"
|
||||
echo ""
|
||||
echo "Typical workflows:"
|
||||
echo " $0 release # Complete release workflow"
|
||||
@@ -125,6 +126,10 @@ sync)
|
||||
"$SCRIPT_DIR/sync-repo" "$@" 2>&1 | tee "$LOG_FILE"
|
||||
exit ${PIPESTATUS[0]}
|
||||
;;
|
||||
push)
|
||||
"$SCRIPT_DIR/push-build" "$@" 2>&1 | tee "$LOG_FILE"
|
||||
exit ${PIPESTATUS[0]}
|
||||
;;
|
||||
-h | --help | help)
|
||||
show_usage
|
||||
;;
|
||||
|
||||
+68
-9
@@ -9,6 +9,7 @@ source "$BUILD_ROOT/helpers/paths.sh"
|
||||
DEFAULT_REMOTE="pkgs.omarchy.org:omarchy-pkgs"
|
||||
REMOTE="$DEFAULT_REMOTE"
|
||||
SKIP_PROD_CHECK=false
|
||||
PRUNE=false
|
||||
|
||||
# Print header
|
||||
print_header "Sync Repository to Remote"
|
||||
@@ -34,6 +35,10 @@ while [[ $# -gt 0 ]]; do
|
||||
SKIP_PROD_CHECK=true
|
||||
shift
|
||||
;;
|
||||
--prune)
|
||||
PRUNE=true
|
||||
shift
|
||||
;;
|
||||
-h | --help)
|
||||
echo "Usage: $0 [OPTIONS]"
|
||||
echo ""
|
||||
@@ -42,7 +47,11 @@ while [[ $# -gt 0 ]]; do
|
||||
echo " --mirror <mirror> Mirror to use (edge or stable, default: edge)"
|
||||
echo " --remote <remote> Rclone remote destination (default: $DEFAULT_REMOTE)"
|
||||
echo " --skip-prod-check Skip production sync confirmation"
|
||||
echo " --prune Also delete remote packages missing locally"
|
||||
echo " -h, --help Show this help message"
|
||||
echo ""
|
||||
echo "Uploads are additive by default. Removing packages from the remote"
|
||||
echo "requires --prune, which only makes sense from a complete local tree."
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
@@ -80,15 +89,65 @@ fi
|
||||
|
||||
print_info "Syncing to: $REMOTE/$DESTINATION_DIRECTORY"
|
||||
|
||||
# First sync packages (excluding database files to ensure packages are uploaded first)
|
||||
# Use --ignore-existing to not overwrite different versions already on remote
|
||||
print_info "Syncing packages..."
|
||||
rclone sync "$REPO_DIR" "$REMOTE/$DESTINATION_DIRECTORY" \
|
||||
--s3-no-head \
|
||||
--exclude "omarchy.db*" \
|
||||
--exclude "omarchy.files*" \
|
||||
--ignore-existing \
|
||||
--copy-links --delete-after -v
|
||||
# The database is what users actually resolve against, and repo-add builds it
|
||||
# from this tree alone. Publishing one built from a partial tree hides every
|
||||
# package it does not know about, even though the files are still on the remote.
|
||||
# Refuse to shrink the package list unless that is the stated intent.
|
||||
LOCAL_COUNT=$(ls -1 "$REPO_DIR"/*.pkg.tar.* 2>/dev/null | grep -vc '\.sig$' || true)
|
||||
# bsdtar, not tar: the database is compressed and GNU tar will not detect that
|
||||
# on a pipe. repo-add has used both gzip and zstd, so let libarchive decide.
|
||||
REMOTE_COUNT=$(rclone cat "$REMOTE/$DESTINATION_DIRECTORY/omarchy.db" --s3-no-head 2>/dev/null |
|
||||
bsdtar -tf - 2>/dev/null | sed 's|/.*||' | sort -u | grep -c . || true)
|
||||
|
||||
if [[ "${REMOTE_COUNT:-0}" -gt 0 && "${LOCAL_COUNT:-0}" -lt "$REMOTE_COUNT" && "$PRUNE" != true ]]; then
|
||||
print_error "Local tree has $LOCAL_COUNT package(s); the remote database lists $REMOTE_COUNT"
|
||||
echo ""
|
||||
echo "Publishing this database would hide the $((REMOTE_COUNT - LOCAL_COUNT)) package(s)"
|
||||
echo "missing from $REPO_DIR."
|
||||
echo ""
|
||||
echo "To publish packages built on this machine, push them to the build host,"
|
||||
echo "which holds the complete repository:"
|
||||
echo " bin/repo push --mirror $MIRROR --arch $ARCH"
|
||||
echo ""
|
||||
echo "If shrinking the repository is genuinely what you want, pass --prune."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Upload packages first, database last, so the remote never advertises a package
|
||||
# it does not yet have.
|
||||
#
|
||||
# This is `copy`, not `sync`: a local tree is not authoritative about what should
|
||||
# exist on the remote. Packages built on another machine live only in that
|
||||
# machine's build-output, and pkgs.omarchy.org/ is gitignored, so any checkout
|
||||
# that has not run a full release is missing nearly everything. `sync` would read
|
||||
# those absences as deletions and empty the repository. --ignore-existing keeps
|
||||
# versions already published from being overwritten.
|
||||
if [[ "$PRUNE" == true ]]; then
|
||||
print_warning "Pruning: remote packages missing from $REPO_DIR will be DELETED"
|
||||
if [[ "$SKIP_PROD_CHECK" != true ]]; then
|
||||
read -p "Prune the remote to match this tree? (y/N) " -n 1 -r
|
||||
echo
|
||||
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
|
||||
print_info "Sync cancelled"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
print_info "Syncing packages (with prune)..."
|
||||
rclone sync "$REPO_DIR" "$REMOTE/$DESTINATION_DIRECTORY" \
|
||||
--s3-no-head \
|
||||
--exclude "omarchy.db*" \
|
||||
--exclude "omarchy.files*" \
|
||||
--ignore-existing \
|
||||
--copy-links --delete-after -v
|
||||
else
|
||||
print_info "Syncing packages..."
|
||||
rclone copy "$REPO_DIR" "$REMOTE/$DESTINATION_DIRECTORY" \
|
||||
--s3-no-head \
|
||||
--exclude "omarchy.db*" \
|
||||
--exclude "omarchy.files*" \
|
||||
--ignore-existing \
|
||||
--copy-links -v
|
||||
fi
|
||||
|
||||
# Then sync database files last to ensure repository integrity
|
||||
print_info "Updating repository database..."
|
||||
|
||||
Reference in New Issue
Block a user