M0: compilable skeleton — Kigi 0.1.0 fork surgery
Hard fork of xai-org/grok-build (Apache-2.0) re-targeted as Kigi, an
unofficial Kimi Code CLI community build.
Rename & identity
- 72 xai-*/xai-grok-* crates -> kigi-* (explicit: xai-grok-pager-bin ->
kigi-bin [binary `kigi`], xai-grok-pager -> kigi-tui; rest mechanical);
ptyctl, ptyctl-cli, third_party/ unchanged; proto package
xai.grok.tools.v1 -> kigi.tools.v1
- Config home ~/.kigi (KIGI_SHARE_DIR override), env prefix GROK_* ->
KIGI_*, `kigi --version` carries the unofficial-community-build notice
- clap identity, help text, startup banner, prompt templates rebranded
(templates re-encrypted)
Deletions (PRD removal list #5/#6/#7/#9/#10)
- voice input (xai-grok-voice) and all TUI wiring
- telemetry: Mixpanel client, external OTel stream, Sentry, OTLP layers,
trace/GCS/S3 upload queues (kigi-file-utils halved), workspace upload
module & dc_log, heap-profile uploader, auth-diagnostics uploader,
session-analytics halves of feedback; local zero-egress observability
preserved in new kigi-log crate (unified log, --debug firehose,
subsystem file logs, opt-in instrumentation)
- announcements (crate, remote-settings fields, TUI surfaces)
- plugin marketplace (crate, sources/browse/CTA/extensions-modal tab);
direct plugin install/uninstall/update via kigi-agent git_install kept
- relay/gateway/assets endpoints and features (agent relay, headless
relay transport, gateway bridge, LeaderEnvUrls); leader IPC socket now
~/.kigi/leader.sock + KIGI_LEADER_SOCKET, no ws-url derivation
- functional types rehomed instead of deleted: PermissionMode ->
kigi-config-types, McpInitStrategy -> kigi-mcp, PrCreationSource ->
session signals, TerminalDiagnostics -> kigi-pager-render, agent_id ->
shell util
Endpoints
- kigi-env rewritten: single production KigiEndpoints {coding_api_base_url
https://api.kimi.com/coding/v1 (KIGI_CODE_BASE_URL), oauth_host
https://auth.kimi.com (KIGI_OAUTH_HOST), update_base_url (GitHub
Releases API), upgrade_page_url}; GrokBuildEnvironment enum deleted
Toolchain & workspace hygiene
- Rust 1.97.0 pinned; edition 2024; full cargo update; git2 hoisted to
workspace at 0.21 (Option->Result API migration), quick-xml 0.41
- Root Cargo.toml hand-maintained (PRD §8.1): version 0.1.0 inherited by
all members, members sorted, unused deps pruned
- cargo-deny advisories gate (deny.toml with documented transitive
exceptions); CI workflow (check/clippy/fmt/deny/test, macOS+Linux)
- cross-crate test seams re-gated behind `test-support` cargo feature;
insta snapshot baselines renamed to the kigi_tui prefix
- clippy --workspace --all-targets: zero warnings; fmt clean
Fixes surfaced by the port
- updater probe/installer divergence (bin/kigi vs bin/grok symlink set)
- idle model-metadata refresh dead under KIGI_CODE_BASE_URL override
(new is_effective_coding_endpoint_url, loopback+override aware)
- macOS symlinked-TMPDIR fixture canonicalization (foreign_sessions,
fast-worktree); RSS measurement tests serialized via serial_test
Docs & legal (Apache §4)
- NOTICE added (upstream attribution + change statement); THIRD-PARTY
notices sustained; kigi-tools ported-code notices extended; README,
CONTRIBUTING, SECURITY, AGENTS.md rewritten
Out of scope for M0 (tracked): Kimi auth/inference (M1), search/fetch,
command parity, config import (M2), Computer Hub excision & final
brand-token sweep (M2), distribution & self-update rewrite (M3).
This commit is contained in:
@@ -0,0 +1,117 @@
|
||||
//! Copy-on-Write file cloning using the reflink-copy crate.
|
||||
//!
|
||||
//! Uses reflink (CoW) when supported by the filesystem, with automatic
|
||||
//! fallback to regular copy.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use anyhow::Result;
|
||||
|
||||
/// Clone a file using CoW if supported, falling back to regular copy.
|
||||
///
|
||||
/// On filesystems that support it (APFS on macOS, Btrfs/XFS on Linux),
|
||||
/// this creates a reflink which shares data blocks until modified.
|
||||
/// On other filesystems, it performs a regular copy.
|
||||
pub(crate) fn clone_file(src: &Path, dest: &Path) -> Result<()> {
|
||||
reflink_copy::reflink_or_copy(src, dest)?;
|
||||
// reflink (FICLONE) only clones data blocks, creating the dest with
|
||||
// default umask permissions. Explicitly propagate the source mode so the
|
||||
// executable bit etc. survive on reflink-capable filesystems.
|
||||
let perms = std::fs::metadata(src)?.permissions();
|
||||
std::fs::set_permissions(dest, perms)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Recreate `dst` as a symlink pointing at `target`, replacing any existing
|
||||
/// entry at `dst`.
|
||||
///
|
||||
/// `symlink()` refuses to overwrite an existing path, so we remove `dst` first
|
||||
/// (a missing `dst` is not an error).
|
||||
pub(crate) fn replace_symlink(target: &Path, dst: &Path) -> std::io::Result<()> {
|
||||
let _ = std::fs::remove_file(dst);
|
||||
symlink_to(target, dst)
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
fn symlink_to(target: &Path, dst: &Path) -> std::io::Result<()> {
|
||||
std::os::unix::fs::symlink(target, dst)
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
fn symlink_to(target: &Path, dst: &Path) -> std::io::Result<()> {
|
||||
std::os::windows::fs::symlink_file(target, dst)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use tempfile::TempDir;
|
||||
|
||||
#[test]
|
||||
fn test_clone_file_with_fallback() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let src = temp.path().join("source.txt");
|
||||
let dest = temp.path().join("dest.txt");
|
||||
|
||||
std::fs::write(&src, "hello world").unwrap();
|
||||
|
||||
// Should work either via CoW or fallback
|
||||
clone_file(&src, &dest).unwrap();
|
||||
|
||||
assert!(dest.exists());
|
||||
assert_eq!(std::fs::read_to_string(&dest).unwrap(), "hello world");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_clone_file_binary() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let src = temp.path().join("source.bin");
|
||||
let dest = temp.path().join("dest.bin");
|
||||
|
||||
let data: Vec<u8> = (0..=255).collect();
|
||||
std::fs::write(&src, &data).unwrap();
|
||||
|
||||
clone_file(&src, &dest).unwrap();
|
||||
|
||||
assert_eq!(std::fs::read(&dest).unwrap(), data);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_clone_preserves_permissions() {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
let temp = TempDir::new().unwrap();
|
||||
let src = temp.path().join("script.sh");
|
||||
let dest = temp.path().join("script_copy.sh");
|
||||
|
||||
std::fs::write(&src, "#!/bin/bash\necho hello").unwrap();
|
||||
|
||||
// Make executable
|
||||
let mut perms = std::fs::metadata(&src).unwrap().permissions();
|
||||
perms.set_mode(0o755);
|
||||
std::fs::set_permissions(&src, perms).unwrap();
|
||||
|
||||
clone_file(&src, &dest).unwrap();
|
||||
|
||||
let dest_perms = std::fs::metadata(&dest).unwrap().permissions();
|
||||
assert_eq!(dest_perms.mode() & 0o777, 0o755);
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn test_replace_symlink_overwrites_and_allows_dangling() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let dst = temp.path().join("link");
|
||||
|
||||
std::fs::write(&dst, "stale").unwrap();
|
||||
// Target is intentionally dangling; it must still be created.
|
||||
replace_symlink(Path::new("does-not-exist"), &dst).unwrap();
|
||||
|
||||
let meta = std::fs::symlink_metadata(&dst).unwrap();
|
||||
assert!(meta.file_type().is_symlink(), "dst must be a symlink");
|
||||
assert_eq!(
|
||||
std::fs::read_link(&dst).unwrap(),
|
||||
Path::new("does-not-exist")
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,442 @@
|
||||
//! Parallel copy engine.
|
||||
|
||||
use std::path::Path;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use anyhow::Result;
|
||||
use crossbeam::channel::{Sender, bounded};
|
||||
use dashmap::{DashMap, DashSet};
|
||||
use ignore::{WalkBuilder, WalkState};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
use crate::copy::shard::shard_for_path;
|
||||
use crate::copy::skip::build_skip_matcher;
|
||||
use crate::copy::types::{
|
||||
CopyEntry, CopyEntryKind, CopyStats, ParallelCopyConfig, ParallelCopyResult,
|
||||
};
|
||||
use crate::copy::worker::{WorkerCtx, run_worker};
|
||||
|
||||
/// Copy files from source to dest using parallel workers with hash-based sharding.
|
||||
///
|
||||
/// Returns both stats and the set of paths that were copied (for deduplication).
|
||||
/// Maximum worker threads to prevent FD exhaustion on macOS.
|
||||
/// macOS default ulimit is 256. With 8 workers + 8 walker threads = 16 threads,
|
||||
/// each can have ~10 FDs open (deeply nested dirs), leaving headroom for other uses.
|
||||
#[cfg(target_os = "macos")]
|
||||
const MAX_PARALLEL_WORKERS: usize = 8;
|
||||
|
||||
#[cfg(not(target_os = "macos"))]
|
||||
const MAX_PARALLEL_WORKERS: usize = 32;
|
||||
|
||||
pub(crate) fn copy_parallel(
|
||||
source: &Path,
|
||||
dest: &Path,
|
||||
config: ParallelCopyConfig,
|
||||
cancellation_token: CancellationToken,
|
||||
) -> Result<ParallelCopyResult> {
|
||||
let num_workers = if config.num_workers == 0 {
|
||||
num_cpus::get().min(MAX_PARALLEL_WORKERS)
|
||||
} else {
|
||||
config.num_workers.min(MAX_PARALLEL_WORKERS)
|
||||
};
|
||||
|
||||
// Build skip patterns matcher.
|
||||
let skip_matcher = if !config.skip_patterns.is_empty() {
|
||||
Some(build_skip_matcher(&config.skip_patterns)?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
// Create bounded channels for each shard.
|
||||
let channels: Vec<_> = (0..num_workers)
|
||||
.map(|_| bounded::<CopyEntry>(config.channel_buffer))
|
||||
.collect();
|
||||
|
||||
// Shared atomic counters for stats.
|
||||
let files_copied = Arc::new(AtomicU64::new(0));
|
||||
let dirs_created = Arc::new(AtomicU64::new(0));
|
||||
let symlinks_copied = Arc::new(AtomicU64::new(0));
|
||||
let files_skipped = Arc::new(AtomicU64::new(0));
|
||||
let issues: Arc<std::sync::Mutex<Vec<String>>> = Arc::new(std::sync::Mutex::new(Vec::new()));
|
||||
|
||||
// Track successfully copied paths for deduplication.
|
||||
let copied_paths: Arc<DashSet<std::path::PathBuf>> = Arc::new(DashSet::new());
|
||||
|
||||
// Collect file metadata for index updates.
|
||||
let file_metadata: Arc<DashMap<std::path::PathBuf, std::fs::Metadata>> =
|
||||
Arc::new(DashMap::new());
|
||||
|
||||
// Spawn worker threads.
|
||||
let workers: Vec<_> = channels
|
||||
.iter()
|
||||
.map(|(_, rx)| {
|
||||
let rx = rx.clone();
|
||||
let ctx = WorkerCtx {
|
||||
source: source.to_path_buf(),
|
||||
dest: dest.to_path_buf(),
|
||||
files_copied: Arc::clone(&files_copied),
|
||||
dirs_created: Arc::clone(&dirs_created),
|
||||
symlinks_copied: Arc::clone(&symlinks_copied),
|
||||
issues: Arc::clone(&issues),
|
||||
copied_paths: Arc::clone(&copied_paths),
|
||||
file_metadata: Arc::clone(&file_metadata),
|
||||
};
|
||||
|
||||
std::thread::spawn(move || run_worker(rx, ctx))
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Collect senders for the walker.
|
||||
let senders: Vec<Sender<CopyEntry>> = channels.iter().map(|(tx, _)| tx.clone()).collect();
|
||||
|
||||
// Build the walker.
|
||||
// IMPORTANT: Limit walker threads to match num_workers to avoid FD exhaustion.
|
||||
// On macOS, the default FD limit (256) can easily be exceeded when:
|
||||
// - num_cpus walker threads (default) × directories open per thread
|
||||
// - Plus num_workers copy workers × files being copied
|
||||
// Deep directory trees (15+ levels) amplify this significantly.
|
||||
let mut builder = WalkBuilder::new(source);
|
||||
builder
|
||||
.hidden(false) // Include hidden files.
|
||||
.git_ignore(config.respect_gitignore)
|
||||
.git_global(false) // Never use global gitignore (~/.config/git/ignore) —
|
||||
// it contains personal preferences irrelevant to worktree creation.
|
||||
.git_exclude(false) // Never use .git/info/exclude — external tooling
|
||||
// can append broad patterns (*.min.js, *.zip) that
|
||||
// incorrectly skip git-tracked files. The `ignore` crate doesn't
|
||||
// check tracking status, so tracked files matching these patterns
|
||||
// get silently dropped during the copy.
|
||||
.threads(num_workers) // Limit walker parallelism to avoid FD exhaustion
|
||||
.filter_entry(|entry| {
|
||||
// Always skip .git directory.
|
||||
entry.file_name() != ".git"
|
||||
});
|
||||
|
||||
let walker = builder.build_parallel();
|
||||
|
||||
// Clone data for the walker closure.
|
||||
let source_for_walker = source.to_path_buf();
|
||||
let skip_files = config.skip_files.clone();
|
||||
let files_skipped_walker = Arc::clone(&files_skipped);
|
||||
let skip_matcher = skip_matcher.map(Arc::new);
|
||||
|
||||
// Run the parallel walker.
|
||||
walker.run(|| {
|
||||
let senders = senders.clone();
|
||||
let source = source_for_walker.clone();
|
||||
let n = num_workers;
|
||||
let skip_files = skip_files.clone();
|
||||
let files_skipped = Arc::clone(&files_skipped_walker);
|
||||
let skip_matcher = skip_matcher.clone();
|
||||
let cancellation_token = cancellation_token.clone();
|
||||
|
||||
Box::new(move |entry_result| {
|
||||
// Check for cancellation
|
||||
if cancellation_token.is_cancelled() {
|
||||
return WalkState::Quit;
|
||||
}
|
||||
|
||||
let entry = match entry_result {
|
||||
Ok(e) => e,
|
||||
Err(_) => return WalkState::Continue,
|
||||
};
|
||||
|
||||
// Get relative path.
|
||||
let rel_path = match entry.path().strip_prefix(&source) {
|
||||
Ok(p) => p.to_path_buf(),
|
||||
Err(_) => return WalkState::Continue,
|
||||
};
|
||||
|
||||
// Skip root.
|
||||
if rel_path.as_os_str().is_empty() {
|
||||
return WalkState::Continue;
|
||||
}
|
||||
|
||||
// Check if this file should be skipped (already copied or explicitly skipped).
|
||||
if let Some(ref skip) = skip_files
|
||||
&& skip.contains(&rel_path)
|
||||
{
|
||||
files_skipped.fetch_add(1, Ordering::Relaxed);
|
||||
return WalkState::Continue;
|
||||
}
|
||||
|
||||
// Check skip patterns.
|
||||
if let Some(ref matcher) = skip_matcher
|
||||
&& matcher.is_match(&rel_path)
|
||||
{
|
||||
files_skipped.fetch_add(1, Ordering::Relaxed);
|
||||
return WalkState::Continue;
|
||||
}
|
||||
|
||||
let file_type = entry.file_type();
|
||||
let is_dir = file_type.as_ref().map(|ft| ft.is_dir()).unwrap_or(false);
|
||||
let is_symlink = file_type
|
||||
.as_ref()
|
||||
.map(|ft| ft.is_symlink())
|
||||
.unwrap_or(false);
|
||||
|
||||
let kind = if is_dir {
|
||||
CopyEntryKind::Dir
|
||||
} else if is_symlink {
|
||||
CopyEntryKind::Symlink
|
||||
} else {
|
||||
CopyEntryKind::File
|
||||
};
|
||||
|
||||
// Compute shard and send.
|
||||
let shard = shard_for_path(&rel_path, n);
|
||||
let _ = senders[shard].send(CopyEntry { rel_path, kind });
|
||||
|
||||
WalkState::Continue
|
||||
})
|
||||
});
|
||||
|
||||
// Close senders to signal workers to finish.
|
||||
drop(senders);
|
||||
for (tx, _) in channels {
|
||||
drop(tx);
|
||||
}
|
||||
|
||||
// Wait for all workers.
|
||||
for worker in workers {
|
||||
let _ = worker.join();
|
||||
}
|
||||
|
||||
// Collect issues.
|
||||
let issues = match Arc::try_unwrap(issues) {
|
||||
Ok(mutex) => mutex.into_inner().unwrap_or_default(),
|
||||
Err(arc) => arc.lock().unwrap().clone(),
|
||||
};
|
||||
|
||||
let copied_paths = match Arc::try_unwrap(copied_paths) {
|
||||
Ok(set) => set,
|
||||
Err(arc) => {
|
||||
let mut set = DashSet::new();
|
||||
set.extend(arc.iter().map(|p| p.clone()));
|
||||
set
|
||||
}
|
||||
};
|
||||
|
||||
let file_metadata = match Arc::try_unwrap(file_metadata) {
|
||||
Ok(map) => map,
|
||||
Err(arc) => {
|
||||
let map = DashMap::new();
|
||||
for entry in arc.iter() {
|
||||
map.insert(entry.key().clone(), entry.value().clone());
|
||||
}
|
||||
map
|
||||
}
|
||||
};
|
||||
|
||||
Ok(ParallelCopyResult {
|
||||
stats: CopyStats {
|
||||
files_copied: files_copied.load(Ordering::Relaxed),
|
||||
dirs_created: dirs_created.load(Ordering::Relaxed),
|
||||
symlinks_copied: symlinks_copied.load(Ordering::Relaxed),
|
||||
files_skipped: files_skipped.load(Ordering::Relaxed),
|
||||
issues,
|
||||
},
|
||||
copied_paths,
|
||||
file_metadata,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::copy::types::ParallelCopyConfig;
|
||||
use std::path::PathBuf;
|
||||
use tempfile::TempDir;
|
||||
|
||||
#[test]
|
||||
fn test_copy_parallel_simple() {
|
||||
let src = TempDir::new().unwrap();
|
||||
let dest = TempDir::new().unwrap();
|
||||
|
||||
// Create some files
|
||||
std::fs::write(src.path().join("file1.txt"), "content1").unwrap();
|
||||
std::fs::write(src.path().join("file2.txt"), "content2").unwrap();
|
||||
std::fs::create_dir(src.path().join("subdir")).unwrap();
|
||||
std::fs::write(src.path().join("subdir/file3.txt"), "content3").unwrap();
|
||||
|
||||
let config = ParallelCopyConfig {
|
||||
num_workers: 2,
|
||||
channel_buffer: 64,
|
||||
respect_gitignore: false,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let result =
|
||||
copy_parallel(src.path(), dest.path(), config, CancellationToken::new()).unwrap();
|
||||
|
||||
assert_eq!(result.stats.files_copied, 3);
|
||||
assert!(dest.path().join("file1.txt").exists());
|
||||
assert!(dest.path().join("file2.txt").exists());
|
||||
assert!(dest.path().join("subdir/file3.txt").exists());
|
||||
assert_eq!(result.copied_paths.len(), 4); // 3 files + 1 dir
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_parallel_with_skip() {
|
||||
let src = TempDir::new().unwrap();
|
||||
let dest = TempDir::new().unwrap();
|
||||
|
||||
std::fs::write(src.path().join("keep.txt"), "keep").unwrap();
|
||||
std::fs::write(src.path().join("skip.txt"), "skip").unwrap();
|
||||
|
||||
let skip = DashSet::new();
|
||||
skip.insert(PathBuf::from("skip.txt"));
|
||||
|
||||
let config = ParallelCopyConfig {
|
||||
num_workers: 2,
|
||||
channel_buffer: 64,
|
||||
skip_files: Some(Arc::new(skip)),
|
||||
respect_gitignore: false,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let result =
|
||||
copy_parallel(src.path(), dest.path(), config, CancellationToken::new()).unwrap();
|
||||
|
||||
assert_eq!(result.stats.files_copied, 1);
|
||||
assert_eq!(result.stats.files_skipped, 1);
|
||||
assert!(dest.path().join("keep.txt").exists());
|
||||
assert!(!dest.path().join("skip.txt").exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_parallel_only_ignored() {
|
||||
kigi_test_utils::require_git!();
|
||||
let src = TempDir::new().unwrap();
|
||||
let dest = TempDir::new().unwrap();
|
||||
|
||||
// Create a tracked file
|
||||
std::fs::write(src.path().join("tracked.txt"), "tracked").unwrap();
|
||||
|
||||
// Create an "ignored" directory
|
||||
std::fs::create_dir(src.path().join("node_modules")).unwrap();
|
||||
std::fs::write(src.path().join("node_modules/pkg.txt"), "pkg").unwrap();
|
||||
|
||||
// Create .gitignore
|
||||
std::fs::write(src.path().join(".gitignore"), "node_modules/").unwrap();
|
||||
|
||||
// Initialize git repo
|
||||
std::process::Command::new("git")
|
||||
.current_dir(src.path())
|
||||
.args(["init"])
|
||||
.output()
|
||||
.unwrap();
|
||||
|
||||
// Copy only ignored files (skip unignored paths)
|
||||
let config = ParallelCopyConfig {
|
||||
num_workers: 2,
|
||||
channel_buffer: 64,
|
||||
skip_files: Some(Arc::new(
|
||||
crate::copy::collect_unignored_paths(src.path(), 1).unwrap(),
|
||||
)),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let _result =
|
||||
copy_parallel(src.path(), dest.path(), config, CancellationToken::new()).unwrap();
|
||||
|
||||
// Should have copied node_modules but not tracked.txt
|
||||
assert!(dest.path().join("node_modules/pkg.txt").exists());
|
||||
assert!(!dest.path().join("tracked.txt").exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_parallel_with_cancellation_token_cancelled() {
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
let src = TempDir::new().unwrap();
|
||||
let dest = TempDir::new().unwrap();
|
||||
|
||||
// Create some files
|
||||
std::fs::write(src.path().join("file1.txt"), "content1").unwrap();
|
||||
std::fs::write(src.path().join("file2.txt"), "content2").unwrap();
|
||||
std::fs::write(src.path().join("file3.txt"), "content3").unwrap();
|
||||
|
||||
// Create cancellation token - cancel immediately (pre-cancelled)
|
||||
let token = CancellationToken::new();
|
||||
token.cancel();
|
||||
|
||||
let config = ParallelCopyConfig {
|
||||
num_workers: 2,
|
||||
channel_buffer: 64,
|
||||
respect_gitignore: false,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
// Pass the PRE-CANCELLED token (not a fresh one): the walker checks
|
||||
// cancellation first thing in every callback and quits, so nothing is
|
||||
// ever queued to the workers.
|
||||
let result = copy_parallel(src.path(), dest.path(), config, token).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
result.stats.files_copied, 0,
|
||||
"a pre-cancelled token must short-circuit the copy before any file is written"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_parallel_cancellation_token_not_cancelled() {
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
let src = TempDir::new().unwrap();
|
||||
let dest = TempDir::new().unwrap();
|
||||
|
||||
// Create some files
|
||||
std::fs::write(src.path().join("file1.txt"), "content1").unwrap();
|
||||
std::fs::write(src.path().join("file2.txt"), "content2").unwrap();
|
||||
|
||||
let config = ParallelCopyConfig {
|
||||
num_workers: 2,
|
||||
channel_buffer: 64,
|
||||
respect_gitignore: false,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let result =
|
||||
copy_parallel(src.path(), dest.path(), config, CancellationToken::new()).unwrap();
|
||||
|
||||
// All files should be copied
|
||||
assert_eq!(result.stats.files_copied, 2);
|
||||
assert!(dest.path().join("file1.txt").exists());
|
||||
assert!(dest.path().join("file2.txt").exists());
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn test_copy_parallel_replicates_symlink() {
|
||||
// Exercises the worker's CopyEntryKind::Symlink arm: a symlink in the
|
||||
// source tree must be replicated AS a symlink (not dereferenced).
|
||||
let src = TempDir::new().unwrap();
|
||||
let dest = TempDir::new().unwrap();
|
||||
|
||||
std::fs::write(src.path().join("target.txt"), "content").unwrap();
|
||||
std::os::unix::fs::symlink("target.txt", src.path().join("link.txt")).unwrap();
|
||||
|
||||
let config = ParallelCopyConfig {
|
||||
num_workers: 2,
|
||||
channel_buffer: 64,
|
||||
respect_gitignore: false,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let result =
|
||||
copy_parallel(src.path(), dest.path(), config, CancellationToken::new()).unwrap();
|
||||
|
||||
assert_eq!(result.stats.symlinks_copied, 1);
|
||||
let meta = std::fs::symlink_metadata(dest.path().join("link.txt")).unwrap();
|
||||
assert!(
|
||||
meta.file_type().is_symlink(),
|
||||
"link must be replicated as a symlink"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read_link(dest.path().join("link.txt")).unwrap(),
|
||||
PathBuf::from("target.txt")
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,569 @@
|
||||
//! Selective CoW copy of `.git/` directory for standalone repository cloning.
|
||||
//!
|
||||
//! Copies essential git internal files using reflink (CoW) when supported,
|
||||
//! skipping transient state, lock files, and stale worktree registrations.
|
||||
//!
|
||||
//! The `objects/` directory (often the largest subtree) is copied in parallel
|
||||
//! using a thread pool for better throughput on SSDs.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
|
||||
use crate::copy::cow::clone_file;
|
||||
|
||||
/// Statistics from copying the `.git/` directory.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub(crate) struct GitDirCopyStats {
|
||||
pub files_copied: u64,
|
||||
pub dirs_created: u64,
|
||||
pub symlinks_copied: u64,
|
||||
pub entries_skipped: u64,
|
||||
}
|
||||
|
||||
/// Top-level `.git/` entries to skip when creating a standalone copy.
|
||||
///
|
||||
/// These are either transient state (merge/rebase in-progress markers) or
|
||||
/// linked-worktree metadata that would be stale in the copy.
|
||||
const SKIP_TOP_LEVEL: &[&str] = &[
|
||||
// Linked worktree registrations — stale in a standalone copy
|
||||
"worktrees",
|
||||
// Transient HEAD-like state files
|
||||
"FETCH_HEAD",
|
||||
"ORIG_HEAD",
|
||||
"MERGE_HEAD",
|
||||
"CHERRY_PICK_HEAD",
|
||||
"REVERT_HEAD",
|
||||
"REBASE_HEAD",
|
||||
"AUTO_MERGE",
|
||||
"BISECT_LOG",
|
||||
// In-progress multi-step operation state
|
||||
"sequencer",
|
||||
"rebase-merge",
|
||||
"rebase-apply",
|
||||
// GC state
|
||||
"gc.log",
|
||||
// fsmonitor daemon state — a host-local Unix-domain IPC socket
|
||||
// (`fsmonitor--daemon.ipc`, which cannot be reflinked/copied) plus its
|
||||
// transient `cookies/` dir. Both are runtime state of the source repo's
|
||||
// daemon and must never be inherited by a standalone copy.
|
||||
"fsmonitor--daemon",
|
||||
"fsmonitor--daemon.ipc",
|
||||
];
|
||||
|
||||
/// A work item for the parallel copy pool.
|
||||
struct CopyWork {
|
||||
source: PathBuf,
|
||||
dest: PathBuf,
|
||||
}
|
||||
|
||||
/// Copy `.git/` directory contents using CoW, skipping unnecessary entries.
|
||||
///
|
||||
/// Creates a standalone git repository's `.git/` at `dest_git` by selectively
|
||||
/// copying from `source_git`. Files are copied using reflink (CoW) when the
|
||||
/// filesystem supports it, falling back to regular copy otherwise.
|
||||
///
|
||||
/// The `objects/` subtree is copied in parallel (it's typically the largest
|
||||
/// part and has no ordering dependencies). Other top-level entries are copied
|
||||
/// sequentially.
|
||||
///
|
||||
/// Skips:
|
||||
/// - Lock files (`*.lock`) at any depth
|
||||
/// - Stale worktree registrations (`worktrees/`)
|
||||
/// - Transient state files (`MERGE_HEAD`, `CHERRY_PICK_HEAD`, etc.)
|
||||
/// - In-progress rebase/cherry-pick state (`sequencer/`, `rebase-merge/`)
|
||||
pub(crate) fn copy_git_dir(source_git: &Path, dest_git: &Path) -> Result<GitDirCopyStats> {
|
||||
copy_git_dir_with_workers(source_git, dest_git, num_cpus::get())
|
||||
}
|
||||
|
||||
/// `copy_git_dir` with an explicit worker cap, so tests can force the parallel
|
||||
/// branch (`max_workers >= 2`) deterministically regardless of `num_cpus`.
|
||||
fn copy_git_dir_with_workers(
|
||||
source_git: &Path,
|
||||
dest_git: &Path,
|
||||
max_workers: usize,
|
||||
) -> Result<GitDirCopyStats> {
|
||||
anyhow::ensure!(
|
||||
source_git.is_dir(),
|
||||
"source .git must be a directory (not a linked worktree .git file): {}",
|
||||
source_git.display()
|
||||
);
|
||||
|
||||
let files_copied = AtomicU64::new(0);
|
||||
let dirs_created = AtomicU64::new(0);
|
||||
let symlinks_copied = AtomicU64::new(0);
|
||||
let entries_skipped = AtomicU64::new(0);
|
||||
|
||||
// First pass: collect work items for parallel copy.
|
||||
// We collect all (source, dest) pairs, then process them in parallel.
|
||||
let mut work_items: Vec<CopyWork> = Vec::new();
|
||||
collect_work_recursive(
|
||||
source_git,
|
||||
dest_git,
|
||||
0,
|
||||
&mut work_items,
|
||||
&dirs_created,
|
||||
&entries_skipped,
|
||||
)?;
|
||||
|
||||
// Process file copies in parallel using scoped threads.
|
||||
let num_workers = max_workers.min(work_items.len().max(1));
|
||||
|
||||
if num_workers <= 1 || work_items.len() < 64 {
|
||||
// Not enough work to justify parallelism.
|
||||
for item in &work_items {
|
||||
copy_single_entry(&item.source, &item.dest, &files_copied, &symlinks_copied)?;
|
||||
}
|
||||
} else {
|
||||
// Shard work items across threads (simple round-robin). Each thread
|
||||
// returns its first copy error; the sequential branch propagates errors
|
||||
// with `?`, so this branch must too — a failed `.git/index`/pack copy
|
||||
// would otherwise yield a silently-corrupt standalone repo.
|
||||
let chunk_size = work_items.len().div_ceil(num_workers);
|
||||
let first_error = crossbeam::scope(|scope| {
|
||||
let handles: Vec<_> = work_items
|
||||
.chunks(chunk_size)
|
||||
.map(|chunk| {
|
||||
let files_copied = &files_copied;
|
||||
let symlinks_copied = &symlinks_copied;
|
||||
scope.spawn(move |_| -> Result<()> {
|
||||
for item in chunk {
|
||||
copy_single_entry(
|
||||
&item.source,
|
||||
&item.dest,
|
||||
files_copied,
|
||||
symlinks_copied,
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Join in spawn order so "first error" is deterministic.
|
||||
let mut first_error: Option<anyhow::Error> = None;
|
||||
for handle in handles {
|
||||
let chunk_result = match handle.join() {
|
||||
Ok(r) => r,
|
||||
Err(_) => Err(anyhow::anyhow!("parallel .git/ copy thread panicked")),
|
||||
};
|
||||
if let Err(e) = chunk_result
|
||||
&& first_error.is_none()
|
||||
{
|
||||
first_error = Some(e);
|
||||
}
|
||||
}
|
||||
first_error
|
||||
})
|
||||
.map_err(|_| anyhow::anyhow!("parallel .git/ copy panicked"))?;
|
||||
|
||||
if let Some(e) = first_error {
|
||||
return Err(e);
|
||||
}
|
||||
}
|
||||
|
||||
let stats = GitDirCopyStats {
|
||||
files_copied: files_copied.load(Ordering::Relaxed),
|
||||
dirs_created: dirs_created.load(Ordering::Relaxed),
|
||||
symlinks_copied: symlinks_copied.load(Ordering::Relaxed),
|
||||
entries_skipped: entries_skipped.load(Ordering::Relaxed),
|
||||
};
|
||||
|
||||
tracing::debug!(
|
||||
files = stats.files_copied,
|
||||
dirs = stats.dirs_created,
|
||||
symlinks = stats.symlinks_copied,
|
||||
skipped = stats.entries_skipped,
|
||||
workers = num_workers,
|
||||
"git dir copy complete"
|
||||
);
|
||||
|
||||
Ok(stats)
|
||||
}
|
||||
|
||||
/// Recursively collect work items (files/symlinks to copy), creating directories eagerly.
|
||||
///
|
||||
/// Directories are created immediately (they must exist before files are written),
|
||||
/// but file copies are deferred to the work list for parallel processing.
|
||||
fn collect_work_recursive(
|
||||
source: &Path,
|
||||
dest: &Path,
|
||||
depth: usize,
|
||||
work_items: &mut Vec<CopyWork>,
|
||||
dirs_created: &AtomicU64,
|
||||
entries_skipped: &AtomicU64,
|
||||
) -> Result<()> {
|
||||
std::fs::create_dir_all(dest)
|
||||
.with_context(|| format!("failed to create directory {}", dest.display()))?;
|
||||
dirs_created.fetch_add(1, Ordering::Relaxed);
|
||||
|
||||
let entries = std::fs::read_dir(source)
|
||||
.with_context(|| format!("failed to read directory {}", source.display()))?;
|
||||
|
||||
for entry_result in entries {
|
||||
let entry = entry_result
|
||||
.with_context(|| format!("failed to read entry in {}", source.display()))?;
|
||||
let name = entry.file_name();
|
||||
let name_str = name.to_string_lossy();
|
||||
|
||||
if should_skip(&name_str, depth) {
|
||||
entries_skipped.fetch_add(1, Ordering::Relaxed);
|
||||
tracing::trace!(entry = %name_str, depth, "skipping .git/ entry");
|
||||
continue;
|
||||
}
|
||||
|
||||
let source_path = entry.path();
|
||||
let dest_path = dest.join(&name);
|
||||
|
||||
let file_type = entry
|
||||
.file_type()
|
||||
.with_context(|| format!("failed to get file type for {}", source_path.display()))?;
|
||||
|
||||
if file_type.is_dir() {
|
||||
collect_work_recursive(
|
||||
&source_path,
|
||||
&dest_path,
|
||||
depth + 1,
|
||||
work_items,
|
||||
dirs_created,
|
||||
entries_skipped,
|
||||
)?;
|
||||
} else if file_type.is_file() || file_type.is_symlink() {
|
||||
// Regular file or symlink — add to work list.
|
||||
work_items.push(CopyWork {
|
||||
source: source_path,
|
||||
dest: dest_path,
|
||||
});
|
||||
} else {
|
||||
// Non-regular file (Unix socket, FIFO, device): it cannot be
|
||||
// reflinked or copied as a file, and it is transient host-local
|
||||
// state with no meaning in a copy (e.g. git's leftover
|
||||
// `fsmonitor--daemon.ipc` socket). Skip it instead of failing the
|
||||
// whole `.git/` copy.
|
||||
entries_skipped.fetch_add(1, Ordering::Relaxed);
|
||||
tracing::debug!(entry = %name_str, depth, "skipping non-regular .git/ entry");
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Copy a single file or symlink entry.
|
||||
fn copy_single_entry(
|
||||
source_path: &Path,
|
||||
dest_path: &Path,
|
||||
files_copied: &AtomicU64,
|
||||
symlinks_copied: &AtomicU64,
|
||||
) -> Result<()> {
|
||||
// Check if it's a symlink by querying symlink metadata.
|
||||
let metadata = std::fs::symlink_metadata(source_path)
|
||||
.with_context(|| format!("failed to stat {}", source_path.display()))?;
|
||||
|
||||
if metadata.is_symlink() {
|
||||
// `target` is only used by the Unix symlink-recreate path. On
|
||||
// Windows we copy the link as a regular file (no native symlink),
|
||||
// so the target is never inspected.
|
||||
#[cfg(unix)]
|
||||
{
|
||||
let target = std::fs::read_link(source_path)
|
||||
.with_context(|| format!("failed to read symlink {}", source_path.display()))?;
|
||||
std::os::unix::fs::symlink(&target, dest_path).with_context(|| {
|
||||
format!(
|
||||
"failed to create symlink {} -> {}",
|
||||
dest_path.display(),
|
||||
target.display()
|
||||
)
|
||||
})?;
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
{
|
||||
clone_file(source_path, dest_path).with_context(|| {
|
||||
format!(
|
||||
"failed to copy symlink as file {} -> {}",
|
||||
source_path.display(),
|
||||
dest_path.display()
|
||||
)
|
||||
})?;
|
||||
}
|
||||
symlinks_copied.fetch_add(1, Ordering::Relaxed);
|
||||
} else {
|
||||
clone_file(source_path, dest_path).with_context(|| {
|
||||
format!(
|
||||
"failed to copy {} -> {}",
|
||||
source_path.display(),
|
||||
dest_path.display()
|
||||
)
|
||||
})?;
|
||||
files_copied.fetch_add(1, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Decide whether to skip a `.git/` entry based on its name and depth.
|
||||
fn should_skip(name: &str, depth: usize) -> bool {
|
||||
// Skip lock files at any depth
|
||||
if name.ends_with(".lock") {
|
||||
return true;
|
||||
}
|
||||
// Skip known top-level entries
|
||||
if depth == 0 && SKIP_TOP_LEVEL.contains(&name) {
|
||||
return true;
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use tempfile::TempDir;
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_basic() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
// Create a minimal .git structure
|
||||
std::fs::create_dir_all(source_git.join("objects/pack")).unwrap();
|
||||
std::fs::create_dir_all(source_git.join("refs/heads")).unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
std::fs::write(source_git.join("config"), "[core]\n\tbare = false\n").unwrap();
|
||||
std::fs::write(source_git.join("index"), "fake index data").unwrap();
|
||||
std::fs::write(
|
||||
source_git.join("objects/pack/pack-abc.pack"),
|
||||
"fake pack data",
|
||||
)
|
||||
.unwrap();
|
||||
std::fs::write(source_git.join("refs/heads/main"), "abc123\n").unwrap();
|
||||
|
||||
let stats = copy_git_dir(&source_git, &dest_git).unwrap();
|
||||
|
||||
assert!(dest_git.join("HEAD").exists());
|
||||
assert!(dest_git.join("config").exists());
|
||||
assert!(dest_git.join("index").exists());
|
||||
assert!(dest_git.join("objects/pack/pack-abc.pack").exists());
|
||||
assert!(dest_git.join("refs/heads/main").exists());
|
||||
assert!(stats.files_copied >= 5);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_skips_worktrees() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
std::fs::create_dir_all(source_git.join("worktrees/wt1")).unwrap();
|
||||
std::fs::write(source_git.join("worktrees/wt1/gitdir"), "/some/path").unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
|
||||
let stats = copy_git_dir(&source_git, &dest_git).unwrap();
|
||||
|
||||
assert!(dest_git.join("HEAD").exists());
|
||||
assert!(!dest_git.join("worktrees").exists());
|
||||
assert!(stats.entries_skipped >= 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_skips_lock_files() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
std::fs::create_dir_all(&source_git).unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
std::fs::write(source_git.join("index.lock"), "locked").unwrap();
|
||||
std::fs::write(source_git.join("config.lock"), "locked").unwrap();
|
||||
|
||||
let stats = copy_git_dir(&source_git, &dest_git).unwrap();
|
||||
|
||||
assert!(dest_git.join("HEAD").exists());
|
||||
assert!(!dest_git.join("index.lock").exists());
|
||||
assert!(!dest_git.join("config.lock").exists());
|
||||
assert!(stats.entries_skipped >= 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_skips_lock_files_in_subdirs() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
std::fs::create_dir_all(source_git.join("refs/heads")).unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
std::fs::write(source_git.join("refs/heads/main"), "abc123\n").unwrap();
|
||||
std::fs::write(source_git.join("refs/heads/main.lock"), "locked").unwrap();
|
||||
|
||||
let stats = copy_git_dir(&source_git, &dest_git).unwrap();
|
||||
|
||||
assert!(dest_git.join("refs/heads/main").exists());
|
||||
assert!(!dest_git.join("refs/heads/main.lock").exists());
|
||||
assert!(stats.entries_skipped >= 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_skips_transient_state() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
std::fs::create_dir_all(&source_git).unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
std::fs::write(source_git.join("MERGE_HEAD"), "abc123").unwrap();
|
||||
std::fs::write(source_git.join("CHERRY_PICK_HEAD"), "def456").unwrap();
|
||||
std::fs::write(source_git.join("ORIG_HEAD"), "ghi789").unwrap();
|
||||
std::fs::write(source_git.join("FETCH_HEAD"), "jkl012").unwrap();
|
||||
std::fs::create_dir_all(source_git.join("rebase-merge")).unwrap();
|
||||
std::fs::write(source_git.join("rebase-merge/head-name"), "main").unwrap();
|
||||
std::fs::create_dir_all(source_git.join("sequencer")).unwrap();
|
||||
std::fs::write(source_git.join("sequencer/todo"), "pick abc123").unwrap();
|
||||
|
||||
let stats = copy_git_dir(&source_git, &dest_git).unwrap();
|
||||
|
||||
assert!(dest_git.join("HEAD").exists());
|
||||
assert!(!dest_git.join("MERGE_HEAD").exists());
|
||||
assert!(!dest_git.join("CHERRY_PICK_HEAD").exists());
|
||||
assert!(!dest_git.join("ORIG_HEAD").exists());
|
||||
assert!(!dest_git.join("FETCH_HEAD").exists());
|
||||
assert!(!dest_git.join("rebase-merge").exists());
|
||||
assert!(!dest_git.join("sequencer").exists());
|
||||
assert!(stats.entries_skipped >= 6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_skips_fsmonitor_daemon_state() {
|
||||
// git's fsmonitor leaves a `fsmonitor--daemon/` dir (and an `.ipc`
|
||||
// socket) of host-local runtime state. It must not be inherited by a
|
||||
// standalone copy.
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
std::fs::create_dir_all(source_git.join("fsmonitor--daemon/cookies")).unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
|
||||
let stats = copy_git_dir(&source_git, &dest_git).unwrap();
|
||||
|
||||
assert!(dest_git.join("HEAD").exists());
|
||||
assert!(!dest_git.join("fsmonitor--daemon").exists());
|
||||
assert!(stats.entries_skipped >= 1);
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn test_copy_git_dir_skips_non_regular_files() {
|
||||
// A leftover Unix-domain socket (e.g. git's `fsmonitor--daemon.ipc`)
|
||||
// cannot be reflinked or copied as a file. It must be skipped, not fail
|
||||
// the whole `.git/` copy. Uses a non-fsmonitor name so this exercises
|
||||
// the type-based skip rather than the SKIP_TOP_LEVEL name match.
|
||||
use std::os::unix::net::UnixListener;
|
||||
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
std::fs::create_dir_all(&source_git).unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
let _socket = UnixListener::bind(source_git.join("daemon.sock")).unwrap();
|
||||
|
||||
let stats = copy_git_dir(&source_git, &dest_git).unwrap();
|
||||
|
||||
assert!(dest_git.join("HEAD").exists());
|
||||
assert!(!dest_git.join("daemon.sock").exists());
|
||||
assert!(stats.entries_skipped >= 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_preserves_hooks() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
std::fs::create_dir_all(source_git.join("hooks")).unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
std::fs::write(
|
||||
source_git.join("hooks/pre-commit"),
|
||||
"#!/bin/bash\necho check",
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let _stats = copy_git_dir(&source_git, &dest_git).unwrap();
|
||||
|
||||
assert!(dest_git.join("hooks/pre-commit").exists());
|
||||
assert_eq!(
|
||||
std::fs::read_to_string(dest_git.join("hooks/pre-commit")).unwrap(),
|
||||
"#!/bin/bash\necho check"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_preserves_worktree_source_marker() {
|
||||
// A worktree-from-worktree (standalone) must inherit the source's
|
||||
// `grok-worktree-source` marker so it still points at the ultimate
|
||||
// main repo rather than the intermediate worktree.
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
std::fs::create_dir_all(&source_git).unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
std::fs::write(source_git.join("grok-worktree-source"), "/main/repo").unwrap();
|
||||
|
||||
copy_git_dir(&source_git, &dest_git).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
std::fs::read_to_string(dest_git.join("grok-worktree-source")).unwrap(),
|
||||
"/main/repo"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_propagates_entry_copy_error() {
|
||||
// A failed entry copy must surface as an error, not a silently-corrupt
|
||||
// "success". `max_workers = 4` + >= 64 items forces the PARALLEL branch
|
||||
// deterministically (independent of num_cpus).
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
std::fs::create_dir_all(&source_git).unwrap();
|
||||
std::fs::write(source_git.join("HEAD"), "ref: refs/heads/main\n").unwrap();
|
||||
for i in 0..128 {
|
||||
std::fs::write(source_git.join(format!("obj{i}")), "data").unwrap();
|
||||
}
|
||||
|
||||
// Pre-create the dest entry for `obj0` as a DIRECTORY so the file copy
|
||||
// onto it fails (EISDIR) deterministically, even as root.
|
||||
std::fs::create_dir_all(dest_git.join("obj0")).unwrap();
|
||||
|
||||
let err = copy_git_dir_with_workers(&source_git, &dest_git, 4)
|
||||
.expect_err("a failed .git/ entry copy must propagate as an error");
|
||||
// The error names the failing entry, not some unrelated setup failure.
|
||||
let chain = format!("{err:#}");
|
||||
assert!(
|
||||
chain.contains("obj0"),
|
||||
"error should reference the failing entry, got: {chain}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_copy_git_dir_rejects_git_file() {
|
||||
let temp = TempDir::new().unwrap();
|
||||
let source_git = temp.path().join("source/.git");
|
||||
let dest_git = temp.path().join("dest/.git");
|
||||
|
||||
// Create .git as a file (linked worktree), not a directory
|
||||
std::fs::create_dir_all(temp.path().join("source")).unwrap();
|
||||
std::fs::write(&source_git, "gitdir: /some/other/path").unwrap();
|
||||
|
||||
let result = copy_git_dir(&source_git, &dest_git);
|
||||
assert!(result.is_err());
|
||||
assert!(
|
||||
result
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("must be a directory")
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
//! Filesystem replication engine used by fast worktree creation.
|
||||
|
||||
pub(crate) mod cow;
|
||||
pub(crate) mod engine;
|
||||
pub(crate) mod gitdir;
|
||||
pub(crate) mod shard;
|
||||
pub(crate) mod skip;
|
||||
pub(crate) mod types;
|
||||
pub(crate) mod worker;
|
||||
|
||||
pub(crate) use engine::copy_parallel;
|
||||
pub(crate) use skip::collect_unignored_paths;
|
||||
pub use types::CopyStats;
|
||||
pub use types::DirtyFilesReport;
|
||||
pub(crate) use types::ParallelCopyConfig;
|
||||
@@ -0,0 +1,90 @@
|
||||
//! Hash-based shard assignment for parallel file operations.
|
||||
|
||||
#[cfg(unix)]
|
||||
use std::os::unix::ffi::OsStrExt;
|
||||
use std::path::Path;
|
||||
|
||||
use rapidhash::v3::rapidhash_v3;
|
||||
|
||||
/// rapidhash of a path's raw bytes (lossy UTF-8 on non-unix).
|
||||
fn rapidhash_path(path: &Path) -> u64 {
|
||||
#[cfg(unix)]
|
||||
let bytes = path.as_os_str().as_bytes();
|
||||
#[cfg(not(unix))]
|
||||
let lossy = path.as_os_str().to_string_lossy();
|
||||
#[cfg(not(unix))]
|
||||
let bytes = lossy.as_bytes();
|
||||
rapidhash_v3(bytes)
|
||||
}
|
||||
|
||||
/// Compute the shard index for a path based on its parent directory.
|
||||
///
|
||||
/// Files in the same directory will always be assigned to the same shard,
|
||||
/// which avoids lock contention when creating parent directories.
|
||||
pub(crate) fn shard_for_path(path: &Path, num_shards: usize) -> usize {
|
||||
let parent = path.parent().unwrap_or(path);
|
||||
(rapidhash_path(parent) as usize) % num_shards
|
||||
}
|
||||
|
||||
/// Deterministic 16-hex-char (full 64-bit) hash of a path's full bytes.
|
||||
///
|
||||
/// Disambiguates same-basename worktrees that share a basename-derived key (btrfs
|
||||
/// snapshot name, worktree DB id). Full 64 bits keep a collision astronomically
|
||||
/// unlikely.
|
||||
#[cfg(any(target_os = "linux", feature = "metadata"))]
|
||||
pub(crate) fn short_path_hash(path: &Path) -> String {
|
||||
format!("{:016x}", rapidhash_path(path))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::path::PathBuf;
|
||||
|
||||
#[test]
|
||||
fn test_same_directory_same_shard() {
|
||||
let file1 = PathBuf::from("src/foo.rs");
|
||||
let file2 = PathBuf::from("src/bar.rs");
|
||||
let file3 = PathBuf::from("src/baz.rs");
|
||||
|
||||
let num_shards = 8;
|
||||
|
||||
let shard1 = shard_for_path(&file1, num_shards);
|
||||
let shard2 = shard_for_path(&file2, num_shards);
|
||||
let shard3 = shard_for_path(&file3, num_shards);
|
||||
|
||||
// All files in src/ should go to the same shard
|
||||
assert_eq!(shard1, shard2);
|
||||
assert_eq!(shard2, shard3);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_different_directories_may_differ() {
|
||||
let file1 = PathBuf::from("src/foo.rs");
|
||||
let file2 = PathBuf::from("tests/foo.rs");
|
||||
|
||||
let num_shards = 8;
|
||||
|
||||
// Different directories may (but don't have to) produce different shards
|
||||
let _shard1 = shard_for_path(&file1, num_shards);
|
||||
let _shard2 = shard_for_path(&file2, num_shards);
|
||||
// Just verify it doesn't panic
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_shard_in_range() {
|
||||
let path = PathBuf::from("some/deep/nested/path/file.txt");
|
||||
|
||||
for num_shards in 1..=16 {
|
||||
let shard = shard_for_path(&path, num_shards);
|
||||
assert!(shard < num_shards);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_root_file() {
|
||||
let path = PathBuf::from("file.txt");
|
||||
let shard = shard_for_path(&path, 8);
|
||||
assert!(shard < 8);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,112 @@
|
||||
//! Skip logic for copy operations (gitignore + additional patterns).
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
|
||||
use anyhow::Result;
|
||||
use dashmap::DashSet;
|
||||
use ignore::{WalkBuilder, WalkState};
|
||||
|
||||
/// Build a globset matcher for skip patterns.
|
||||
pub(crate) fn build_skip_matcher(patterns: &[String]) -> Result<globset::GlobSet> {
|
||||
let mut builder = globset::GlobSetBuilder::new();
|
||||
for pattern in patterns {
|
||||
builder.add(globset::Glob::new(pattern)?);
|
||||
}
|
||||
Ok(builder.build()?)
|
||||
}
|
||||
|
||||
/// Collect all *unignored* paths in `source` (relative).
|
||||
///
|
||||
/// This is used to implement an "ignored-only" copy: by collecting unignored paths
|
||||
/// and then skipping them during a second pass with `respect_gitignore=false`.
|
||||
pub(crate) fn collect_unignored_paths(
|
||||
source: &Path,
|
||||
parallelism: usize,
|
||||
) -> Result<DashSet<PathBuf>> {
|
||||
let unignored: Arc<DashSet<PathBuf>> = Arc::new(DashSet::new());
|
||||
|
||||
// git_exclude/git_global off: external tools append broad patterns (e.g.
|
||||
// *.zip) to `.git/info/exclude`; the `ignore` crate would then drop matching
|
||||
// TRACKED files from the unignored set, so the ignored-copy clobbers them.
|
||||
let walker = WalkBuilder::new(source)
|
||||
.hidden(false)
|
||||
.git_ignore(true)
|
||||
.git_global(false)
|
||||
.git_exclude(false)
|
||||
.filter_entry(|entry| entry.file_name() != ".git")
|
||||
.threads(parallelism)
|
||||
.build_parallel();
|
||||
|
||||
walker.run(|| {
|
||||
let unignored = Arc::clone(&unignored);
|
||||
Box::new(move |entry_result| {
|
||||
let entry = match entry_result {
|
||||
Ok(e) => e,
|
||||
Err(_) => return WalkState::Continue,
|
||||
};
|
||||
|
||||
let rel_path = match entry.path().strip_prefix(source) {
|
||||
Ok(p) => p.to_path_buf(),
|
||||
Err(_) => return WalkState::Continue,
|
||||
};
|
||||
|
||||
unignored.insert(rel_path);
|
||||
WalkState::Continue
|
||||
})
|
||||
});
|
||||
|
||||
Ok(match Arc::try_unwrap(unignored) {
|
||||
Ok(set) => set,
|
||||
Err(arc) => {
|
||||
let mut set = DashSet::new();
|
||||
set.extend(arc.iter().map(|p| p.clone()));
|
||||
set
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use kigi_test_utils::git::{git_commit_all, init_git_repo};
|
||||
use tempfile::TempDir;
|
||||
|
||||
#[test]
|
||||
fn collect_unignored_includes_tracked_file_matching_git_exclude() {
|
||||
kigi_test_utils::require_git!();
|
||||
// A tracked file matching `.git/info/exclude` must stay "unignored" so
|
||||
// the ignored-copy doesn't re-copy and clobber it.
|
||||
let temp = TempDir::new().unwrap();
|
||||
let repo = temp.path();
|
||||
init_git_repo(repo);
|
||||
|
||||
std::fs::write(repo.join("data.zip"), "tracked-archive").unwrap();
|
||||
std::fs::write(repo.join("main.rs"), "fn main() {}").unwrap();
|
||||
std::fs::write(repo.join(".gitignore"), "build/\n").unwrap();
|
||||
git_commit_all(repo, "initial");
|
||||
|
||||
// External tooling appends broad patterns here (e.g., *.min.js, *.zip).
|
||||
// `git init` does not always create `.git/info/` (the hermetic git on
|
||||
// arm64 CI ships no init template), so create it before writing.
|
||||
let info_dir = repo.join(".git").join("info");
|
||||
std::fs::create_dir_all(&info_dir).unwrap();
|
||||
std::fs::write(info_dir.join("exclude"), "*.zip\n").unwrap();
|
||||
|
||||
// A truly-ignored (gitignored, untracked) artifact.
|
||||
std::fs::create_dir(repo.join("build")).unwrap();
|
||||
std::fs::write(repo.join("build/out.o"), "obj").unwrap();
|
||||
|
||||
let unignored = collect_unignored_paths(repo, 1).unwrap();
|
||||
|
||||
assert!(
|
||||
unignored.contains(&PathBuf::from("data.zip")),
|
||||
"tracked file matching .git/info/exclude must be classed unignored"
|
||||
);
|
||||
assert!(unignored.contains(&PathBuf::from("main.rs")));
|
||||
assert!(
|
||||
!unignored.contains(&PathBuf::from("build/out.o")),
|
||||
"a real .gitignore'd file must remain ignored (not unignored)"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
//! Shared types for copy operations.
|
||||
|
||||
use std::fs::Metadata;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
|
||||
use dashmap::{DashMap, DashSet};
|
||||
|
||||
/// A structured report about dirty (modified/untracked/deleted) files in the source worktree.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct DirtyFilesReport {
|
||||
pub modified_files: u64,
|
||||
pub untracked_files: u64,
|
||||
pub deleted_files: u64,
|
||||
}
|
||||
|
||||
/// Statistics from a copy operation.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct CopyStats {
|
||||
pub files_copied: u64,
|
||||
pub dirs_created: u64,
|
||||
pub symlinks_copied: u64,
|
||||
pub files_skipped: u64,
|
||||
/// Non-fatal issues encountered while copying.
|
||||
pub issues: Vec<String>,
|
||||
}
|
||||
|
||||
impl CopyStats {
|
||||
/// Merge another stats into this one.
|
||||
pub fn merge(&mut self, other: CopyStats) {
|
||||
self.files_copied += other.files_copied;
|
||||
self.dirs_created += other.dirs_created;
|
||||
self.symlinks_copied += other.symlinks_copied;
|
||||
self.files_skipped += other.files_skipped;
|
||||
self.issues.extend(other.issues);
|
||||
}
|
||||
}
|
||||
|
||||
/// Kind of filesystem entry to replicate.
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub(crate) enum CopyEntryKind {
|
||||
File,
|
||||
Dir,
|
||||
Symlink,
|
||||
}
|
||||
|
||||
/// Entry to be processed by a worker.
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct CopyEntry {
|
||||
pub(crate) rel_path: PathBuf,
|
||||
pub(crate) kind: CopyEntryKind,
|
||||
}
|
||||
|
||||
/// Configuration for the parallel copy operation.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub(crate) struct ParallelCopyConfig {
|
||||
/// Number of parallel workers (0 = num_cpus)
|
||||
pub num_workers: usize,
|
||||
/// Channel buffer size per shard
|
||||
pub channel_buffer: usize,
|
||||
/// Files to skip (relative paths)
|
||||
pub skip_files: Option<Arc<DashSet<PathBuf>>>,
|
||||
/// Whether to respect `.gitignore` rules
|
||||
pub respect_gitignore: bool,
|
||||
/// Additional patterns to skip (glob patterns)
|
||||
pub skip_patterns: Vec<String>,
|
||||
}
|
||||
|
||||
/// Result of a parallel copy operation, including stats and the set of copied paths.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub(crate) struct ParallelCopyResult {
|
||||
pub stats: CopyStats,
|
||||
/// All relative paths that were successfully copied (for deduplication in subsequent copies).
|
||||
pub copied_paths: DashSet<PathBuf>,
|
||||
/// Metadata for files that were copied (for index updates). Only regular files, not symlinks/dirs.
|
||||
pub file_metadata: DashMap<PathBuf, Metadata>,
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
//! Worker logic for replicating a single filesystem entry.
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::fs::Metadata;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use dashmap::{DashMap, DashSet};
|
||||
|
||||
use crate::copy::cow;
|
||||
use crate::copy::types::{CopyEntry, CopyEntryKind};
|
||||
|
||||
pub(crate) struct WorkerCtx {
|
||||
pub source: PathBuf,
|
||||
pub dest: PathBuf,
|
||||
pub files_copied: Arc<AtomicU64>,
|
||||
pub dirs_created: Arc<AtomicU64>,
|
||||
pub symlinks_copied: Arc<AtomicU64>,
|
||||
pub issues: Arc<std::sync::Mutex<Vec<String>>>,
|
||||
pub copied_paths: Arc<DashSet<PathBuf>>,
|
||||
pub file_metadata: Arc<DashMap<PathBuf, Metadata>>,
|
||||
}
|
||||
|
||||
pub(crate) fn run_worker(rx: crossbeam::channel::Receiver<CopyEntry>, ctx: WorkerCtx) {
|
||||
// Track created directories to avoid redundant mkdir calls.
|
||||
let mut created_dirs: HashSet<PathBuf> = HashSet::new();
|
||||
|
||||
for entry in rx {
|
||||
let src = ctx.source.join(&entry.rel_path);
|
||||
let dst = ctx.dest.join(&entry.rel_path);
|
||||
|
||||
let success = process_entry(
|
||||
&entry,
|
||||
&src,
|
||||
&dst,
|
||||
&mut created_dirs,
|
||||
&ctx.files_copied,
|
||||
&ctx.dirs_created,
|
||||
&ctx.symlinks_copied,
|
||||
&ctx.issues,
|
||||
&ctx.file_metadata,
|
||||
);
|
||||
|
||||
if success {
|
||||
ctx.copied_paths.insert(entry.rel_path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn process_entry(
|
||||
entry: &CopyEntry,
|
||||
src: &Path,
|
||||
dst: &Path,
|
||||
created_dirs: &mut HashSet<PathBuf>,
|
||||
files_copied: &AtomicU64,
|
||||
dirs_created: &AtomicU64,
|
||||
symlinks_copied: &AtomicU64,
|
||||
issues: &std::sync::Mutex<Vec<String>>,
|
||||
file_metadata: &DashMap<PathBuf, Metadata>,
|
||||
) -> bool {
|
||||
// Ensure parent directory exists.
|
||||
if let Some(parent) = dst.parent()
|
||||
&& !parent.as_os_str().is_empty()
|
||||
&& created_dirs.insert(parent.to_path_buf())
|
||||
&& let Err(e) = std::fs::create_dir_all(parent)
|
||||
&& e.kind() != std::io::ErrorKind::AlreadyExists
|
||||
{
|
||||
issues
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(format!("mkdir {}: {}", parent.display(), e));
|
||||
return false;
|
||||
}
|
||||
|
||||
match entry.kind {
|
||||
CopyEntryKind::Dir => match std::fs::create_dir_all(dst) {
|
||||
Ok(()) => {
|
||||
dirs_created.fetch_add(1, Ordering::Relaxed);
|
||||
true
|
||||
}
|
||||
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => true,
|
||||
Err(e) => {
|
||||
issues
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(format!("mkdir {}: {}", entry.rel_path.display(), e));
|
||||
false
|
||||
}
|
||||
},
|
||||
CopyEntryKind::Symlink => match std::fs::read_link(src) {
|
||||
Ok(target) => match cow::replace_symlink(&target, dst) {
|
||||
Ok(()) => {
|
||||
symlinks_copied.fetch_add(1, Ordering::Relaxed);
|
||||
true
|
||||
}
|
||||
Err(e) => {
|
||||
issues.lock().unwrap().push(format!(
|
||||
"symlink {}: {}",
|
||||
entry.rel_path.display(),
|
||||
e
|
||||
));
|
||||
false
|
||||
}
|
||||
},
|
||||
Err(e) => {
|
||||
issues.lock().unwrap().push(format!(
|
||||
"read_link {}: {}",
|
||||
entry.rel_path.display(),
|
||||
e
|
||||
));
|
||||
false
|
||||
}
|
||||
},
|
||||
CopyEntryKind::File => match cow::clone_file(src, dst) {
|
||||
Ok(()) => {
|
||||
files_copied.fetch_add(1, Ordering::Relaxed);
|
||||
|
||||
// Collect file metadata for index updates
|
||||
if let Ok(metadata) = std::fs::metadata(dst) {
|
||||
file_metadata.insert(entry.rel_path.clone(), metadata);
|
||||
}
|
||||
|
||||
true
|
||||
}
|
||||
Err(e) => {
|
||||
issues
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(format!("copy {}: {}", entry.rel_path.display(), e));
|
||||
false
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user