M0: compilable skeleton — Kigi 0.1.0 fork surgery
Hard fork of xai-org/grok-build (Apache-2.0) re-targeted as Kigi, an
unofficial Kimi Code CLI community build.
Rename & identity
- 72 xai-*/xai-grok-* crates -> kigi-* (explicit: xai-grok-pager-bin ->
kigi-bin [binary `kigi`], xai-grok-pager -> kigi-tui; rest mechanical);
ptyctl, ptyctl-cli, third_party/ unchanged; proto package
xai.grok.tools.v1 -> kigi.tools.v1
- Config home ~/.kigi (KIGI_SHARE_DIR override), env prefix GROK_* ->
KIGI_*, `kigi --version` carries the unofficial-community-build notice
- clap identity, help text, startup banner, prompt templates rebranded
(templates re-encrypted)
Deletions (PRD removal list #5/#6/#7/#9/#10)
- voice input (xai-grok-voice) and all TUI wiring
- telemetry: Mixpanel client, external OTel stream, Sentry, OTLP layers,
trace/GCS/S3 upload queues (kigi-file-utils halved), workspace upload
module & dc_log, heap-profile uploader, auth-diagnostics uploader,
session-analytics halves of feedback; local zero-egress observability
preserved in new kigi-log crate (unified log, --debug firehose,
subsystem file logs, opt-in instrumentation)
- announcements (crate, remote-settings fields, TUI surfaces)
- plugin marketplace (crate, sources/browse/CTA/extensions-modal tab);
direct plugin install/uninstall/update via kigi-agent git_install kept
- relay/gateway/assets endpoints and features (agent relay, headless
relay transport, gateway bridge, LeaderEnvUrls); leader IPC socket now
~/.kigi/leader.sock + KIGI_LEADER_SOCKET, no ws-url derivation
- functional types rehomed instead of deleted: PermissionMode ->
kigi-config-types, McpInitStrategy -> kigi-mcp, PrCreationSource ->
session signals, TerminalDiagnostics -> kigi-pager-render, agent_id ->
shell util
Endpoints
- kigi-env rewritten: single production KigiEndpoints {coding_api_base_url
https://api.kimi.com/coding/v1 (KIGI_CODE_BASE_URL), oauth_host
https://auth.kimi.com (KIGI_OAUTH_HOST), update_base_url (GitHub
Releases API), upgrade_page_url}; GrokBuildEnvironment enum deleted
Toolchain & workspace hygiene
- Rust 1.97.0 pinned; edition 2024; full cargo update; git2 hoisted to
workspace at 0.21 (Option->Result API migration), quick-xml 0.41
- Root Cargo.toml hand-maintained (PRD §8.1): version 0.1.0 inherited by
all members, members sorted, unused deps pruned
- cargo-deny advisories gate (deny.toml with documented transitive
exceptions); CI workflow (check/clippy/fmt/deny/test, macOS+Linux)
- cross-crate test seams re-gated behind `test-support` cargo feature;
insta snapshot baselines renamed to the kigi_tui prefix
- clippy --workspace --all-targets: zero warnings; fmt clean
Fixes surfaced by the port
- updater probe/installer divergence (bin/kigi vs bin/grok symlink set)
- idle model-metadata refresh dead under KIGI_CODE_BASE_URL override
(new is_effective_coding_endpoint_url, loopback+override aware)
- macOS symlinked-TMPDIR fixture canonicalization (foreign_sessions,
fast-worktree); RSS measurement tests serialized via serial_test
Docs & legal (Apache §4)
- NOTICE added (upstream attribution + change statement); THIRD-PARTY
notices sustained; kigi-tools ported-code notices extended; README,
CONTRIBUTING, SECURITY, AGENTS.md rewritten
Out of scope for M0 (tracked): Kimi auth/inference (M1), search/fetch,
command parity, config import (M2), Computer Hub excision & final
brand-token sweep (M2), distribution & self-update rewrite (M3).
This commit is contained in:
@@ -0,0 +1,179 @@
|
||||
use crate::file_system::{AsyncFileSystem, FsError};
|
||||
use agent_client_protocol as acp;
|
||||
use kigi_acp_lib::AcpAgentGatewaySender as GatewaySender;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
pub struct AcpSessionFs {
|
||||
root: PathBuf,
|
||||
gateway: GatewaySender,
|
||||
session_id: acp::SessionId,
|
||||
/// When set, any path under `display_cwd` is rewritten to `root` before
|
||||
/// being sent to the extension. This is the defense-in-depth guard for
|
||||
/// AB overlay isolation: if a tool accidentally passes the display path
|
||||
/// (e.g., `/testbed/project/foo.rs`) instead of the overlay path
|
||||
/// (`~/.kigi/worktrees/.../b-overlay/foo.rs`), the adapter rewrites it
|
||||
/// so the extension reads/writes to the correct overlay location.
|
||||
display_cwd: Option<PathBuf>,
|
||||
}
|
||||
|
||||
impl AcpSessionFs {
|
||||
pub fn new(root: PathBuf, session_id: acp::SessionId, gateway: GatewaySender) -> Self {
|
||||
Self {
|
||||
root,
|
||||
session_id,
|
||||
gateway,
|
||||
display_cwd: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Set the display CWD for path rewriting.
|
||||
///
|
||||
/// When AB FS isolation is active, the model sees `display_cwd`
|
||||
/// (e.g., `/testbed/project`) but writes should go to `root`
|
||||
/// (the overlay path). Any path under `display_cwd` is rewritten
|
||||
/// to the equivalent path under `root`.
|
||||
pub fn with_display_cwd(mut self, display_cwd: PathBuf) -> Self {
|
||||
self.display_cwd = Some(display_cwd);
|
||||
self
|
||||
}
|
||||
|
||||
/// Rewrite a display path to the overlay path if needed.
|
||||
fn resolve_path(&self, path: &Path) -> PathBuf {
|
||||
if let Some(ref display) = self.display_cwd
|
||||
&& let Ok(suffix) = path.strip_prefix(display)
|
||||
{
|
||||
let resolved = self.root.join(suffix);
|
||||
tracing::debug!(
|
||||
display_path = %path.display(),
|
||||
resolved_path = %resolved.display(),
|
||||
"AcpSessionFs: rewrote display path to overlay path"
|
||||
);
|
||||
return resolved;
|
||||
}
|
||||
path.to_path_buf()
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl AsyncFileSystem for AcpSessionFs {
|
||||
fn root(&self) -> &Path {
|
||||
&self.root
|
||||
}
|
||||
|
||||
async fn exists(&self, path: &Path) -> Result<bool, FsError> {
|
||||
let resolved = self.resolve_path(path);
|
||||
let read_req = acp::ReadTextFileRequest::new(self.session_id.clone(), resolved).limit(0);
|
||||
match self.gateway.send(read_req).await {
|
||||
Ok(_) => Ok(true),
|
||||
Err(e) if e.code == acp::ErrorCode::ResourceNotFound => Ok(false),
|
||||
Err(e) => Err(FsError::Other(e.to_string())),
|
||||
}
|
||||
}
|
||||
|
||||
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, FsError> {
|
||||
let resolved = self.resolve_path(path);
|
||||
let read_req = acp::ReadTextFileRequest::new(self.session_id.clone(), resolved);
|
||||
let response = self
|
||||
.gateway
|
||||
.send(read_req)
|
||||
.await
|
||||
.map_err(|e| FsError::Other(e.to_string()))?;
|
||||
Ok(response.content.into_bytes())
|
||||
}
|
||||
|
||||
async fn try_read_file(&self, path: &Path) -> Result<Option<Vec<u8>>, FsError> {
|
||||
let resolved = self.resolve_path(path);
|
||||
let read_req = acp::ReadTextFileRequest::new(self.session_id.clone(), resolved);
|
||||
match self.gateway.send(read_req).await {
|
||||
Ok(response) => Ok(Some(response.content.into_bytes())),
|
||||
Err(e) if e.code == acp::ErrorCode::ResourceNotFound => Ok(None),
|
||||
Err(e) => Err(FsError::Other(e.to_string())),
|
||||
}
|
||||
}
|
||||
|
||||
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), FsError> {
|
||||
let resolved = self.resolve_path(path);
|
||||
let write_req = acp::WriteTextFileRequest::new(
|
||||
self.session_id.clone(),
|
||||
resolved,
|
||||
String::from_utf8(data.to_vec()).map_err(|e| FsError::Other(e.to_string()))?,
|
||||
);
|
||||
self.gateway
|
||||
.send(write_req)
|
||||
.await
|
||||
.map_err(|e| FsError::Other(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn delete_file(&self, path: &Path) -> Result<(), FsError> {
|
||||
// ACP protocol doesn't support file deletion yet
|
||||
// For now, we'll log a warning and return Ok (no-op)
|
||||
tracing::warn!(?path, "ACP filesystem does not support file deletion");
|
||||
Err(FsError::Other(
|
||||
"File deletion not supported via ACP".to_string(),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
// resolve_path only uses self.root and self.display_cwd — extract
|
||||
// the logic into a standalone test helper that doesn't need a gateway.
|
||||
fn test_resolve(root: &str, display_cwd: Option<&str>, input: &str) -> PathBuf {
|
||||
let root = PathBuf::from(root);
|
||||
let display = display_cwd.map(PathBuf::from);
|
||||
// Inline the same logic as resolve_path
|
||||
if let Some(ref display) = display
|
||||
&& let Ok(suffix) = Path::new(input).strip_prefix(display)
|
||||
{
|
||||
return root.join(suffix);
|
||||
}
|
||||
PathBuf::from(input)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_path_rewrites_display_to_overlay() {
|
||||
let result = test_resolve(
|
||||
"/root/.kigi/worktrees/proj/ab-123-b-overlay",
|
||||
Some("/testbed/proj"),
|
||||
"/testbed/proj/src/main.rs",
|
||||
);
|
||||
assert_eq!(
|
||||
result,
|
||||
PathBuf::from("/root/.kigi/worktrees/proj/ab-123-b-overlay/src/main.rs")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_path_passes_through_overlay_path() {
|
||||
let overlay_path = "/root/.kigi/worktrees/proj/ab-123-b-overlay/src/main.rs";
|
||||
let result = test_resolve(
|
||||
"/root/.kigi/worktrees/proj/ab-123-b-overlay",
|
||||
Some("/testbed/proj"),
|
||||
overlay_path,
|
||||
);
|
||||
assert_eq!(result, PathBuf::from(overlay_path));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_path_no_display_cwd_passthrough() {
|
||||
let result = test_resolve(
|
||||
"/root/.kigi/worktrees/proj/ab-123-b-overlay",
|
||||
None,
|
||||
"/testbed/proj/src/main.rs",
|
||||
);
|
||||
assert_eq!(result, PathBuf::from("/testbed/proj/src/main.rs"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolve_path_relative_path_passthrough() {
|
||||
let result = test_resolve(
|
||||
"/root/.kigi/worktrees/proj/ab-123-b-overlay",
|
||||
Some("/testbed/proj"),
|
||||
"src/main.rs",
|
||||
);
|
||||
assert_eq!(result, PathBuf::from("src/main.rs"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
//! AcpFsAdapter: implements `kigi-tools::AsyncFileSystem` using ACP gateway calls.
|
||||
//!
|
||||
//! This adapter enables file tool execution over ACP (remote filesystem).
|
||||
//! It translates kigi-tools' `AsyncFileSystem` trait into ACP protocol calls:
|
||||
//! `read_file()` → read_text_file
|
||||
//! `write_file()` → write_text_file
|
||||
//! `delete_file()` → not supported by ACP (returns error)
|
||||
//!
|
||||
//! Mirrors the pattern of `AcpTerminalAdapter` for terminal execution.
|
||||
|
||||
use std::path::Path;
|
||||
|
||||
use agent_client_protocol as acp;
|
||||
use kigi_acp_lib::AcpAgentGatewaySender as GatewaySender;
|
||||
use kigi_tools::computer::types::{AsyncFileSystem, ComputerError};
|
||||
|
||||
/// Wraps kigi-shell's ACP gateway to satisfy kigi-tools' AsyncFileSystem.
|
||||
///
|
||||
/// When a client advertises `clientCapabilities.fs.readTextFile` and `writeTextFile`,
|
||||
/// file operations from tools (read_file, search_replace, etc.) are routed through
|
||||
/// the ACP gateway back to the client instead of hitting the local disk directly.
|
||||
pub struct AcpFsAdapter {
|
||||
gateway: GatewaySender,
|
||||
session_id: acp::SessionId,
|
||||
}
|
||||
|
||||
impl AcpFsAdapter {
|
||||
pub fn new(gateway: GatewaySender, session_id: acp::SessionId) -> Self {
|
||||
Self {
|
||||
gateway,
|
||||
session_id,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl AsyncFileSystem for AcpFsAdapter {
|
||||
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, ComputerError> {
|
||||
let read_req = acp::ReadTextFileRequest::new(self.session_id.clone(), path.to_path_buf());
|
||||
|
||||
let response = self
|
||||
.gateway
|
||||
.send(read_req)
|
||||
.await
|
||||
.map_err(acp_error_to_computer_error)?;
|
||||
|
||||
Ok(response.content.into_bytes())
|
||||
}
|
||||
|
||||
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), ComputerError> {
|
||||
let content =
|
||||
String::from_utf8(data.to_vec()).map_err(|e| ComputerError::io(e.to_string()))?;
|
||||
|
||||
let write_req =
|
||||
acp::WriteTextFileRequest::new(self.session_id.clone(), path.to_path_buf(), content);
|
||||
|
||||
self.gateway
|
||||
.send(write_req)
|
||||
.await
|
||||
.map_err(acp_error_to_computer_error)?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn delete_file(&self, path: &Path) -> Result<(), ComputerError> {
|
||||
// ACP protocol doesn't support file deletion yet
|
||||
tracing::warn!(?path, "ACP filesystem does not support file deletion");
|
||||
Err(ComputerError::io("File deletion not supported via ACP"))
|
||||
}
|
||||
}
|
||||
|
||||
fn acp_error_to_computer_error(err: acp::Error) -> ComputerError {
|
||||
match acp_error_to_io_kind(&err) {
|
||||
Some(kind) => ComputerError::io_with_kind(err.to_string(), kind),
|
||||
None => ComputerError::io(err.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
fn acp_error_to_io_kind(err: &acp::Error) -> Option<std::io::ErrorKind> {
|
||||
let msg_lower = err.message.to_ascii_lowercase();
|
||||
|
||||
if err.code == acp::ErrorCode::ResourceNotFound {
|
||||
Some(std::io::ErrorKind::NotFound)
|
||||
} else if msg_lower.contains("permission denied") {
|
||||
Some(std::io::ErrorKind::PermissionDenied)
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,540 @@
|
||||
//! Contains utility functions to attach file content and render it according to the
|
||||
//! training format we have been using
|
||||
use agent_client_protocol::{BlobResourceContents, EmbeddedResource, EmbeddedResourceResource};
|
||||
use base64::{Engine as _, engine::general_purpose};
|
||||
use kigi_tools::util::truncate::estimate_tokens;
|
||||
use regex::Regex;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::path::PathBuf;
|
||||
use tracing::warn;
|
||||
#[cfg(test)]
|
||||
mod persistence {
|
||||
use std::path::PathBuf;
|
||||
pub fn session_dir(_suffix: &str) -> PathBuf {
|
||||
super::session_scratch_root()
|
||||
}
|
||||
}
|
||||
/// Maximum number of estimated tokens for a file to be included inline.
|
||||
/// Files exceeding this limit are represented as a metadata-only stub so the
|
||||
/// model knows the file exists without blowing up the context window.
|
||||
const MAX_FILE_TOKENS: usize = 5_000;
|
||||
/// 8-char content hash for dedup + collision avoidance.
|
||||
fn content_hash(content: &[u8]) -> String {
|
||||
format!("{:x}", Sha256::digest(content))[..8].to_string()
|
||||
}
|
||||
/// Parsed file reference with optional line range.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct FileReference {
|
||||
pub path: PathBuf,
|
||||
/// 1-indexed start line (inclusive)
|
||||
pub start_line: Option<usize>,
|
||||
/// 1-indexed end line (inclusive)
|
||||
pub end_line: Option<usize>,
|
||||
}
|
||||
impl FileReference {
|
||||
/// Parse a file reference string in the format `@{file_path}` or `@{file_path}:L?{start_line}-L?{end_line}`.
|
||||
pub fn parse(input: &str) -> Option<Self> {
|
||||
let re = Regex::new(r"^@?([^@].*?)(?::L?(\d+)-L?(\d+))?$").ok()?;
|
||||
let caps = re.captures(input)?;
|
||||
let path = PathBuf::from(caps.get(1)?.as_str());
|
||||
let start_line: Option<usize> = caps.get(2).and_then(|m| m.as_str().parse().ok());
|
||||
let end_line: Option<usize> = caps.get(3).and_then(|m| m.as_str().parse().ok());
|
||||
Some(Self {
|
||||
path,
|
||||
start_line,
|
||||
end_line,
|
||||
})
|
||||
}
|
||||
}
|
||||
/// Render file content from a `FileReference`.
|
||||
///
|
||||
/// When `is_cursor` is true, renders `<code_selection path="..." lines="X-Y">` format.
|
||||
/// When `is_cursor` is false, renders the original `<file_contents path="..." startLine/endLine/isFullFile>` format.
|
||||
///
|
||||
/// If the rendered content exceeds [`MAX_FILE_TOKENS`] estimated tokens the
|
||||
/// full body is omitted and a metadata-only stub is returned instead so the
|
||||
/// model still knows the file exists.
|
||||
pub async fn render_file_reference(file_ref: FileReference, is_cursor: bool) -> Option<String> {
|
||||
let read_file = tokio::fs::read(&file_ref.path).await;
|
||||
let file_content = if let Ok(read_file_output) = read_file {
|
||||
String::from_utf8(read_file_output).ok()
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let path = file_ref.path.to_string_lossy();
|
||||
let start_line = file_ref.start_line;
|
||||
let end_line = file_ref.end_line;
|
||||
file_content
|
||||
.map(|file_content| {
|
||||
let lines: Vec<&str> = file_content.lines().collect();
|
||||
let line_offset = start_line.unwrap_or(1);
|
||||
let start_idx = (line_offset.saturating_sub(1)).min(lines.len());
|
||||
let end_idx = end_line.unwrap_or(lines.len()).min(lines.len());
|
||||
let sliced_lines = &lines[start_idx..end_idx];
|
||||
let file_content = sliced_lines
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(line_number, content)| {
|
||||
format!("{}→{content}", line_offset + line_number)
|
||||
})
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
let _ = is_cursor;
|
||||
let attrs = match (start_line, end_line) {
|
||||
(Some(s), Some(e)) => format!(r#"startLine="{}" endLine="{}""#, s, e),
|
||||
_ => r#"isFullFile="true""#.to_string(),
|
||||
};
|
||||
if estimate_tokens(&file_content) > MAX_FILE_TOKENS {
|
||||
return format!(
|
||||
r#"<file_contents path="{path}" {attrs} skipped="true" reason="file too large (~{} estimated tokens, limit {MAX_FILE_TOKENS}). Use read_file tool to read specific sections."/>"#,
|
||||
estimate_tokens(& file_content),
|
||||
);
|
||||
}
|
||||
format!(
|
||||
r#"<file_contents path="{path}" {attrs}>
|
||||
{file_content}
|
||||
</file_contents>"#
|
||||
)
|
||||
})
|
||||
}
|
||||
const FILE_REGEX: &str = r"^(?:file://)?([^#]+)(?:#L(\d+)-L?(\d+))?$";
|
||||
/// Render an ACP EmbeddedResource.
|
||||
///
|
||||
/// When `is_cursor` is true, renders `<code_selection>` tags. Otherwise uses `<file_contents>`.
|
||||
/// Parses URIs in the format: `file://[path]#L[start]-[end]` or `file://[path]#L[start]-L[end]`
|
||||
///
|
||||
/// When content exceeds [`MAX_FILE_TOKENS`], the text is written to
|
||||
/// `~/.kigi/sessions/{cwd}/{session_id}/pasted/` so the model can `read_file`
|
||||
/// specific sections instead of receiving the full content inline.
|
||||
///
|
||||
/// Binary blob resources are written to `attachments/` and a path hint is returned.
|
||||
pub async fn render_embedded_resource(
|
||||
resource: &EmbeddedResource,
|
||||
is_cursor: bool,
|
||||
) -> Option<String> {
|
||||
match &resource.resource {
|
||||
EmbeddedResourceResource::TextResourceContents(text_resource)
|
||||
if text_resource.mime_type.as_deref() == Some("text/x-diff") =>
|
||||
{
|
||||
render_diff_resource(text_resource).await
|
||||
}
|
||||
EmbeddedResourceResource::TextResourceContents(text_resource) => {
|
||||
render_text_resource(text_resource, is_cursor).await
|
||||
}
|
||||
EmbeddedResourceResource::BlobResourceContents(blob) => render_blob_attachment(blob).await,
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
async fn render_text_resource(
|
||||
text_resource: &agent_client_protocol::TextResourceContents,
|
||||
is_cursor: bool,
|
||||
) -> Option<String> {
|
||||
let re = Regex::new(FILE_REGEX).ok()?;
|
||||
let caps = re.captures(&text_resource.uri)?;
|
||||
let path = caps.get(1)?.as_str();
|
||||
let start_line: Option<usize> = caps.get(2).and_then(|m| m.as_str().parse().ok());
|
||||
let end_line: Option<usize> = caps.get(3).and_then(|m| m.as_str().parse().ok());
|
||||
let line_offset = start_line.unwrap_or(1);
|
||||
let file_content = text_resource
|
||||
.text
|
||||
.lines()
|
||||
.enumerate()
|
||||
.map(|(line_number, content)| format!("{}→{content}", line_offset + line_number))
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
let attrs = match (start_line, end_line) {
|
||||
(Some(s), Some(e)) => format!(r#"startLine="{}" endLine="{}""#, s, e),
|
||||
_ => r#"isFullFile="true""#.to_string(),
|
||||
};
|
||||
let (tag, attrs_str) = ("file_contents", attrs);
|
||||
let _ = is_cursor;
|
||||
if estimate_tokens(&file_content) > MAX_FILE_TOKENS {
|
||||
let dest = write_to_session_subdir("pasted", path, text_resource.text.as_bytes()).await;
|
||||
let display_path = dest
|
||||
.as_ref()
|
||||
.map(|p| p.to_string_lossy().into_owned())
|
||||
.unwrap_or_else(|| path.to_string());
|
||||
return Some(format!(
|
||||
r#"<{tag} path="{display_path}" {attrs_str} skipped="true" reason="file too large (~{} estimated tokens, limit {MAX_FILE_TOKENS}). Use read_file tool to read specific sections."/>"#,
|
||||
estimate_tokens(&file_content),
|
||||
));
|
||||
}
|
||||
Some(format!(
|
||||
r#"<{tag} path="{path}" {attrs_str}>
|
||||
{file_content}
|
||||
</{tag}>"#
|
||||
))
|
||||
}
|
||||
/// Render a diff citation as `<diff_contents>`.
|
||||
///
|
||||
/// The text is passed through verbatim (no line-number rewriting) since it
|
||||
/// represents a change, not a file snapshot.
|
||||
async fn render_diff_resource(
|
||||
text_resource: &agent_client_protocol::TextResourceContents,
|
||||
) -> Option<String> {
|
||||
let re = Regex::new(FILE_REGEX).ok()?;
|
||||
let caps = re.captures(&text_resource.uri)?;
|
||||
let path = caps.get(1)?.as_str();
|
||||
let start_line: Option<usize> = caps.get(2).and_then(|m| m.as_str().parse().ok());
|
||||
let end_line: Option<usize> = caps.get(3).and_then(|m| m.as_str().parse().ok());
|
||||
let mut attrs = Vec::new();
|
||||
if let Some(s) = start_line {
|
||||
attrs.push(format!(r#"startLine="{s}""#));
|
||||
}
|
||||
if let Some(e) = end_line {
|
||||
attrs.push(format!(r#"endLine="{e}""#));
|
||||
}
|
||||
let attrs_str = if attrs.is_empty() {
|
||||
String::new()
|
||||
} else {
|
||||
format!(" {}", attrs.join(" "))
|
||||
};
|
||||
if estimate_tokens(&text_resource.text) > MAX_FILE_TOKENS {
|
||||
let dest = write_to_session_subdir("pasted", path, text_resource.text.as_bytes()).await;
|
||||
let display_path = dest
|
||||
.as_ref()
|
||||
.map(|p| p.to_string_lossy().into_owned())
|
||||
.unwrap_or_else(|| path.to_string());
|
||||
return Some(format!(
|
||||
r#"<diff_contents path="{display_path}"{attrs_str} skipped="true" reason="diff too large (~{} estimated tokens, limit {MAX_FILE_TOKENS}). Use read_file tool to read specific sections."/>"#,
|
||||
estimate_tokens(&text_resource.text),
|
||||
));
|
||||
}
|
||||
Some(format!(
|
||||
r#"<diff_contents path="{path}"{attrs_str}>
|
||||
{text}
|
||||
</diff_contents>"#,
|
||||
text = text_resource.text,
|
||||
))
|
||||
}
|
||||
/// Decode base64 blob, write to `attachments/`, return `<file_contents type="binary">` hint.
|
||||
async fn render_blob_attachment(blob: &BlobResourceContents) -> Option<String> {
|
||||
let raw_name = blob.uri.strip_prefix("file://").unwrap_or(&blob.uri);
|
||||
let filename = PathBuf::from(raw_name)
|
||||
.file_name()
|
||||
.map(|n| n.to_string_lossy().into_owned())
|
||||
.unwrap_or_else(|| "attachment".to_string());
|
||||
let bytes = match general_purpose::STANDARD.decode(&blob.blob) {
|
||||
Ok(b) => b,
|
||||
Err(e) => {
|
||||
warn!("binary attachment {filename}: base64 decode failed: {e}");
|
||||
return None;
|
||||
}
|
||||
};
|
||||
let size = bytes.len();
|
||||
let mime = blob
|
||||
.mime_type
|
||||
.as_deref()
|
||||
.unwrap_or("application/octet-stream");
|
||||
let dest = write_to_session_subdir("attachments", &filename, &bytes).await?;
|
||||
let path = dest.to_string_lossy();
|
||||
Some(format!(
|
||||
r#"<file_contents type="binary" path="{path}" mime_type="{mime}" size="{size}"/>"#
|
||||
))
|
||||
}
|
||||
/// Base directory for Phase-1 session-scoped scratch files. Namespaced by PID so
|
||||
/// concurrent test processes (repeated or parallel CI test runs that share
|
||||
/// `/tmp`) never collide on identical content-hash paths and
|
||||
/// race each other's cleanup. The real session_dir lives in shell persistence.
|
||||
fn session_scratch_root() -> PathBuf {
|
||||
std::env::temp_dir().join(format!("grok-test-sessions-{}", std::process::id()))
|
||||
}
|
||||
/// Write content to session subdir, return absolute path on success.
|
||||
/// Uses content hash prefix for dedup: identical content → same path, different content → unique path.
|
||||
async fn write_to_session_subdir(subdir: &str, filename: &str, content: &[u8]) -> Option<PathBuf> {
|
||||
let dir = session_scratch_root().join(subdir);
|
||||
if let Err(e) = tokio::fs::create_dir_all(&dir).await {
|
||||
warn!("failed to create {subdir} directory: {e}");
|
||||
return None;
|
||||
}
|
||||
let basename = PathBuf::from(filename)
|
||||
.file_name()
|
||||
.map(|n| n.to_string_lossy().into_owned())
|
||||
.unwrap_or_else(|| filename.to_string());
|
||||
let hash = content_hash(content);
|
||||
let dest = dir.join(format!("{hash}-{basename}"));
|
||||
if dest.exists() {
|
||||
return Some(dest);
|
||||
}
|
||||
if let Err(e) = tokio::fs::write(&dest, content).await {
|
||||
warn!("failed to write to {}: {e}", dest.display());
|
||||
return None;
|
||||
}
|
||||
Some(dest)
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn test_file_url_regex() {
|
||||
let expected = vec![
|
||||
(
|
||||
"file://Users/alice/first.txt#L10-20",
|
||||
"Users/alice/first.txt",
|
||||
"10",
|
||||
"20",
|
||||
),
|
||||
(
|
||||
"file://Users/alice/second.txt#L12-L44",
|
||||
"Users/alice/second.txt",
|
||||
"12",
|
||||
"44",
|
||||
),
|
||||
];
|
||||
let re = Regex::new(FILE_REGEX).unwrap();
|
||||
for (path, expected_path, expected_start, expected_end) in expected {
|
||||
let captures = re.captures(path).unwrap();
|
||||
assert_eq!(captures.get(1).unwrap().as_str(), expected_path);
|
||||
assert_eq!(captures.get(2).unwrap().as_str(), expected_start);
|
||||
assert_eq!(captures.get(3).unwrap().as_str(), expected_end);
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn test_file_path_regex() {
|
||||
let path = "/Users/test/019c6024-aef0-7472-89ec-65b62c577c09/prompt_3.txt#L929-L933";
|
||||
let re = Regex::new(FILE_REGEX).unwrap();
|
||||
let captures = re.captures(path).unwrap();
|
||||
assert_eq!(
|
||||
captures.get(1).unwrap().as_str(),
|
||||
"/Users/test/019c6024-aef0-7472-89ec-65b62c577c09/prompt_3.txt"
|
||||
);
|
||||
assert_eq!(captures.get(2).unwrap().as_str(), "929");
|
||||
assert_eq!(captures.get(3).unwrap().as_str(), "933");
|
||||
}
|
||||
fn file_reference(
|
||||
path: &str,
|
||||
start_line: Option<usize>,
|
||||
end_line: Option<usize>,
|
||||
) -> Option<FileReference> {
|
||||
Some(FileReference {
|
||||
path: PathBuf::from(path),
|
||||
start_line,
|
||||
end_line,
|
||||
})
|
||||
}
|
||||
#[test]
|
||||
fn test_parse_file_references() {
|
||||
let data: Vec<(&str, Option<FileReference>)> = vec![
|
||||
("", None),
|
||||
("@", None),
|
||||
("@@foo", None),
|
||||
("foo", file_reference("foo", None, None)),
|
||||
("@foo", file_reference("foo", None, None)),
|
||||
(
|
||||
"@Users/test/bar",
|
||||
file_reference("Users/test/bar", None, None),
|
||||
),
|
||||
(
|
||||
"@Users/test/bar:1-12",
|
||||
file_reference("Users/test/bar", Some(1), Some(12)),
|
||||
),
|
||||
(
|
||||
"@/asdf/asdf/asdf/asdf/asdf:L1-12",
|
||||
file_reference("/asdf/asdf/asdf/asdf/asdf", Some(1), Some(12)),
|
||||
),
|
||||
(
|
||||
"@ssasdf/asdf/dsa/fsda/f/sdf/:L1-L12",
|
||||
file_reference("ssasdf/asdf/dsa/fsda/f/sdf/", Some(1), Some(12)),
|
||||
),
|
||||
(
|
||||
"/home/user/project/src/main.rs",
|
||||
file_reference("/home/user/project/src/main.rs", None, None),
|
||||
),
|
||||
(
|
||||
"src/lib.rs:10-20",
|
||||
file_reference("src/lib.rs", Some(10), Some(20)),
|
||||
),
|
||||
(
|
||||
"@my.project/src/file.test.rs:L100-L200",
|
||||
file_reference("my.project/src/file.test.rs", Some(100), Some(200)),
|
||||
),
|
||||
("@foo.rs:L5-L5", file_reference("foo.rs", Some(5), Some(5))),
|
||||
];
|
||||
for (input, expected) in data {
|
||||
let reference = FileReference::parse(input);
|
||||
assert_eq!(reference, expected, "Failed for input: {input:?}");
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn test_estimate_tokens() {
|
||||
assert_eq!(estimate_tokens(""), 0);
|
||||
assert_eq!(estimate_tokens("abcd"), 1);
|
||||
assert_eq!(estimate_tokens("abcdefgh"), 2);
|
||||
assert_eq!(estimate_tokens(&"x".repeat(20_000)), 5_000);
|
||||
}
|
||||
fn test_info(suffix: &str) -> String {
|
||||
format!("test-session-{suffix}")
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn test_render_embedded_resource_large_file_skipped() {
|
||||
let large_content = "x".repeat(80).repeat(300);
|
||||
let _info = test_info("large-file");
|
||||
let resource = EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
|
||||
agent_client_protocol::TextResourceContents::new(
|
||||
large_content.clone(),
|
||||
"file:///project/huge.rs",
|
||||
),
|
||||
));
|
||||
let rendered = render_embedded_resource(&resource, true).await.unwrap();
|
||||
assert!(rendered.contains("skipped=\"true\""));
|
||||
assert!(!rendered.contains("xxxxxxxx"));
|
||||
assert!(rendered.contains("pasted"));
|
||||
assert!(rendered.contains("huge.rs"));
|
||||
let hash = content_hash(large_content.as_bytes());
|
||||
let expected_path = persistence::session_dir(&_info).join(format!("pasted/{hash}-huge.rs"));
|
||||
assert!(
|
||||
expected_path.exists(),
|
||||
"expected {}",
|
||||
expected_path.display()
|
||||
);
|
||||
let _ = std::fs::remove_file(&expected_path);
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn test_render_embedded_resource_small_file_included() {
|
||||
let _info = test_info("small-file");
|
||||
let resource = EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
|
||||
agent_client_protocol::TextResourceContents::new(
|
||||
"fn main() {}\n",
|
||||
"file:///project/small.rs",
|
||||
),
|
||||
));
|
||||
let rendered = render_embedded_resource(&resource, true).await.unwrap();
|
||||
assert!(rendered.contains("fn main()"));
|
||||
assert!(!rendered.contains("skipped"));
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn test_render_blob_attachment_written_to_disk() {
|
||||
use base64::{Engine as _, engine::general_purpose};
|
||||
let _info = test_info("blob-write");
|
||||
let content = b"fake pdf bytes";
|
||||
let encoded = general_purpose::STANDARD.encode(content);
|
||||
let resource = EmbeddedResource::new(EmbeddedResourceResource::BlobResourceContents(
|
||||
agent_client_protocol::BlobResourceContents::new(encoded.clone(), "file://report.pdf")
|
||||
.mime_type(Some("application/pdf".to_string())),
|
||||
));
|
||||
let rendered = render_embedded_resource(&resource, false).await.unwrap();
|
||||
assert!(rendered.contains("file_contents"), "got: {rendered}");
|
||||
assert!(rendered.contains(r#"type="binary""#), "got: {rendered}");
|
||||
assert!(rendered.contains("report.pdf"), "got: {rendered}");
|
||||
assert!(rendered.contains("application/pdf"), "got: {rendered}");
|
||||
assert!(
|
||||
rendered.contains(&content.len().to_string()),
|
||||
"got: {rendered}"
|
||||
);
|
||||
assert!(
|
||||
!rendered.contains(&encoded),
|
||||
"blob leaked into hint: {rendered}"
|
||||
);
|
||||
let hash = content_hash(content);
|
||||
let expected =
|
||||
persistence::session_dir(&_info).join(format!("attachments/{hash}-report.pdf"));
|
||||
assert!(
|
||||
expected.exists(),
|
||||
"attachment not written to disk at {}",
|
||||
expected.display()
|
||||
);
|
||||
assert_eq!(std::fs::read(&expected).unwrap(), content);
|
||||
let _ = std::fs::remove_file(&expected);
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn test_render_blob_attachment_bad_base64_returns_none() {
|
||||
let _info = test_info("blob-bad-b64");
|
||||
let resource = EmbeddedResource::new(EmbeddedResourceResource::BlobResourceContents(
|
||||
agent_client_protocol::BlobResourceContents::new(
|
||||
"!!! not valid base64 !!!",
|
||||
"file://bad.pdf",
|
||||
),
|
||||
));
|
||||
let result = render_embedded_resource(&resource, false).await;
|
||||
assert!(
|
||||
result.is_none(),
|
||||
"expected None for bad base64, got: {result:?}"
|
||||
);
|
||||
}
|
||||
fn diff_resource(uri: &str, text: &str) -> EmbeddedResource {
|
||||
EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
|
||||
agent_client_protocol::TextResourceContents::new(text, uri)
|
||||
.mime_type(Some("text/x-diff".to_string())),
|
||||
))
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn test_render_diff_resource_uses_diff_contents_tag() {
|
||||
let _info = test_info("diff-small");
|
||||
let resource = diff_resource(
|
||||
"file:///project/main.rs#L10-L12",
|
||||
"+ new line\n context\n- old line",
|
||||
);
|
||||
let rendered = render_embedded_resource(&resource, true).await.unwrap();
|
||||
assert!(rendered.contains("<diff_contents"), "got: {rendered}");
|
||||
assert!(rendered.contains("</diff_contents>"), "got: {rendered}");
|
||||
assert!(
|
||||
rendered.contains(r#"path="/project/main.rs""#),
|
||||
"got: {rendered}"
|
||||
);
|
||||
assert!(rendered.contains(r#"startLine="10""#), "got: {rendered}");
|
||||
assert!(rendered.contains(r#"endLine="12""#), "got: {rendered}");
|
||||
assert!(rendered.contains("+ new line"), "got: {rendered}");
|
||||
assert!(rendered.contains("- old line"), "got: {rendered}");
|
||||
assert!(!rendered.contains("code_selection"), "got: {rendered}");
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn test_render_diff_resource_without_line_range() {
|
||||
let _info = test_info("diff-no-range");
|
||||
let resource = diff_resource("file:///project/lib.rs", " unchanged line");
|
||||
let rendered = render_embedded_resource(&resource, true).await.unwrap();
|
||||
assert!(rendered.contains("<diff_contents"), "got: {rendered}");
|
||||
assert!(
|
||||
rendered.contains(r#"path="/project/lib.rs""#),
|
||||
"got: {rendered}"
|
||||
);
|
||||
assert!(!rendered.contains("startLine"), "got: {rendered}");
|
||||
assert!(!rendered.contains("endLine"), "got: {rendered}");
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn test_render_diff_resource_large_skipped() {
|
||||
let large_diff = "x".repeat(80).repeat(300);
|
||||
let _info = test_info("diff-large");
|
||||
let resource = diff_resource("file:///project/big.rs#L1-L999", &large_diff);
|
||||
let rendered = render_embedded_resource(&resource, true).await.unwrap();
|
||||
assert!(rendered.contains("<diff_contents"), "got: {rendered}");
|
||||
assert!(rendered.contains("skipped=\"true\""), "got: {rendered}");
|
||||
assert!(
|
||||
!rendered.contains("xxxxxxxx"),
|
||||
"full content leaked: {rendered}"
|
||||
);
|
||||
assert!(rendered.contains("pasted"), "got: {rendered}");
|
||||
let hash = content_hash(large_diff.as_bytes());
|
||||
let expected = persistence::session_dir(&_info).join(format!("pasted/{hash}-big.rs"));
|
||||
let _ = std::fs::remove_file(&expected);
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn test_grok_render_embedded_resource_uses_file_contents_tag() {
|
||||
let _info = test_info("grok-text");
|
||||
let resource = EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
|
||||
agent_client_protocol::TextResourceContents::new(
|
||||
"const x = 1;\nconst y = 2;\n",
|
||||
"file:///project/app.ts#L5-L6",
|
||||
),
|
||||
));
|
||||
let rendered = render_embedded_resource(&resource, false).await.unwrap();
|
||||
assert!(rendered.contains("<file_contents"), "got: {rendered}");
|
||||
assert!(rendered.contains("</file_contents>"), "got: {rendered}");
|
||||
assert!(rendered.contains(r#"startLine="5""#), "got: {rendered}");
|
||||
assert!(rendered.contains(r#"endLine="6""#), "got: {rendered}");
|
||||
assert!(!rendered.contains("code_selection"), "got: {rendered}");
|
||||
}
|
||||
#[tokio::test]
|
||||
async fn test_grok_render_embedded_resource_full_file_uses_is_full_file() {
|
||||
let _info = test_info("grok-full-file");
|
||||
let resource = EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
|
||||
agent_client_protocol::TextResourceContents::new(
|
||||
"fn main() {}\n",
|
||||
"file:///project/main.rs",
|
||||
),
|
||||
));
|
||||
let rendered = render_embedded_resource(&resource, false).await.unwrap();
|
||||
assert!(rendered.contains("<file_contents"), "got: {rendered}");
|
||||
assert!(rendered.contains(r#"isFullFile="true""#), "got: {rendered}");
|
||||
assert!(!rendered.contains("code_selection"), "got: {rendered}");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,775 @@
|
||||
//! Read-only filesystem helpers backing the client-facing
|
||||
//! `workspace.client_fs_*` RPCs (the grok.com conversation-files UI,
|
||||
//! tunneled through the server).
|
||||
//!
|
||||
//! Deliberately separate from the shell-facing ext ops in
|
||||
//! [`ext_fs`](super::ext_fs): every path here is workspace-root-relative
|
||||
//! and resolves through the root-confinement helper
|
||||
//! (`WorkspaceHandle::resolve_service_path`), the list walk excludes
|
||||
//! symlinks that resolve outside the root (and never descends into
|
||||
//! them), listings paginate with stable post-sort slices, and reads are
|
||||
//! binary-safe (base64 chunks).
|
||||
//!
|
||||
//! Wire types live in `kigi_workspace_types::rpc::fs` (the
|
||||
//! `ClientFs*` types), shared with the backend caller.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use kigi_workspace_types::rpc::fs::{
|
||||
ClientFsListNode as FsListNode, ClientFsListReq as FsListReq, ClientFsListRes as FsListRes,
|
||||
ClientFsReadFileReq as FsReadFileReq, ClientFsReadFileRes as FsReadFileRes,
|
||||
ClientFsStatReq as FsStatReq, ClientFsStatRes as FsStatRes, FsContentType, FsNodeType,
|
||||
};
|
||||
|
||||
use crate::error::{WorkspaceError, WorkspaceResult};
|
||||
use crate::handle::WorkspaceHandle;
|
||||
|
||||
/// Hard cap on entries collected per list call before sorting (shared
|
||||
/// across all fs surfaces; see [`super::walk::MAX_LIST_COLLECT`]).
|
||||
const MAX_LIST_COLLECT: usize = super::walk::MAX_LIST_COLLECT;
|
||||
|
||||
/// Server-side cap on `FsListReq::limit`.
|
||||
const MAX_LIST_LIMIT: u32 = 1000;
|
||||
|
||||
/// Server-side cap on a single read's effective byte budget (shared
|
||||
/// across all fs surfaces; see [`super::walk::MAX_READ_BYTES`]). Only
|
||||
/// referenced by tests now that the clamp lives in `walk::clamp_read_length`.
|
||||
#[cfg(test)]
|
||||
const MAX_READ_BYTES: u64 = super::walk::MAX_READ_BYTES;
|
||||
|
||||
/// Bound on memoized hashes; the memo is cleared (not LRU-evicted) when
|
||||
/// full — entries simply re-hash on next use.
|
||||
const HASH_MEMO_CAPACITY: usize = 4096;
|
||||
|
||||
// =========================================================================
|
||||
// (path, size, mtime_ms) → hash memo
|
||||
// =========================================================================
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
struct MemoEntry {
|
||||
size: u64,
|
||||
mtime_ms: i64,
|
||||
hash: String,
|
||||
}
|
||||
|
||||
/// Memo of full-content SHA-256 digests keyed by absolute path and
|
||||
/// validated against `(size, mtime_ms)`, so unchanged files hash once
|
||||
/// instead of on every `client_fs_stat`. The memo only avoids redundant
|
||||
/// hashing — it never substitutes mtime for content addressing: a
|
||||
/// `(size, mtime_ms)` mismatch is a miss and the caller re-hashes.
|
||||
#[derive(Debug, Default)]
|
||||
pub(crate) struct FileHashMemo {
|
||||
entries: parking_lot::Mutex<HashMap<PathBuf, MemoEntry>>,
|
||||
}
|
||||
|
||||
impl FileHashMemo {
|
||||
/// Return the memoized hash when `(size, mtime_ms)` still match.
|
||||
pub(crate) fn lookup(&self, path: &Path, size: u64, mtime_ms: i64) -> Option<String> {
|
||||
let entries = self.entries.lock();
|
||||
let entry = entries.get(path)?;
|
||||
(entry.size == size && entry.mtime_ms == mtime_ms).then(|| entry.hash.clone())
|
||||
}
|
||||
|
||||
/// Record a freshly computed hash, replacing any stale entry for the
|
||||
/// same path. Clears the whole memo when inserting a new path would
|
||||
/// exceed [`HASH_MEMO_CAPACITY`].
|
||||
pub(crate) fn store(&self, path: &Path, size: u64, mtime_ms: i64, hash: String) {
|
||||
let mut entries = self.entries.lock();
|
||||
if !entries.contains_key(path) && entries.len() >= HASH_MEMO_CAPACITY {
|
||||
entries.clear();
|
||||
}
|
||||
entries.insert(
|
||||
path.to_path_buf(),
|
||||
MemoEntry {
|
||||
size,
|
||||
mtime_ms,
|
||||
hash,
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Path resolution
|
||||
// =========================================================================
|
||||
|
||||
/// Resolve a root-relative request path through the workspace's
|
||||
/// root-confinement helper, returning the resolved path together with the
|
||||
/// canonical root it was checked against. `""` and `"."` mean the
|
||||
/// workspace root; absolute paths, `..` escapes, and symlink escapes are
|
||||
/// rejected there.
|
||||
async fn resolve_with_root(
|
||||
ws: &WorkspaceHandle,
|
||||
path: &str,
|
||||
) -> WorkspaceResult<(PathBuf, PathBuf)> {
|
||||
let rel = if path.is_empty() { "." } else { path };
|
||||
let canonical_root = ws.canonical_root().await?;
|
||||
let abs = ws.resolve_service_path(rel, &canonical_root).await?;
|
||||
Ok((abs, canonical_root))
|
||||
}
|
||||
|
||||
/// [`resolve_with_root`] for callers that don't need the canonical root.
|
||||
async fn resolve(ws: &WorkspaceHandle, path: &str) -> WorkspaceResult<PathBuf> {
|
||||
resolve_with_root(ws, path).await.map(|(abs, _)| abs)
|
||||
}
|
||||
|
||||
fn system_time_ms(st: std::time::SystemTime) -> i64 {
|
||||
match st.duration_since(std::time::UNIX_EPOCH) {
|
||||
Ok(d) => i64::try_from(d.as_millis()).unwrap_or(i64::MAX),
|
||||
Err(e) => -i64::try_from(e.duration().as_millis()).unwrap_or(i64::MAX),
|
||||
}
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// list
|
||||
// =========================================================================
|
||||
|
||||
/// List `req.path` (workspace-root-relative) with stable pagination:
|
||||
/// collect the full walk (bounded by [`MAX_LIST_COLLECT`]), sort
|
||||
/// directories-first / case-insensitive by name, then slice
|
||||
/// `[offset, offset + limit)`. Symlinks resolving outside the workspace
|
||||
/// root are excluded from the walk (and never descended into).
|
||||
pub(crate) async fn list(ws: &WorkspaceHandle, req: &FsListReq) -> WorkspaceResult<FsListRes> {
|
||||
let (abs, canonical_root) = resolve_with_root(ws, &req.path).await?;
|
||||
let root = ws.root_cwd()?;
|
||||
let req = req.clone();
|
||||
// The walk does synchronous traversal + metadata syscalls; run it off
|
||||
// the async executor (matching the ext_fs ops).
|
||||
tokio::task::spawn_blocking(move || {
|
||||
list_blocking(&abs, &root, &canonical_root, &req, MAX_LIST_COLLECT)
|
||||
})
|
||||
.await
|
||||
.map_err(|e| WorkspaceError::JoinError(e.to_string()))?
|
||||
}
|
||||
|
||||
fn list_blocking(
|
||||
abs_dir: &Path,
|
||||
root: &Path,
|
||||
canonical_root: &Path,
|
||||
req: &FsListReq,
|
||||
max_collect: usize,
|
||||
) -> WorkspaceResult<FsListRes> {
|
||||
// Root confinement also holds mid-walk: a symlink inside the root
|
||||
// pointing outside must not enumerate outside metadata.
|
||||
let page = super::walk::list_directory_paged(
|
||||
abs_dir,
|
||||
super::walk::ListOptions {
|
||||
depth: req.depth as usize,
|
||||
follow_symlinks: req.follow_symlinks,
|
||||
respect_git_ignore: req.respect_git_ignore,
|
||||
include_hidden: req.include_hidden,
|
||||
include_globs: &req.include_globs,
|
||||
exclude_globs: &req.exclude_globs,
|
||||
offset: req.offset,
|
||||
limit: req.limit.min(MAX_LIST_LIMIT) as usize,
|
||||
confine_to_canonical_root: Some(canonical_root.to_path_buf()),
|
||||
},
|
||||
max_collect,
|
||||
);
|
||||
|
||||
let nodes: Vec<FsListNode> = page
|
||||
.entries
|
||||
.into_iter()
|
||||
.map(|e| FsListNode {
|
||||
node_type: if e.is_dir {
|
||||
FsNodeType::Directory
|
||||
} else {
|
||||
FsNodeType::File
|
||||
},
|
||||
size: e.size,
|
||||
mtime_ms: e.modified.map(system_time_ms),
|
||||
is_symlink: e.is_symlink.then_some(true),
|
||||
// Root-relative path (divergent from the shell's absolute path).
|
||||
// A walk under a symlinked root yields canonical-root-spelled
|
||||
// entries, so strip either spelling.
|
||||
path: e
|
||||
.abs_path
|
||||
.strip_prefix(root)
|
||||
.or_else(|_| e.abs_path.strip_prefix(canonical_root))
|
||||
.unwrap_or(&e.abs_path)
|
||||
.to_string_lossy()
|
||||
.into_owned(),
|
||||
name: e.name,
|
||||
})
|
||||
.collect();
|
||||
|
||||
Ok(FsListRes {
|
||||
nodes,
|
||||
truncated: page.truncated,
|
||||
})
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// stat
|
||||
// =========================================================================
|
||||
|
||||
/// Stat `req.path`: existence, kind, size, mtime, and — for files — a
|
||||
/// full-content SHA-256 served through the workspace hash memo.
|
||||
pub(crate) async fn stat(ws: &WorkspaceHandle, req: &FsStatReq) -> WorkspaceResult<FsStatRes> {
|
||||
let abs = resolve(ws, &req.path).await?;
|
||||
let md = match tokio::fs::metadata(&abs).await {
|
||||
Ok(md) => md,
|
||||
// NotADirectory: a *file* sits mid-path (e.g. `a.txt/sub`) — for an
|
||||
// existence probe that is a miss, not an RPC error.
|
||||
Err(e)
|
||||
if e.kind() == std::io::ErrorKind::NotFound
|
||||
|| e.kind() == std::io::ErrorKind::NotADirectory =>
|
||||
{
|
||||
return Ok(FsStatRes {
|
||||
exists: false,
|
||||
node_type: None,
|
||||
size: None,
|
||||
mtime_ms: None,
|
||||
hash: None,
|
||||
});
|
||||
}
|
||||
Err(e) => {
|
||||
return Err(WorkspaceError::HubError(format!(
|
||||
"stat failed for {}: {e}",
|
||||
req.path
|
||||
)));
|
||||
}
|
||||
};
|
||||
let mtime_ms = md.modified().ok().map(system_time_ms);
|
||||
if md.is_dir() {
|
||||
return Ok(FsStatRes {
|
||||
exists: true,
|
||||
node_type: Some(FsNodeType::Directory),
|
||||
size: None,
|
||||
mtime_ms,
|
||||
hash: None,
|
||||
});
|
||||
}
|
||||
let size = md.len();
|
||||
let memo = &ws.shared.client_fs_hash_memo;
|
||||
let hash = match mtime_ms.and_then(|m| memo.lookup(&abs, size, m)) {
|
||||
Some(hash) => hash,
|
||||
None => {
|
||||
let (hash, _, _) = crate::handle::stream_hash_and_range(&abs, 0, 0)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
WorkspaceError::HubError(format!("hash failed for {}: {e}", req.path))
|
||||
})?;
|
||||
if let Some(m) = mtime_ms {
|
||||
memo.store(&abs, size, m, hash.clone());
|
||||
}
|
||||
hash
|
||||
}
|
||||
};
|
||||
Ok(FsStatRes {
|
||||
exists: true,
|
||||
node_type: Some(FsNodeType::File),
|
||||
size: Some(size),
|
||||
mtime_ms,
|
||||
hash: Some(hash),
|
||||
})
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// read_file
|
||||
// =========================================================================
|
||||
|
||||
/// Read a byte range of `req.path` (binary-safe, capped at
|
||||
/// `min(req.max_bytes, MAX_READ_BYTES)`) together with the full-file
|
||||
/// SHA-256. When the hash is memoized for the current `(size, mtime)`
|
||||
/// only the requested range is read; otherwise the whole file streams
|
||||
/// once (via the shared [`crate::handle::stream_hash_and_range`]) to
|
||||
/// hash it.
|
||||
pub(crate) async fn read_file(
|
||||
ws: &WorkspaceHandle,
|
||||
req: &FsReadFileReq,
|
||||
) -> WorkspaceResult<FsReadFileRes> {
|
||||
let abs = resolve(ws, &req.path).await?;
|
||||
let read_err =
|
||||
|e: std::io::Error| WorkspaceError::HubError(format!("read failed for {}: {e}", req.path));
|
||||
let md = tokio::fs::metadata(&abs).await.map_err(read_err)?;
|
||||
if md.is_dir() {
|
||||
return Err(WorkspaceError::HubError(format!(
|
||||
"not a file: {}",
|
||||
req.path
|
||||
)));
|
||||
}
|
||||
let size = md.len();
|
||||
let mtime_ms = md.modified().ok().map(system_time_ms);
|
||||
let offset = req.offset.unwrap_or(0);
|
||||
// Server-side clamp: a hostile/buggy caller cannot lift the per-chunk
|
||||
// budget past MAX_READ_BYTES regardless of `maxBytes`.
|
||||
let length = super::walk::clamp_read_length(req.length, req.max_bytes);
|
||||
|
||||
let memo = &ws.shared.client_fs_hash_memo;
|
||||
let (hash, chunk, size) = match mtime_ms.and_then(|m| memo.lookup(&abs, size, m)) {
|
||||
Some(hash) => {
|
||||
let chunk = super::walk::read_range(&abs, offset, length)
|
||||
.await
|
||||
.map_err(read_err)?;
|
||||
(hash, chunk, size)
|
||||
}
|
||||
None => {
|
||||
let (hash, chunk, streamed) =
|
||||
crate::handle::stream_hash_and_range(&abs, offset, length)
|
||||
.await
|
||||
.map_err(read_err)?;
|
||||
if let Some(m) = mtime_ms {
|
||||
memo.store(&abs, streamed, m, hash.clone());
|
||||
}
|
||||
(hash, chunk, streamed)
|
||||
}
|
||||
};
|
||||
|
||||
// Shared encoder keeps the paired wire fields coherent with one UTF-8
|
||||
// validation pass; `type` is text iff the bytes were valid UTF-8.
|
||||
let (payload, is_text) = super::walk::encode_chunk(chunk, req.encoding);
|
||||
let (content, content_base64) = match payload {
|
||||
super::walk::ChunkPayload::Text(t) => (Some(t), None),
|
||||
super::walk::ChunkPayload::Base64(b) => (None, Some(b)),
|
||||
};
|
||||
let content_type = if is_text {
|
||||
FsContentType::Text
|
||||
} else {
|
||||
FsContentType::Binary
|
||||
};
|
||||
Ok(FsReadFileRes {
|
||||
content,
|
||||
content_base64,
|
||||
size,
|
||||
hash,
|
||||
content_type,
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use base64::Engine;
|
||||
use kigi_workspace_types::rpc::fs::FsReadEncoding;
|
||||
|
||||
use super::*;
|
||||
use crate::handle::tests::make_handle;
|
||||
|
||||
fn list_req(path: &str) -> FsListReq {
|
||||
FsListReq {
|
||||
path: path.to_owned(),
|
||||
depth: 1,
|
||||
include_hidden: true,
|
||||
limit: 1000,
|
||||
offset: 0,
|
||||
follow_symlinks: true,
|
||||
respect_git_ignore: false,
|
||||
include_globs: vec![],
|
||||
exclude_globs: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
/// Fixture: root with files `b.txt`, `A.txt`, `c.txt` and dirs
|
||||
/// `Zeta`, `alpha`. Expected order: dirs first case-insensitive
|
||||
/// (`alpha`, `Zeta`), then files (`A.txt`, `b.txt`, `c.txt`).
|
||||
fn populate(root: &Path) {
|
||||
std::fs::write(root.join("b.txt"), b"bb").unwrap();
|
||||
std::fs::write(root.join("A.txt"), b"a").unwrap();
|
||||
std::fs::write(root.join("c.txt"), b"ccc").unwrap();
|
||||
std::fs::create_dir(root.join("Zeta")).unwrap();
|
||||
std::fs::create_dir(root.join("alpha")).unwrap();
|
||||
}
|
||||
|
||||
/// `list_blocking` against `dir` as both walk root and workspace root
|
||||
/// (canonicalized for the confinement check, like production).
|
||||
fn list_dir(dir: &Path, req: &FsListReq, max_collect: usize) -> FsListRes {
|
||||
let canonical = dunce::canonicalize(dir).unwrap();
|
||||
list_blocking(dir, dir, &canonical, req, max_collect).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_sorts_dirs_first_case_insensitive() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
populate(dir.path());
|
||||
let res = list_dir(dir.path(), &list_req(""), MAX_LIST_COLLECT);
|
||||
let names: Vec<&str> = res.nodes.iter().map(|n| n.name.as_str()).collect();
|
||||
assert_eq!(names, ["alpha", "Zeta", "A.txt", "b.txt", "c.txt"]);
|
||||
assert!(!res.truncated);
|
||||
assert_eq!(res.nodes[0].node_type, FsNodeType::Directory);
|
||||
assert_eq!(res.nodes[2].node_type, FsNodeType::File);
|
||||
assert_eq!(res.nodes[2].size, Some(1));
|
||||
assert!(res.nodes[2].mtime_ms.is_some());
|
||||
// Paths are workspace-root-relative.
|
||||
assert_eq!(res.nodes[2].path, "A.txt");
|
||||
}
|
||||
|
||||
/// Pagination slices the *sorted* listing, so consecutive pages have
|
||||
/// stable boundaries and concatenate to the full listing.
|
||||
#[test]
|
||||
fn list_paginates_post_sort_with_stable_boundaries() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
populate(dir.path());
|
||||
let full = list_dir(dir.path(), &list_req(""), MAX_LIST_COLLECT);
|
||||
|
||||
let mut paged = Vec::new();
|
||||
for page_start in [0u64, 2, 4] {
|
||||
let req = FsListReq {
|
||||
limit: 2,
|
||||
offset: page_start,
|
||||
..list_req("")
|
||||
};
|
||||
let page = list_dir(dir.path(), &req, MAX_LIST_COLLECT);
|
||||
// truncated while more entries remain past this page.
|
||||
assert_eq!(page.truncated, page_start + 2 < full.nodes.len() as u64);
|
||||
paged.extend(page.nodes);
|
||||
}
|
||||
assert_eq!(paged, full.nodes);
|
||||
|
||||
// Offset past the end yields an empty, non-truncated page.
|
||||
let req = FsListReq {
|
||||
offset: 100,
|
||||
..list_req("")
|
||||
};
|
||||
let page = list_dir(dir.path(), &req, MAX_LIST_COLLECT);
|
||||
assert!(page.nodes.is_empty());
|
||||
assert!(!page.truncated);
|
||||
}
|
||||
|
||||
/// The collection cap marks the result truncated even when the page
|
||||
/// itself is not full.
|
||||
#[test]
|
||||
fn list_collection_cap_truncates() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
populate(dir.path());
|
||||
let res = list_dir(dir.path(), &list_req(""), 2);
|
||||
assert_eq!(res.nodes.len(), 2);
|
||||
assert!(res.truncated);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_caps_limit_at_server_max() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
populate(dir.path());
|
||||
let req = FsListReq {
|
||||
limit: u32::MAX,
|
||||
..list_req("")
|
||||
};
|
||||
// Must not panic / overflow; the page is everything (< 1000).
|
||||
let res = list_dir(dir.path(), &req, MAX_LIST_COLLECT);
|
||||
assert_eq!(res.nodes.len(), 5);
|
||||
}
|
||||
|
||||
/// Regression: the list walk must not traverse — or
|
||||
/// even surface — in-root symlinks that resolve outside the workspace
|
||||
/// root, while symlinks staying inside the root keep working.
|
||||
#[test]
|
||||
#[cfg(unix)]
|
||||
fn list_excludes_symlink_escapes_mid_walk() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let root = dir.path();
|
||||
let outside = tempfile::tempdir().unwrap();
|
||||
std::fs::write(outside.path().join("secret.txt"), b"secret").unwrap();
|
||||
// Escaping symlink: root/escape_link -> <outside>.
|
||||
std::os::unix::fs::symlink(outside.path(), root.join("escape_link")).unwrap();
|
||||
// In-root symlink: root/good_link -> root/real_dir.
|
||||
std::fs::create_dir(root.join("real_dir")).unwrap();
|
||||
std::fs::write(root.join("real_dir/inner.txt"), b"inner").unwrap();
|
||||
std::os::unix::fs::symlink(root.join("real_dir"), root.join("good_link")).unwrap();
|
||||
|
||||
let req = FsListReq {
|
||||
depth: 2,
|
||||
follow_symlinks: true,
|
||||
..list_req("")
|
||||
};
|
||||
let res = list_dir(root, &req, MAX_LIST_COLLECT);
|
||||
let paths: Vec<&str> = res.nodes.iter().map(|n| n.path.as_str()).collect();
|
||||
assert!(
|
||||
!paths.iter().any(|p| p.contains("escape_link")),
|
||||
"escaping symlink (and its subtree) must be excluded: {paths:?}"
|
||||
);
|
||||
assert!(
|
||||
!paths.iter().any(|p| p.contains("secret.txt")),
|
||||
"outside entries must not be enumerated: {paths:?}"
|
||||
);
|
||||
// Confinement must not over-filter: in-root symlinks survive,
|
||||
// including descent through them.
|
||||
assert!(paths.contains(&"good_link"), "{paths:?}");
|
||||
assert!(paths.contains(&"good_link/inner.txt"), "{paths:?}");
|
||||
assert!(paths.contains(&"real_dir/inner.txt"), "{paths:?}");
|
||||
let good = res.nodes.iter().find(|n| n.path == "good_link").unwrap();
|
||||
assert_eq!(good.is_symlink, Some(true));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn memo_lookup_hits_and_invalidates_on_mismatch() {
|
||||
let memo = FileHashMemo::default();
|
||||
let path = Path::new("/ws/a.txt");
|
||||
memo.store(path, 10, 1000, "h1".into());
|
||||
assert_eq!(memo.lookup(path, 10, 1000).as_deref(), Some("h1"));
|
||||
// Size change ⇒ miss.
|
||||
assert_eq!(memo.lookup(path, 11, 1000), None);
|
||||
// Mtime change ⇒ miss.
|
||||
assert_eq!(memo.lookup(path, 10, 2000), None);
|
||||
// Re-store replaces the stale entry.
|
||||
memo.store(path, 11, 2000, "h2".into());
|
||||
assert_eq!(memo.lookup(path, 11, 2000).as_deref(), Some("h2"));
|
||||
assert_eq!(memo.lookup(path, 10, 1000), None);
|
||||
}
|
||||
|
||||
/// `stat` consults the memo (no re-hash for an unchanged file) and
|
||||
/// recomputes when `(size, mtime)` no longer match.
|
||||
#[tokio::test]
|
||||
async fn stat_uses_memo_until_file_changes() {
|
||||
let ws = make_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
std::fs::write(root.join("data.txt"), b"hello world").unwrap();
|
||||
|
||||
let req = FsStatReq {
|
||||
path: "data.txt".into(),
|
||||
};
|
||||
let first = stat(&ws, &req).await.unwrap();
|
||||
assert!(first.exists);
|
||||
assert_eq!(first.node_type, Some(FsNodeType::File));
|
||||
assert_eq!(first.size, Some(11));
|
||||
let real_hash = first.hash.clone().expect("hash for files");
|
||||
|
||||
// Plant a sentinel hash for the file's current (size, mtime). A
|
||||
// second stat must return the sentinel — proof it did not re-hash.
|
||||
let abs = root.join("data.txt");
|
||||
let md = std::fs::metadata(&abs).unwrap();
|
||||
let mtime = system_time_ms(md.modified().unwrap());
|
||||
ws.shared
|
||||
.client_fs_hash_memo
|
||||
.store(&abs, md.len(), mtime, "sentinel".into());
|
||||
let memoized = stat(&ws, &req).await.unwrap();
|
||||
assert_eq!(memoized.hash.as_deref(), Some("sentinel"));
|
||||
|
||||
// A size change invalidates the memo entry and re-hashes.
|
||||
std::fs::write(&abs, b"hello brave new world").unwrap();
|
||||
let rehashed = stat(&ws, &req).await.unwrap();
|
||||
let new_hash = rehashed.hash.expect("hash for files");
|
||||
assert_ne!(new_hash, "sentinel");
|
||||
assert_ne!(new_hash, real_hash);
|
||||
}
|
||||
|
||||
/// A path with a *file* as an intermediate component (`ENOTDIR`) is an
|
||||
/// existence miss, not an RPC error.
|
||||
#[tokio::test]
|
||||
async fn stat_enotdir_intermediate_reports_not_exists() {
|
||||
let ws = make_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
std::fs::write(root.join("file.txt"), b"x").unwrap();
|
||||
let res = stat(
|
||||
&ws,
|
||||
&FsStatReq {
|
||||
path: "file.txt/nested".into(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(!res.exists);
|
||||
assert_eq!(res.node_type, None);
|
||||
assert_eq!(res.hash, None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stat_missing_path_reports_not_exists() {
|
||||
let ws = make_handle();
|
||||
let res = stat(
|
||||
&ws,
|
||||
&FsStatReq {
|
||||
path: "nope.txt".into(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(!res.exists);
|
||||
assert_eq!(res.node_type, None);
|
||||
assert_eq!(res.hash, None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn read_file_chunks_are_binary_safe_and_capped() {
|
||||
let ws = make_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
// Non-UTF-8 payload: every byte value once.
|
||||
let payload: Vec<u8> = (0u8..=255).collect();
|
||||
std::fs::write(root.join("blob.bin"), &payload).unwrap();
|
||||
|
||||
let req = FsReadFileReq {
|
||||
path: "blob.bin".into(),
|
||||
// Bytes 200..210 are bare continuation bytes — never valid UTF-8.
|
||||
offset: Some(200),
|
||||
length: Some(50),
|
||||
max_bytes: 10, // cap below the requested length
|
||||
encoding: FsReadEncoding::Base64,
|
||||
};
|
||||
let res = read_file(&ws, &req).await.unwrap();
|
||||
assert_eq!(res.size, 256);
|
||||
assert_eq!(res.content, None);
|
||||
assert_eq!(res.content_type, FsContentType::Binary);
|
||||
let bytes = base64::engine::general_purpose::STANDARD
|
||||
.decode(res.content_base64.unwrap())
|
||||
.unwrap();
|
||||
assert_eq!(bytes, payload[200..210], "maxBytes caps the chunk");
|
||||
|
||||
// Full-file hash regardless of the requested range.
|
||||
use sha2::{Digest, Sha256};
|
||||
assert_eq!(res.hash, format!("{:x}", Sha256::digest(&payload)));
|
||||
|
||||
// Memoized second read (range-only fast path) returns the
|
||||
// identical chunk + hash.
|
||||
let again = read_file(&ws, &req).await.unwrap();
|
||||
assert_eq!(again.hash, res.hash);
|
||||
let again_bytes = base64::engine::general_purpose::STANDARD
|
||||
.decode(again.content_base64.unwrap())
|
||||
.unwrap();
|
||||
assert_eq!(again_bytes, payload[200..210]);
|
||||
}
|
||||
|
||||
/// `maxBytes` is server-capped at [`MAX_READ_BYTES`]:
|
||||
/// a caller-supplied huge budget cannot make the workspace buffer the
|
||||
/// whole file.
|
||||
#[tokio::test]
|
||||
async fn read_file_server_caps_max_bytes() {
|
||||
let ws = make_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
let payload = vec![0u8; (MAX_READ_BYTES + 100) as usize];
|
||||
std::fs::write(root.join("big.bin"), &payload).unwrap();
|
||||
|
||||
let res = read_file(
|
||||
&ws,
|
||||
&FsReadFileReq {
|
||||
path: "big.bin".into(),
|
||||
offset: None,
|
||||
length: None,
|
||||
max_bytes: u64::MAX,
|
||||
encoding: FsReadEncoding::Base64,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(res.size, payload.len() as u64);
|
||||
let bytes = base64::engine::general_purpose::STANDARD
|
||||
.decode(res.content_base64.unwrap())
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
bytes.len() as u64,
|
||||
MAX_READ_BYTES,
|
||||
"clamped to the server cap"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn read_file_utf8_default_and_binary_fallback() {
|
||||
let ws = make_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
std::fs::write(root.join("text.txt"), "héllo").unwrap();
|
||||
std::fs::write(root.join("bin.dat"), [0xff, 0xfe, 0x00]).unwrap();
|
||||
|
||||
let text = read_file(
|
||||
&ws,
|
||||
&FsReadFileReq {
|
||||
path: "text.txt".into(),
|
||||
offset: None,
|
||||
length: None,
|
||||
max_bytes: 1_048_576,
|
||||
encoding: FsReadEncoding::Utf8,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(text.content.as_deref(), Some("héllo"));
|
||||
assert_eq!(text.content_base64, None);
|
||||
assert_eq!(text.content_type, FsContentType::Text);
|
||||
|
||||
// Invalid UTF-8 under the utf8 default degrades to base64.
|
||||
let bin = read_file(
|
||||
&ws,
|
||||
&FsReadFileReq {
|
||||
path: "bin.dat".into(),
|
||||
offset: None,
|
||||
length: None,
|
||||
max_bytes: 1_048_576,
|
||||
encoding: FsReadEncoding::Utf8,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(bin.content, None);
|
||||
assert_eq!(bin.content_type, FsContentType::Binary);
|
||||
let bytes = base64::engine::general_purpose::STANDARD
|
||||
.decode(bin.content_base64.unwrap())
|
||||
.unwrap();
|
||||
assert_eq!(bytes, [0xff, 0xfe, 0x00]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn resolve_rejects_escapes() {
|
||||
let ws = make_handle();
|
||||
for path in ["/etc/passwd", "../escape.txt"] {
|
||||
let err = stat(
|
||||
&ws,
|
||||
&FsStatReq {
|
||||
path: path.to_owned(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect_err("escape must be rejected");
|
||||
assert!(matches!(err, WorkspaceError::HubError(_)), "{err:?}");
|
||||
}
|
||||
}
|
||||
|
||||
/// An absolute path *inside* the workspace root is accepted and stats
|
||||
/// the same file as its root-relative form.
|
||||
#[tokio::test]
|
||||
async fn resolve_accepts_absolute_within_root() {
|
||||
let ws = make_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
std::fs::write(root.join("data.txt"), b"hello").unwrap();
|
||||
|
||||
let rel = stat(
|
||||
&ws,
|
||||
&FsStatReq {
|
||||
path: "data.txt".into(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(rel.exists);
|
||||
|
||||
let abs_path = root.join("data.txt").to_string_lossy().into_owned();
|
||||
let abs = stat(&ws, &FsStatReq { path: abs_path }).await.unwrap();
|
||||
assert!(abs.exists);
|
||||
assert_eq!(abs.node_type, rel.node_type);
|
||||
assert_eq!(abs.size, rel.size);
|
||||
assert_eq!(abs.hash, rel.hash);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[cfg(unix)]
|
||||
async fn resolve_rejects_symlink_escape() {
|
||||
let ws = make_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
let outside = tempfile::tempdir().unwrap();
|
||||
std::fs::write(outside.path().join("secret.txt"), b"secret").unwrap();
|
||||
std::os::unix::fs::symlink(outside.path(), root.join("escape_link")).unwrap();
|
||||
|
||||
let err = read_file(
|
||||
&ws,
|
||||
&FsReadFileReq {
|
||||
path: "escape_link/secret.txt".into(),
|
||||
offset: None,
|
||||
length: None,
|
||||
max_bytes: 1_048_576,
|
||||
encoding: FsReadEncoding::Base64,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect_err("symlink escape must be rejected");
|
||||
assert!(
|
||||
err.to_string().contains("symlink escape"),
|
||||
"unexpected error: {err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn list_empty_path_lists_root() {
|
||||
let ws = make_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
std::fs::write(root.join("rooted.txt"), b"x").unwrap();
|
||||
let res = list(&ws, &list_req("")).await.unwrap();
|
||||
assert!(res.nodes.iter().any(|n| n.name == "rooted.txt"));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,306 @@
|
||||
//! Codebase Index Manager
|
||||
//!
|
||||
//! Manages code graph indexes for code navigation features (go-to-definition, go-to-references).
|
||||
//! Indexes are shared across sessions with the same cwd to avoid duplicate work.
|
||||
//!
|
||||
//! ## Deduplication
|
||||
//!
|
||||
//! Deduplication happens at two levels:
|
||||
//! 1. **Process-level**: `IndexManager::spawn()` ensures at most one manager per workspace per process
|
||||
//! 2. **Cross-process**: File-based locking prevents duplicate background operations
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::{Arc, Weak};
|
||||
|
||||
use kigi_codebase_graph::{IndexManager, IndexManagerConfig, IndexManagerHandle};
|
||||
|
||||
use kigi_tools::util::kigi_home::kigi_home;
|
||||
|
||||
/// Get the cache path for a cwd's index.
|
||||
///
|
||||
/// Cache is stored in: `~/.kigi/indexes/{url_encoded_cwd}/goto_index.bin`
|
||||
pub fn get_index_cache_path(cwd: &Path) -> PathBuf {
|
||||
let encoded = urlencoding::encode(&cwd.to_string_lossy()).into_owned();
|
||||
kigi_home()
|
||||
.join("indexes")
|
||||
.join(encoded)
|
||||
.join("goto_index.bin")
|
||||
}
|
||||
|
||||
/// Manages code graph indexes across sessions.
|
||||
///
|
||||
/// Wraps `IndexManager::spawn()` with cache-path config and cross-session
|
||||
/// handle reuse. Keeps only `Weak` refs — sessions hold the strong `Arc`s,
|
||||
/// so the actor is reaped when the last session in a git-root closes.
|
||||
pub struct CodebaseIndexManager {
|
||||
indexes: HashMap<PathBuf, Weak<IndexManagerHandle>>,
|
||||
}
|
||||
|
||||
impl Default for CodebaseIndexManager {
|
||||
fn default() -> Self {
|
||||
Self::new()
|
||||
}
|
||||
}
|
||||
|
||||
impl CodebaseIndexManager {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
indexes: HashMap::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Get or create the index for `cwd`. Returns `(handle, was_newly_spawned)`.
|
||||
/// Caller must hold the `Arc` to keep the index alive.
|
||||
pub fn get_or_create(&mut self, cwd: PathBuf) -> (Arc<IndexManagerHandle>, bool) {
|
||||
self.indexes.retain(|_, weak| weak.strong_count() > 0);
|
||||
|
||||
if let Some(handle) = self.indexes.get(&cwd).and_then(Weak::upgrade) {
|
||||
tracing::info!(
|
||||
cwd = %cwd.display(),
|
||||
event = "index_reused",
|
||||
"Codebase index already running — reusing shared handle"
|
||||
);
|
||||
return (handle, false);
|
||||
}
|
||||
|
||||
// Create cache directory if needed
|
||||
let cache_path = get_index_cache_path(&cwd);
|
||||
if let Some(parent) = cache_path.parent()
|
||||
&& let Err(e) = std::fs::create_dir_all(parent)
|
||||
{
|
||||
tracing::warn!(
|
||||
path = %parent.display(),
|
||||
error = %e,
|
||||
"Failed to create cache directory"
|
||||
);
|
||||
}
|
||||
|
||||
tracing::info!(
|
||||
cwd = %cwd.display(),
|
||||
cache_path = %cache_path.display(),
|
||||
event = "index_lazy_started",
|
||||
"Codebase index lazy-started — spawning IndexManager actor"
|
||||
);
|
||||
|
||||
// IndexManager::spawn() handles global deduplication.
|
||||
let config = IndexManagerConfig::new(cwd.clone()).with_cache_path(cache_path);
|
||||
let handle = IndexManager::spawn(config);
|
||||
|
||||
self.indexes.insert(cwd, Arc::downgrade(&handle));
|
||||
(handle, true)
|
||||
}
|
||||
|
||||
/// Get the running index for `cwd`, or `None` if not started / already reaped.
|
||||
pub fn get(&self, cwd: &Path) -> Option<Arc<IndexManagerHandle>> {
|
||||
self.indexes.get(cwd).and_then(Weak::upgrade)
|
||||
}
|
||||
|
||||
/// Returns the number of currently-live indexes (test helper).
|
||||
#[cfg(test)]
|
||||
pub(crate) fn active_count(&self) -> usize {
|
||||
self.indexes
|
||||
.values()
|
||||
.filter(|weak| weak.strong_count() > 0)
|
||||
.count()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_cache_path_encoding() {
|
||||
let cwd = Path::new("/Users/test/my project");
|
||||
let cache_path = get_index_cache_path(cwd);
|
||||
|
||||
// Should contain URL-encoded path
|
||||
assert!(cache_path.to_string_lossy().contains("%2F"));
|
||||
assert!(cache_path.to_string_lossy().ends_with("goto_index.bin"));
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Lazy-start mechanic tests
|
||||
//
|
||||
// These tests verify the core lazy-start behavior:
|
||||
// `CodebaseIndexManager::get()` returns None before the index is created,
|
||||
// which maps to `x.ai/code/status` reporting `reason: notStarted`.
|
||||
// `get_or_create()` is the lazy-start entry point called by
|
||||
// `MvpAgent::start_codebase_index_for_code_nav` on the first code-nav
|
||||
// request for an eligible session.
|
||||
// =========================================================================
|
||||
|
||||
/// An empty CodebaseIndexManager returns None for any path.
|
||||
///
|
||||
/// This is the steady-state before ANY code-nav request has been made.
|
||||
/// In `x.ai/code/status`, `resolve_index_handle()` calls
|
||||
/// `agent.get_codebase_index(cwd)` which calls `mgr.get(cwd)`.
|
||||
/// When this returns None the status reports `reason: notStarted` — the
|
||||
/// key non-starting guarantee from the plan.
|
||||
#[test]
|
||||
fn test_get_returns_none_before_any_index_created() {
|
||||
let mgr = CodebaseIndexManager::new();
|
||||
assert!(
|
||||
mgr.get(Path::new("/some/repo")).is_none(),
|
||||
"get() must return None before get_or_create() is called — this is \
|
||||
the CodebaseIndexManager state that causes code/status to report notStarted"
|
||||
);
|
||||
}
|
||||
|
||||
/// get() returns None even after another path was indexed.
|
||||
///
|
||||
/// Proves path-scoped isolation: a different cwd's index does not
|
||||
/// satisfy a lookup for an unrelated path.
|
||||
#[test]
|
||||
fn test_get_returns_none_for_different_cwd() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root_a = temp.path().join("repo_a");
|
||||
let root_b = temp.path().join("repo_b");
|
||||
std::fs::create_dir_all(&root_a).unwrap();
|
||||
|
||||
let mut mgr = CodebaseIndexManager::new();
|
||||
let (_handle, _) = mgr.get_or_create(root_a.clone());
|
||||
|
||||
// root_b was never indexed — must return None.
|
||||
assert!(
|
||||
mgr.get(&root_b).is_none(),
|
||||
"get() for an un-indexed path must return None (path-scoped isolation)"
|
||||
);
|
||||
}
|
||||
|
||||
/// After get_or_create() the same path is found by get().
|
||||
///
|
||||
/// This is the lazy-start path: `get_or_create` is called by
|
||||
/// `start_codebase_index_for_code_nav` on the first eligible code-nav
|
||||
/// request, after which `get_codebase_index` (used by `resolve_index_handle`
|
||||
/// in `code_status`) returns `Some`.
|
||||
#[test]
|
||||
fn test_get_finds_handle_after_get_or_create() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().to_path_buf();
|
||||
|
||||
let mut mgr = CodebaseIndexManager::new();
|
||||
|
||||
// Before lazy-start: get() returns None.
|
||||
assert!(mgr.get(&root).is_none(), "no index before get_or_create");
|
||||
|
||||
// Lazy-start: this is what start_codebase_index_for_code_nav calls.
|
||||
let (handle, was_new) = mgr.get_or_create(root.clone());
|
||||
assert!(was_new, "first get_or_create must report newly spawned");
|
||||
assert_eq!(mgr.active_count(), 1);
|
||||
|
||||
let found = mgr
|
||||
.get(&root)
|
||||
.expect("index must be visible after get_or_create");
|
||||
assert!(
|
||||
Arc::ptr_eq(&handle, &found),
|
||||
"get() must return the same Arc as get_or_create() — dedup / reuse"
|
||||
);
|
||||
}
|
||||
|
||||
/// The index is reaped once the last strong handle drops.
|
||||
///
|
||||
/// This is the eviction guarantee: the manager holds only a `Weak`, so when
|
||||
/// the last session pinning a git-root is torn down (here simulated by
|
||||
/// dropping the sole `Arc`), `get()` stops returning the handle and the
|
||||
/// entry no longer counts as active — no per-repo accumulation in a
|
||||
/// long-lived leader process.
|
||||
#[test]
|
||||
fn test_index_evicted_when_last_strong_ref_dropped() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().to_path_buf();
|
||||
|
||||
let mut mgr = CodebaseIndexManager::new();
|
||||
let (handle, _) = mgr.get_or_create(root.clone());
|
||||
assert_eq!(mgr.active_count(), 1, "index live while strong ref held");
|
||||
assert!(mgr.get(&root).is_some());
|
||||
|
||||
// The agent (here: this test) was the only strong owner.
|
||||
drop(handle);
|
||||
|
||||
assert!(
|
||||
mgr.get(&root).is_none(),
|
||||
"index must be released once the last strong ref drops"
|
||||
);
|
||||
assert_eq!(
|
||||
mgr.active_count(),
|
||||
0,
|
||||
"a reaped index must not count as active"
|
||||
);
|
||||
}
|
||||
|
||||
/// Shared index across sessions: reaped only after the last strong ref drops.
|
||||
#[test]
|
||||
fn test_shared_index_evicted_only_after_last_session_drops() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().to_path_buf();
|
||||
|
||||
let mut mgr = CodebaseIndexManager::new();
|
||||
|
||||
let (session_a, was_new_a) = mgr.get_or_create(root.clone());
|
||||
let (session_b, was_new_b) = mgr.get_or_create(root.clone());
|
||||
|
||||
assert!(was_new_a, "first session must spawn the index");
|
||||
assert!(!was_new_b, "second session must reuse the shared index");
|
||||
assert!(
|
||||
Arc::ptr_eq(&session_a, &session_b),
|
||||
"both sessions must share one index handle (Arc::ptr_eq)"
|
||||
);
|
||||
assert_eq!(
|
||||
mgr.active_count(),
|
||||
1,
|
||||
"two sessions in one git-root back exactly one index"
|
||||
);
|
||||
|
||||
// First session ends: the index stays warm for the surviving session.
|
||||
drop(session_a);
|
||||
assert!(
|
||||
mgr.get(&root).is_some(),
|
||||
"index must survive while another session still pins it"
|
||||
);
|
||||
assert_eq!(
|
||||
mgr.active_count(),
|
||||
1,
|
||||
"index still live after only the first session drops"
|
||||
);
|
||||
|
||||
// Last session ends: now — and only now — the index is reaped.
|
||||
drop(session_b);
|
||||
assert!(
|
||||
mgr.get(&root).is_none(),
|
||||
"index must be released once the LAST session drops its ref"
|
||||
);
|
||||
assert_eq!(
|
||||
mgr.active_count(),
|
||||
0,
|
||||
"a reaped shared index must not count as active"
|
||||
);
|
||||
}
|
||||
|
||||
/// A second get_or_create() for the same path reuses the existing handle.
|
||||
///
|
||||
/// This corresponds to "subsequent code-nav requests reuse the same shared
|
||||
/// handle" from the plan: once the index is running, all subsequent
|
||||
/// get_or_create calls for the same path return the existing Arc.
|
||||
#[test]
|
||||
fn test_get_or_create_is_idempotent_for_same_path() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().to_path_buf();
|
||||
|
||||
let mut mgr = CodebaseIndexManager::new();
|
||||
let (h1, was_new_1) = mgr.get_or_create(root.clone());
|
||||
let (h2, was_new_2) = mgr.get_or_create(root.clone());
|
||||
|
||||
assert!(was_new_1, "first call must report newly spawned");
|
||||
assert!(
|
||||
!was_new_2,
|
||||
"second call must report reused (not newly spawned)"
|
||||
);
|
||||
assert!(
|
||||
Arc::ptr_eq(&h1, &h2),
|
||||
"second get_or_create must return the same Arc as the first (index reuse)"
|
||||
);
|
||||
assert_eq!(mgr.active_count(), 1, "only one index for one path");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
use std::path::Path;
|
||||
use std::process::Stdio;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
use tokio::io::{AsyncBufReadExt, BufReader};
|
||||
use tokio::process::Command;
|
||||
|
||||
// Canonical in kigi-workspace-types; re-exported for existing paths.
|
||||
pub use kigi_workspace_types::rpc::search::{ContentMatch, ContentMatchFile, ContentSearchData};
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct ContentSearchParams {
|
||||
pub pattern: String,
|
||||
pub case_insensitive: bool,
|
||||
pub literal: bool,
|
||||
pub globs: Vec<String>,
|
||||
pub max_files: Option<usize>,
|
||||
pub max_matches: Option<usize>,
|
||||
pub respect_gitignore: bool,
|
||||
}
|
||||
|
||||
/// Batch of results sent during streaming search.
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct ContentSearchBatch {
|
||||
pub files: Vec<ContentMatchFile>,
|
||||
pub total_matches: usize,
|
||||
pub total_files: usize,
|
||||
pub done: bool,
|
||||
pub truncated: bool,
|
||||
}
|
||||
|
||||
const BATCH_INTERVAL_MS: u64 = 50;
|
||||
const DEFAULT_MAX_FILES: usize = 100;
|
||||
const DEFAULT_MAX_MATCHES: usize = 1000;
|
||||
|
||||
fn build_ripgrep_command(root: &Path, params: &ContentSearchParams) -> Command {
|
||||
let rg_path = crate::util::ripgrep::rg_path();
|
||||
|
||||
let mut cmd = Command::new(&rg_path);
|
||||
cmd.current_dir(root);
|
||||
cmd.stdout(Stdio::piped());
|
||||
cmd.stderr(Stdio::null());
|
||||
cmd.stdin(Stdio::null());
|
||||
kigi_tools::util::detach_command(&mut cmd);
|
||||
|
||||
cmd.arg("--json");
|
||||
cmd.arg("--line-number");
|
||||
|
||||
const DEFAULT_EXCLUSIONS: &[&str] = &["!.git/**", "!submodules/**", "!vendor/**"];
|
||||
for glob in DEFAULT_EXCLUSIONS {
|
||||
cmd.arg("--glob").arg(glob);
|
||||
}
|
||||
|
||||
cmd.arg("--max-filesize").arg("1M");
|
||||
cmd.arg("--max-count").arg("50");
|
||||
cmd.arg("--max-columns").arg("500");
|
||||
cmd.arg("--max-columns-preview");
|
||||
|
||||
if params.case_insensitive {
|
||||
cmd.arg("--ignore-case");
|
||||
}
|
||||
if params.literal {
|
||||
cmd.arg("--fixed-strings");
|
||||
}
|
||||
if !params.respect_gitignore {
|
||||
cmd.arg("--no-ignore");
|
||||
}
|
||||
for glob in ¶ms.globs {
|
||||
cmd.arg("--glob").arg(glob);
|
||||
}
|
||||
|
||||
cmd.arg("-e").arg(¶ms.pattern);
|
||||
cmd.arg(".");
|
||||
|
||||
cmd
|
||||
}
|
||||
|
||||
fn extract_match_positions(data: &serde_json::Value) -> (Option<usize>, Option<usize>) {
|
||||
data.get("submatches")
|
||||
.and_then(|s| s.as_array())
|
||||
.and_then(|arr| arr.first())
|
||||
.map(|first| {
|
||||
let start = first
|
||||
.get("start")
|
||||
.and_then(|s| s.as_u64())
|
||||
.map(|s| s as usize);
|
||||
let end = first
|
||||
.get("end")
|
||||
.and_then(|e| e.as_u64())
|
||||
.map(|e| e as usize);
|
||||
(start, end)
|
||||
})
|
||||
.unwrap_or((None, None))
|
||||
}
|
||||
|
||||
fn parse_match_from_json(data: &serde_json::Value) -> Option<ContentMatch> {
|
||||
let line_number = data.get("line_number").and_then(|l| l.as_u64())? as usize;
|
||||
let content = data
|
||||
.get("lines")
|
||||
.and_then(|l| l.get("text"))
|
||||
.and_then(|t| t.as_str())
|
||||
.unwrap_or("")
|
||||
.trim_end_matches('\n')
|
||||
.to_string();
|
||||
let (match_start, match_end) = extract_match_positions(data);
|
||||
|
||||
Some(ContentMatch {
|
||||
line: line_number,
|
||||
content,
|
||||
match_start,
|
||||
match_end,
|
||||
})
|
||||
}
|
||||
|
||||
fn parse_file_path_from_json(root: &Path, json: &serde_json::Value) -> Option<String> {
|
||||
let path = json
|
||||
.get("data")
|
||||
.and_then(|d| d.get("path"))
|
||||
.and_then(|p| p.get("text"))
|
||||
.and_then(|t| t.as_str())?;
|
||||
let normalized = path.strip_prefix("./").unwrap_or(path);
|
||||
if Path::new(normalized).is_absolute() {
|
||||
return Some(normalized.to_string());
|
||||
}
|
||||
Some(root.join(normalized).to_string_lossy().to_string())
|
||||
}
|
||||
|
||||
/// Streaming content search with batched status notifications.
|
||||
/// Set `cancel` to true to abort the search early.
|
||||
pub async fn content_search_streaming<F>(
|
||||
root: &Path,
|
||||
params: &ContentSearchParams,
|
||||
cancel: Arc<AtomicBool>,
|
||||
on_status: F,
|
||||
) -> anyhow::Result<ContentSearchData>
|
||||
where
|
||||
F: Fn(ContentSearchBatch) + Send + 'static,
|
||||
{
|
||||
let max_files = params.max_files.unwrap_or(DEFAULT_MAX_FILES);
|
||||
let max_matches = params.max_matches.unwrap_or(DEFAULT_MAX_MATCHES);
|
||||
|
||||
let mut cmd = build_ripgrep_command(root, params);
|
||||
let mut child = cmd
|
||||
.spawn()
|
||||
.map_err(|e| anyhow::anyhow!("Failed to spawn ripgrep: {}", e))?;
|
||||
|
||||
let stdout = child
|
||||
.stdout
|
||||
.take()
|
||||
.ok_or_else(|| anyhow::anyhow!("Failed to capture ripgrep stdout"))?;
|
||||
|
||||
let mut reader = BufReader::new(stdout).lines();
|
||||
let mut files: Vec<ContentMatchFile> = Vec::new();
|
||||
let mut current_file: Option<ContentMatchFile> = None;
|
||||
let mut total_matches = 0usize;
|
||||
let mut pending_files: Vec<ContentMatchFile> = Vec::new();
|
||||
let mut last_notify = Instant::now();
|
||||
let mut hit_limit = false;
|
||||
|
||||
while let Ok(Some(line)) = reader.next_line().await {
|
||||
if cancel.load(Ordering::Relaxed) {
|
||||
break;
|
||||
}
|
||||
|
||||
if line.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let json: serde_json::Value = match serde_json::from_str(&line) {
|
||||
Ok(v) => v,
|
||||
Err(_) => continue,
|
||||
};
|
||||
|
||||
match json.get("type").and_then(|t| t.as_str()) {
|
||||
Some("begin") => {
|
||||
if let Some(file) = current_file.take()
|
||||
&& !file.matches.is_empty()
|
||||
{
|
||||
pending_files.push(file.clone());
|
||||
files.push(file);
|
||||
}
|
||||
if let Some(path) = parse_file_path_from_json(root, &json) {
|
||||
current_file = Some(ContentMatchFile::new(path));
|
||||
}
|
||||
}
|
||||
Some("match") => {
|
||||
if let Some(ref mut file) = current_file
|
||||
&& let Some(data) = json.get("data")
|
||||
&& let Some(m) = parse_match_from_json(data)
|
||||
{
|
||||
file.matches.push(m);
|
||||
total_matches += 1;
|
||||
}
|
||||
}
|
||||
Some("end") => {
|
||||
if let Some(file) = current_file.take()
|
||||
&& !file.matches.is_empty()
|
||||
{
|
||||
pending_files.push(file.clone());
|
||||
files.push(file);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
|
||||
if files.len() >= max_files || total_matches >= max_matches {
|
||||
hit_limit = true;
|
||||
break;
|
||||
}
|
||||
|
||||
let should_notify = !pending_files.is_empty()
|
||||
&& last_notify.elapsed().as_millis() >= BATCH_INTERVAL_MS as u128;
|
||||
|
||||
if should_notify {
|
||||
on_status(ContentSearchBatch {
|
||||
files: std::mem::take(&mut pending_files),
|
||||
total_matches,
|
||||
total_files: files.len(),
|
||||
done: false,
|
||||
truncated: false,
|
||||
});
|
||||
tokio::task::yield_now().await;
|
||||
last_notify = Instant::now();
|
||||
}
|
||||
}
|
||||
|
||||
let cancelled = cancel.load(Ordering::Relaxed);
|
||||
if hit_limit || cancelled {
|
||||
let _ = child.kill().await;
|
||||
}
|
||||
let _ = child.wait().await;
|
||||
|
||||
if cancelled {
|
||||
let total_files = files.len();
|
||||
return Ok(ContentSearchData {
|
||||
files,
|
||||
total_matches,
|
||||
total_files,
|
||||
truncated: false,
|
||||
});
|
||||
}
|
||||
|
||||
if let Some(file) = current_file
|
||||
&& !file.matches.is_empty()
|
||||
&& files.len() < max_files
|
||||
{
|
||||
pending_files.push(file.clone());
|
||||
files.push(file);
|
||||
}
|
||||
|
||||
let truncated = hit_limit;
|
||||
let total_files = files.len();
|
||||
|
||||
on_status(ContentSearchBatch {
|
||||
files: pending_files,
|
||||
total_matches,
|
||||
total_files,
|
||||
done: true,
|
||||
truncated,
|
||||
});
|
||||
tokio::task::yield_now().await;
|
||||
|
||||
Ok(ContentSearchData {
|
||||
files,
|
||||
total_matches,
|
||||
total_files,
|
||||
truncated,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,560 @@
|
||||
//! Filesystem extension ops (`workspace.fs_*`) — the server-proxied backing
|
||||
//! for the shell's `x.ai/fs/*` ACP extension methods.
|
||||
//!
|
||||
//! These mirror the pure functions that previously lived only in the
|
||||
//! shell (`kigi-shell/src/session/file_system.rs`) so that, in proxy
|
||||
//! mode, a `x.ai/fs/*` request executes on the *remote* workspace server
|
||||
//! instead of the agent host. Each request type implements
|
||||
//! [`WorkspaceOp`], so it runs in-process for local sessions and routes
|
||||
//! over the server `workspace_rpc` tool for proxy sessions — identical wire
|
||||
//! output either way.
|
||||
//!
|
||||
//! Path resolution: an absolute `path` is used directly; a relative
|
||||
//! `path` is joined onto `cwd` (the per-session cwd the shell resolves
|
||||
//! and sends) or, when absent, the workspace root.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use base64::Engine;
|
||||
use chrono::Utc;
|
||||
|
||||
use crate::error::{WorkspaceError, WorkspaceResult};
|
||||
use crate::handle::WorkspaceHandle;
|
||||
use crate::workspace_ops::WorkspaceOp;
|
||||
|
||||
// Canonical in kigi-workspace-types; re-exported for existing paths.
|
||||
use kigi_workspace_types::rpc::fs::FsReadEncoding;
|
||||
pub use kigi_workspace_types::rpc::fs::{
|
||||
FsDeleteFileReq, FsExistsData, FsExistsReq, FsListData, FsListNode, FsListReq, FsReadFileData,
|
||||
FsReadFileReq, FsWriteFileReq,
|
||||
};
|
||||
|
||||
/// Resolve a request `path` to an absolute path. Absolute paths are used
|
||||
/// directly; relative paths join `cwd` (the shell-resolved per-session
|
||||
/// cwd) or, when absent, the workspace root.
|
||||
fn resolve_abs(
|
||||
path: &str,
|
||||
cwd: &Option<PathBuf>,
|
||||
ws: &WorkspaceHandle,
|
||||
) -> WorkspaceResult<PathBuf> {
|
||||
let p = Path::new(path);
|
||||
if p.is_absolute() {
|
||||
return Ok(p.to_path_buf());
|
||||
}
|
||||
let base = match cwd {
|
||||
Some(c) => c.clone(),
|
||||
None => ws.root_cwd()?,
|
||||
};
|
||||
Ok(base.join(p))
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl WorkspaceOp for FsListReq {
|
||||
async fn execute(
|
||||
&self,
|
||||
ws: &WorkspaceHandle,
|
||||
_session_id: Option<&str>,
|
||||
) -> WorkspaceResult<Self::Response> {
|
||||
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
|
||||
let (abs, confine_root) = ws.confine_to_workspace_root(&abs_unconfined).await?;
|
||||
// Off-executor: `list` does synchronous walk + metadata syscalls.
|
||||
let req = self.clone();
|
||||
tokio::task::spawn_blocking(move || list(&abs, &req, confine_root))
|
||||
.await
|
||||
.map_err(|e| WorkspaceError::HubError(e.to_string()))?
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl WorkspaceOp for FsExistsReq {
|
||||
async fn execute(
|
||||
&self,
|
||||
ws: &WorkspaceHandle,
|
||||
_session_id: Option<&str>,
|
||||
) -> WorkspaceResult<Self::Response> {
|
||||
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
|
||||
let (abs, _) = ws.confine_to_workspace_root(&abs_unconfined).await?;
|
||||
let exists = tokio::fs::try_exists(&abs).await.unwrap_or(false);
|
||||
Ok(FsExistsData { exists })
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl WorkspaceOp for FsReadFileReq {
|
||||
async fn execute(
|
||||
&self,
|
||||
ws: &WorkspaceHandle,
|
||||
_session_id: Option<&str>,
|
||||
) -> WorkspaceResult<Self::Response> {
|
||||
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
|
||||
let (abs, _) = ws.confine_to_workspace_root(&abs_unconfined).await?;
|
||||
|
||||
// Legacy full-file read path: preserves the pre-range wire output
|
||||
// (auto utf8/base64 detect, MIME `type`, `lineCount`).
|
||||
let ranged = self.offset.is_some()
|
||||
|| self.length.is_some()
|
||||
|| self.encoding == FsReadEncoding::Base64;
|
||||
if !ranged {
|
||||
let bytes = tokio::fs::read(&abs)
|
||||
.await
|
||||
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
|
||||
return Ok(build_file_entry(&bytes));
|
||||
}
|
||||
|
||||
// Binary-safe ranged read: `size` is the full file size, the
|
||||
// chunk is `[offset, offset + min(length, max_bytes, cap))`.
|
||||
let md = tokio::fs::metadata(&abs)
|
||||
.await
|
||||
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
|
||||
if md.is_dir() {
|
||||
return Err(WorkspaceError::HubError(format!(
|
||||
"not a file: {}",
|
||||
self.path
|
||||
)));
|
||||
}
|
||||
// Best-effort snapshot: a concurrent truncate/grow between here and
|
||||
// read_range can make `size` inconsistent with the returned chunk.
|
||||
let size = md.len();
|
||||
let offset = self.offset.unwrap_or(0);
|
||||
let length = super::walk::clamp_read_length(self.length, self.max_bytes);
|
||||
let chunk = super::walk::read_range(&abs, offset, length)
|
||||
.await
|
||||
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
|
||||
Ok(build_ranged_entry(chunk, size, self.encoding))
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl WorkspaceOp for FsWriteFileReq {
|
||||
async fn execute(
|
||||
&self,
|
||||
ws: &WorkspaceHandle,
|
||||
_session_id: Option<&str>,
|
||||
) -> WorkspaceResult<Self::Response> {
|
||||
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
|
||||
let (abs, _) = ws.confine_to_workspace_root(&abs_unconfined).await?;
|
||||
let content = self.content.clone();
|
||||
let create_dirs = self.create_dirs;
|
||||
tokio::task::spawn_blocking(move || -> std::io::Result<()> {
|
||||
if create_dirs && let Some(parent) = abs.parent() {
|
||||
std::fs::create_dir_all(parent)?;
|
||||
}
|
||||
std::fs::write(&abs, content.as_bytes())
|
||||
})
|
||||
.await
|
||||
.map_err(|e| WorkspaceError::HubError(e.to_string()))?
|
||||
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl WorkspaceOp for FsDeleteFileReq {
|
||||
async fn execute(
|
||||
&self,
|
||||
ws: &WorkspaceHandle,
|
||||
_session_id: Option<&str>,
|
||||
) -> WorkspaceResult<Self::Response> {
|
||||
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
|
||||
let (abs, _) = ws.confine_to_workspace_root(&abs_unconfined).await?;
|
||||
tokio::fs::remove_file(&abs)
|
||||
.await
|
||||
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Pure helpers — ported verbatim from the shell so output is identical.
|
||||
// =========================================================================
|
||||
|
||||
fn list(
|
||||
abs_path: &Path,
|
||||
req: &FsListReq,
|
||||
confine_to_canonical_root: Option<PathBuf>,
|
||||
) -> WorkspaceResult<FsListData> {
|
||||
// Confined to the canonical root when set (escaping symlinks not enumerated);
|
||||
// `None` (the default) walks unconfined.
|
||||
let page = super::walk::list_directory_paged(
|
||||
abs_path,
|
||||
super::walk::ListOptions {
|
||||
depth: req.depth,
|
||||
follow_symlinks: req.follow_symlinks,
|
||||
respect_git_ignore: req.respect_git_ignore,
|
||||
include_hidden: req.include_hidden,
|
||||
include_globs: &req.include_globs,
|
||||
exclude_globs: &req.exclude_globs,
|
||||
offset: req.offset,
|
||||
limit: req.limit,
|
||||
confine_to_canonical_root,
|
||||
},
|
||||
super::walk::MAX_LIST_COLLECT,
|
||||
);
|
||||
|
||||
let nodes: Vec<FsListNode> = page
|
||||
.entries
|
||||
.into_iter()
|
||||
.map(|e| FsListNode {
|
||||
node_type: if e.is_dir { "directory" } else { "file" }.to_string(),
|
||||
size: e.size,
|
||||
modified_at: e.modified.map(|st| {
|
||||
let dt: chrono::DateTime<Utc> = st.into();
|
||||
dt.to_rfc3339()
|
||||
}),
|
||||
is_symlink: e.is_symlink.then_some(true),
|
||||
path: e.abs_path.to_string_lossy().into_owned(),
|
||||
name: e.name,
|
||||
})
|
||||
.collect();
|
||||
|
||||
Ok(FsListData {
|
||||
nodes,
|
||||
truncated: page.truncated,
|
||||
})
|
||||
}
|
||||
|
||||
/// Map a binary-safe ranged chunk to the shell-facing `FsReadFileData`.
|
||||
/// `size` is the full file size; `lineCount` is omitted for ranged reads
|
||||
/// and the MIME `type` is a coarse text/binary tag (mid-file chunks make
|
||||
/// magic-byte sniffing meaningless).
|
||||
fn build_ranged_entry(chunk: Vec<u8>, size: u64, encoding: FsReadEncoding) -> FsReadFileData {
|
||||
let (payload, is_text) = super::walk::encode_chunk(chunk, encoding);
|
||||
let (content, content_base64) = match payload {
|
||||
super::walk::ChunkPayload::Text(t) => (t, None),
|
||||
super::walk::ChunkPayload::Base64(b) => (String::new(), Some(b)),
|
||||
};
|
||||
FsReadFileData {
|
||||
content,
|
||||
content_base64,
|
||||
size,
|
||||
line_count: None,
|
||||
content_type: if is_text {
|
||||
"text/plain".to_string()
|
||||
} else {
|
||||
"application/octet-stream".to_string()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn build_file_entry(bytes: &[u8]) -> FsReadFileData {
|
||||
let size = bytes.len() as u64;
|
||||
let inferred = infer::get(bytes).map(|t| t.mime_type().to_string());
|
||||
match String::from_utf8(bytes.to_vec()) {
|
||||
Ok(text) => FsReadFileData {
|
||||
line_count: Some(text.lines().count() as u64),
|
||||
content: text,
|
||||
content_base64: None,
|
||||
size,
|
||||
content_type: inferred.unwrap_or_else(|| "text/plain".to_string()),
|
||||
},
|
||||
Err(_) => FsReadFileData {
|
||||
content: String::new(),
|
||||
content_base64: Some(base64::engine::general_purpose::STANDARD.encode(bytes)),
|
||||
size,
|
||||
line_count: None,
|
||||
content_type: inferred.unwrap_or_else(|| "application/octet-stream".to_string()),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Tests for the pure helpers (no `WorkspaceHandle` required).
|
||||
// =========================================================================
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Build an `FsListReq` for a temp dir test. `path`/`cwd` are unused by
|
||||
/// `list` (it takes the resolved abs path directly); `respect_git_ignore`
|
||||
/// is off so the temp dir's location can't filter out our fixtures.
|
||||
fn list_req(limit: usize) -> FsListReq {
|
||||
FsListReq {
|
||||
path: String::new(),
|
||||
cwd: None,
|
||||
depth: 1,
|
||||
limit,
|
||||
offset: 0,
|
||||
include_hidden: true,
|
||||
follow_symlinks: true,
|
||||
respect_git_ignore: false,
|
||||
include_globs: Vec::new(),
|
||||
exclude_globs: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_file_entry_utf8_sets_content_and_line_count() {
|
||||
let bytes = b"line one\nline two\n";
|
||||
let entry = build_file_entry(bytes);
|
||||
assert_eq!(entry.content, "line one\nline two\n");
|
||||
assert!(entry.content_base64.is_none());
|
||||
assert_eq!(entry.line_count, Some(2));
|
||||
assert_eq!(entry.size, bytes.len() as u64);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_file_entry_invalid_utf8_uses_base64() {
|
||||
let bytes: &[u8] = &[0xff, 0xfe, 0x00];
|
||||
let entry = build_file_entry(bytes);
|
||||
assert!(entry.content.is_empty());
|
||||
assert_eq!(
|
||||
entry.content_base64,
|
||||
Some(base64::engine::general_purpose::STANDARD.encode(bytes)),
|
||||
);
|
||||
assert!(entry.line_count.is_none());
|
||||
assert_eq!(entry.size, bytes.len() as u64);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_enumerates_nodes_without_truncation() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let root = dir.path();
|
||||
std::fs::write(root.join("a.txt"), b"a").expect("write a");
|
||||
std::fs::write(root.join("b.txt"), b"bb").expect("write b");
|
||||
std::fs::create_dir(root.join("sub")).expect("mkdir sub");
|
||||
|
||||
let data = list(root, &list_req(1000), Some(root.to_path_buf())).expect("list");
|
||||
|
||||
let names: Vec<&str> = data.nodes.iter().map(|n| n.name.as_str()).collect();
|
||||
assert_eq!(data.nodes.len(), 3, "names: {names:?}");
|
||||
assert!(names.contains(&"a.txt"));
|
||||
assert!(names.contains(&"b.txt"));
|
||||
assert!(names.contains(&"sub"));
|
||||
assert!(!data.truncated);
|
||||
// Directories sort ahead of files.
|
||||
assert_eq!(data.nodes[0].name, "sub");
|
||||
assert_eq!(data.nodes[0].node_type, "directory");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_marks_truncated_when_limit_reached() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let root = dir.path();
|
||||
std::fs::write(root.join("a.txt"), b"a").expect("write a");
|
||||
std::fs::write(root.join("b.txt"), b"bb").expect("write b");
|
||||
std::fs::create_dir(root.join("sub")).expect("mkdir sub");
|
||||
|
||||
let data = list(root, &list_req(1), Some(root.to_path_buf())).expect("list");
|
||||
assert_eq!(data.nodes.len(), 1);
|
||||
assert!(data.truncated);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_paginates_with_offset() {
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let root = dir.path();
|
||||
for n in ["a.txt", "b.txt", "c.txt", "d.txt"] {
|
||||
std::fs::write(root.join(n), b"x").expect("write");
|
||||
}
|
||||
let names = |d: FsListData| d.nodes.into_iter().map(|n| n.name).collect::<Vec<_>>();
|
||||
|
||||
let mut req = list_req(2);
|
||||
let p0 = list(root, &req, Some(root.to_path_buf())).expect("list");
|
||||
assert!(p0.truncated);
|
||||
assert_eq!(names(p0), vec!["a.txt", "b.txt"]);
|
||||
|
||||
req.offset = 2;
|
||||
let p1 = list(root, &req, Some(root.to_path_buf())).expect("list");
|
||||
assert!(!p1.truncated, "last page is not truncated");
|
||||
assert_eq!(names(p1), vec!["c.txt", "d.txt"]);
|
||||
|
||||
// Offset past the end yields an empty, non-truncated page.
|
||||
req.offset = 10;
|
||||
let p2 = list(root, &req, Some(root.to_path_buf())).expect("list");
|
||||
assert!(!p2.truncated);
|
||||
assert!(p2.nodes.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_ranged_entry_encodes_utf8_and_binary() {
|
||||
// UTF-8 chunk under the default encoding → `content`, text/plain,
|
||||
// full `size` echoed, no line count.
|
||||
let e = build_ranged_entry(b"hello".to_vec(), 100, FsReadEncoding::Utf8);
|
||||
assert_eq!(e.content, "hello");
|
||||
assert!(e.content_base64.is_none());
|
||||
assert_eq!(e.size, 100);
|
||||
assert!(e.line_count.is_none());
|
||||
assert_eq!(e.content_type, "text/plain");
|
||||
|
||||
// Explicit base64 of valid UTF-8 stays text/plain but travels in
|
||||
// `contentBase64`.
|
||||
let e = build_ranged_entry(b"hi".to_vec(), 2, FsReadEncoding::Base64);
|
||||
assert!(e.content.is_empty());
|
||||
assert_eq!(
|
||||
e.content_base64,
|
||||
Some(base64::engine::general_purpose::STANDARD.encode(b"hi")),
|
||||
);
|
||||
assert_eq!(e.content_type, "text/plain");
|
||||
|
||||
// Non-UTF-8 bytes fall back to base64 + octet-stream.
|
||||
let raw = vec![0xff_u8, 0x00, 0xfe];
|
||||
let e = build_ranged_entry(raw.clone(), 3, FsReadEncoding::Utf8);
|
||||
assert!(e.content.is_empty());
|
||||
assert_eq!(
|
||||
e.content_base64,
|
||||
Some(base64::engine::general_purpose::STANDARD.encode(&raw)),
|
||||
);
|
||||
assert_eq!(e.content_type, "application/octet-stream");
|
||||
}
|
||||
|
||||
// Confinement (WorkspaceOp::execute) — covers both local and proxy dispatch.
|
||||
|
||||
#[tokio::test]
|
||||
async fn read_write_within_root_ok() {
|
||||
let ws = crate::handle::tests::make_confining_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
FsWriteFileReq {
|
||||
path: "sub/data.txt".into(),
|
||||
cwd: Some(root.clone()),
|
||||
content: "hello".into(),
|
||||
create_dirs: true,
|
||||
}
|
||||
.execute(&ws, None)
|
||||
.await
|
||||
.expect("in-root write must succeed");
|
||||
let data = FsReadFileReq {
|
||||
path: "sub/data.txt".into(),
|
||||
cwd: Some(root.clone()),
|
||||
offset: None,
|
||||
length: None,
|
||||
max_bytes: 1 << 20,
|
||||
encoding: FsReadEncoding::Utf8,
|
||||
}
|
||||
.execute(&ws, None)
|
||||
.await
|
||||
.expect("in-root read must succeed");
|
||||
assert_eq!(data.content, "hello");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn read_file_rejects_absolute_escape() {
|
||||
let ws = crate::handle::tests::make_confining_handle();
|
||||
let err = FsReadFileReq {
|
||||
path: "/etc/passwd".into(),
|
||||
cwd: None,
|
||||
offset: None,
|
||||
length: None,
|
||||
max_bytes: 1 << 20,
|
||||
encoding: FsReadEncoding::Utf8,
|
||||
}
|
||||
.execute(&ws, None)
|
||||
.await
|
||||
.expect_err("absolute escape must be rejected");
|
||||
assert!(
|
||||
err.to_string().contains("workspace root"),
|
||||
"unexpected error: {err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[cfg(unix)]
|
||||
async fn read_file_rejects_symlink_escape() {
|
||||
use std::os::unix::fs::symlink;
|
||||
let ws = crate::handle::tests::make_confining_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
let outside = tempfile::tempdir().unwrap();
|
||||
std::fs::write(outside.path().join("secret.txt"), b"secret").unwrap();
|
||||
symlink(outside.path(), root.join("escape_link")).unwrap();
|
||||
|
||||
let err = FsReadFileReq {
|
||||
path: "escape_link/secret.txt".into(),
|
||||
cwd: Some(root.clone()),
|
||||
offset: None,
|
||||
length: None,
|
||||
max_bytes: 1 << 20,
|
||||
encoding: FsReadEncoding::Utf8,
|
||||
}
|
||||
.execute(&ws, None)
|
||||
.await
|
||||
.expect_err("symlink escape must be rejected");
|
||||
assert!(
|
||||
err.to_string().contains("workspace root"),
|
||||
"unexpected error: {err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[cfg(unix)]
|
||||
async fn write_file_rejects_symlink_escape() {
|
||||
use std::os::unix::fs::symlink;
|
||||
let ws = crate::handle::tests::make_confining_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
let outside = tempfile::tempdir().unwrap();
|
||||
symlink(outside.path(), root.join("escape_link")).unwrap();
|
||||
|
||||
let err = FsWriteFileReq {
|
||||
path: "escape_link/injected.txt".into(),
|
||||
cwd: Some(root.clone()),
|
||||
content: "x".into(),
|
||||
create_dirs: true,
|
||||
}
|
||||
.execute(&ws, None)
|
||||
.await
|
||||
.expect_err("symlink escape write must be rejected");
|
||||
assert!(
|
||||
err.to_string().contains("workspace root"),
|
||||
"unexpected error: {err}"
|
||||
);
|
||||
assert!(
|
||||
!outside.path().join("injected.txt").exists(),
|
||||
"write must not land outside the workspace root"
|
||||
);
|
||||
}
|
||||
|
||||
// A *dangling* in-root symlink (target outside root, not yet created) must
|
||||
// not let a write escape via `open(O_CREAT)` following the link.
|
||||
#[tokio::test]
|
||||
#[cfg(unix)]
|
||||
async fn write_file_rejects_dangling_symlink_escape() {
|
||||
use std::os::unix::fs::symlink;
|
||||
let ws = crate::handle::tests::make_confining_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
let outside = tempfile::tempdir().unwrap();
|
||||
let target = outside.path().join("new.txt");
|
||||
symlink(&target, root.join("lnk")).unwrap();
|
||||
|
||||
let err = FsWriteFileReq {
|
||||
path: "lnk".into(),
|
||||
cwd: Some(root.clone()),
|
||||
content: "x".into(),
|
||||
create_dirs: true,
|
||||
}
|
||||
.execute(&ws, None)
|
||||
.await
|
||||
.expect_err("dangling symlink escape write must be rejected");
|
||||
assert!(
|
||||
err.to_string().contains("workspace root"),
|
||||
"unexpected error: {err}"
|
||||
);
|
||||
assert!(
|
||||
!target.exists(),
|
||||
"write must not create the file outside root"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[cfg(unix)]
|
||||
async fn list_excludes_symlink_escape() {
|
||||
use std::os::unix::fs::symlink;
|
||||
let ws = crate::handle::tests::make_confining_handle();
|
||||
let root = ws.root_cwd().unwrap();
|
||||
let outside = tempfile::tempdir().unwrap();
|
||||
std::fs::write(outside.path().join("secret.txt"), b"x").unwrap();
|
||||
symlink(outside.path(), root.join("escape_link")).unwrap();
|
||||
std::fs::write(root.join("inside.txt"), b"y").unwrap();
|
||||
|
||||
let mut req = list_req(1000);
|
||||
req.path = ".".into();
|
||||
req.cwd = Some(root.clone());
|
||||
req.depth = 2;
|
||||
let data = req.execute(&ws, None).await.expect("list must succeed");
|
||||
let names: Vec<&str> = data.nodes.iter().map(|n| n.name.as_str()).collect();
|
||||
assert!(names.contains(&"inside.txt"), "in-root file: {names:?}");
|
||||
assert!(
|
||||
!data.nodes.iter().any(|n| n.path.contains("secret.txt")),
|
||||
"escaping symlink target must not be enumerated: {names:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,526 @@
|
||||
//! Generates the file-tree based on how we are using it during training
|
||||
//! This gives the model an overview of the project and helps it navigate and understand
|
||||
//! the repository better, cold-starting the exploration
|
||||
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::Instant;
|
||||
|
||||
use dashmap::DashMap;
|
||||
use ignore::{WalkBuilder, WalkState};
|
||||
|
||||
use crate::file_system::FsError;
|
||||
|
||||
/// Number of threads for parallel directory walking
|
||||
const NUM_WALK_THREADS: usize = 8;
|
||||
|
||||
/// Configuration for limiting file tree traversal to prevent runaway I/O
|
||||
/// on very large or deeply nested directories.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct ListContentsLimits {
|
||||
/// Maximum number of characters in the output
|
||||
pub max_characters: usize,
|
||||
/// Maximum depth to traverse (0 = root only)
|
||||
pub max_depth: usize,
|
||||
/// Maximum number of directories to visit during traversal
|
||||
pub max_dirs_visited: usize,
|
||||
}
|
||||
|
||||
impl Default for ListContentsLimits {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_characters: 10_000,
|
||||
max_depth: 12,
|
||||
max_dirs_visited: 2000,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl ListContentsLimits {
|
||||
pub fn new(max_characters: usize, max_depth: usize, max_dirs_visited: usize) -> Self {
|
||||
Self {
|
||||
max_characters,
|
||||
max_depth,
|
||||
max_dirs_visited,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn get_top_exts(files: &[String]) -> Vec<(String, usize)> {
|
||||
let mut ext_counts: HashMap<String, usize> = HashMap::new();
|
||||
for item in files {
|
||||
let ext = if let Some(e) = Path::new(item).extension() {
|
||||
format!(".{}", e.to_str().unwrap_or("").to_lowercase())
|
||||
} else {
|
||||
String::new()
|
||||
};
|
||||
*ext_counts.entry(ext).or_insert(0) += 1;
|
||||
}
|
||||
let mut vec: Vec<_> = ext_counts.into_iter().collect();
|
||||
vec.sort_by_key(|&(_, count)| std::cmp::Reverse(count));
|
||||
vec
|
||||
}
|
||||
|
||||
fn get_file_ext_str(files: &[String], k: usize) -> String {
|
||||
let top_exts = get_top_exts(files);
|
||||
let include_dots = top_exts.len() > k
|
||||
|| (top_exts.len() == k && top_exts.iter().any(|(ext, _)| ext.is_empty()));
|
||||
let top_k_exts = &top_exts[0..std::cmp::min(k, top_exts.len())];
|
||||
if top_k_exts.is_empty() {
|
||||
return String::new();
|
||||
}
|
||||
if top_k_exts.len() == 1 && top_k_exts[0].0.is_empty() {
|
||||
return "(...)".to_string();
|
||||
}
|
||||
let filtered_top_k_exts: Vec<_> = top_k_exts
|
||||
.iter()
|
||||
.filter(|(ext, _)| !ext.is_empty())
|
||||
.collect();
|
||||
let top_counts = filtered_top_k_exts
|
||||
.iter()
|
||||
.map(|(ext, cnt)| format!("{} *{}", cnt, ext))
|
||||
.collect::<Vec<_>>()
|
||||
.join(", ");
|
||||
if include_dots {
|
||||
format!("({top_counts}, ...)")
|
||||
} else if top_counts.is_empty() {
|
||||
String::new()
|
||||
} else {
|
||||
format!("({top_counts})")
|
||||
}
|
||||
}
|
||||
|
||||
/// Pre-collected directory contents from a single walk
|
||||
struct DirContents {
|
||||
files: Vec<String>,
|
||||
dirs: Vec<String>,
|
||||
}
|
||||
|
||||
/// Performs a single parallel walk and collects all directory contents into a map.
|
||||
/// Returns a map from directory path -> (files, subdirs) in that directory.
|
||||
fn collect_all_contents(
|
||||
root: &Path,
|
||||
max_depth: usize,
|
||||
max_dirs: usize,
|
||||
) -> Result<HashMap<PathBuf, DirContents>, FsError> {
|
||||
let _timer = (); // instrumentation_timer noop (dev infra)
|
||||
|
||||
let contents_map: DashMap<PathBuf, DirContents> = DashMap::new();
|
||||
let files_count = std::sync::atomic::AtomicUsize::new(0);
|
||||
let dirs_count = std::sync::atomic::AtomicUsize::new(0);
|
||||
let entries_visited = std::sync::atomic::AtomicUsize::new(0);
|
||||
|
||||
// Initialize root entry
|
||||
contents_map.insert(
|
||||
root.to_path_buf(),
|
||||
DirContents {
|
||||
files: Vec::new(),
|
||||
dirs: Vec::new(),
|
||||
},
|
||||
);
|
||||
|
||||
let walker = WalkBuilder::new(root)
|
||||
.max_depth(Some(max_depth + 1)) // +1 because depth 0 is root itself
|
||||
.follow_links(false)
|
||||
.same_file_system(true)
|
||||
.ignore(true)
|
||||
.git_ignore(true)
|
||||
.git_global(true)
|
||||
.git_exclude(true)
|
||||
.hidden(true)
|
||||
.threads(NUM_WALK_THREADS)
|
||||
.build_parallel();
|
||||
|
||||
tracing::debug!(
|
||||
root = %root.display(),
|
||||
max_depth = max_depth,
|
||||
max_dirs = max_dirs,
|
||||
threads = NUM_WALK_THREADS,
|
||||
"Starting parallel file walk"
|
||||
);
|
||||
|
||||
{
|
||||
let _timer = (); // instrumentation_timer noop (dev infra)
|
||||
walker.run(|| {
|
||||
let contents_map = &contents_map;
|
||||
let files_count = &files_count;
|
||||
let dirs_count = &dirs_count;
|
||||
let entries_visited = &entries_visited;
|
||||
Box::new(move |entry| {
|
||||
entries_visited.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
|
||||
|
||||
// Stop early if we've collected enough directories
|
||||
if dirs_count.load(std::sync::atomic::Ordering::Relaxed) >= max_dirs {
|
||||
return WalkState::Quit;
|
||||
}
|
||||
|
||||
let entry = match entry {
|
||||
Ok(e) => e,
|
||||
Err(_) => return WalkState::Continue,
|
||||
};
|
||||
|
||||
// Skip root itself
|
||||
if entry.depth() == 0 {
|
||||
return WalkState::Continue;
|
||||
}
|
||||
|
||||
let Some(file_type) = entry.file_type() else {
|
||||
return WalkState::Continue;
|
||||
};
|
||||
|
||||
let path = entry.path();
|
||||
let Some(parent) = path.parent() else {
|
||||
return WalkState::Continue;
|
||||
};
|
||||
let name = entry.file_name().to_string_lossy().to_string();
|
||||
|
||||
if file_type.is_dir() {
|
||||
dirs_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
|
||||
contents_map
|
||||
.entry(parent.to_path_buf())
|
||||
.or_insert_with(|| DirContents {
|
||||
files: Vec::new(),
|
||||
dirs: Vec::new(),
|
||||
})
|
||||
.dirs
|
||||
.push(format!("{name}/"));
|
||||
// Pre-create entry for this directory (even if empty)
|
||||
contents_map
|
||||
.entry(path.to_path_buf())
|
||||
.or_insert_with(|| DirContents {
|
||||
files: Vec::new(),
|
||||
dirs: Vec::new(),
|
||||
});
|
||||
} else if file_type.is_file() {
|
||||
files_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
|
||||
contents_map
|
||||
.entry(parent.to_path_buf())
|
||||
.or_insert_with(|| DirContents {
|
||||
files: Vec::new(),
|
||||
dirs: Vec::new(),
|
||||
})
|
||||
.files
|
||||
.push(name);
|
||||
}
|
||||
|
||||
WalkState::Continue
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
let _entries_total = entries_visited.load(std::sync::atomic::Ordering::Relaxed);
|
||||
// (instrumentation_timer.with_field calls removed — dev-only, not part of Phase 1 move)
|
||||
|
||||
let mut contents_map: HashMap<PathBuf, DirContents> = {
|
||||
let _timer = (); // instrumentation_timer noop (dev infra)
|
||||
contents_map.into_iter().collect()
|
||||
};
|
||||
|
||||
// Sort all entries for stable output
|
||||
{
|
||||
let _timer = (); // instrumentation_timer noop (dev infra)
|
||||
for contents in contents_map.values_mut() {
|
||||
contents.files.sort_by_cached_key(|n| n.to_lowercase());
|
||||
contents.dirs.sort_by_cached_key(|n| n.to_lowercase());
|
||||
}
|
||||
}
|
||||
|
||||
Ok(contents_map)
|
||||
}
|
||||
|
||||
struct DirectoryNode {
|
||||
depth: usize,
|
||||
path: PathBuf,
|
||||
files: Vec<String>,
|
||||
dirs: Vec<String>,
|
||||
summary_str: String,
|
||||
children: Option<HashMap<String, DirectoryNode>>,
|
||||
num_listed_files: usize,
|
||||
}
|
||||
|
||||
impl DirectoryNode {
|
||||
fn new(path: PathBuf, depth: usize, contents: &HashMap<PathBuf, DirContents>) -> Self {
|
||||
let (files, dirs) = contents
|
||||
.get(&path)
|
||||
.map(|c| (c.files.clone(), c.dirs.clone()))
|
||||
.unwrap_or_default();
|
||||
|
||||
let mut node = Self {
|
||||
depth,
|
||||
path,
|
||||
files,
|
||||
dirs,
|
||||
summary_str: String::new(),
|
||||
children: None,
|
||||
num_listed_files: 0,
|
||||
};
|
||||
node.summary_str = node.get_remaining_str(&[], false, 3);
|
||||
node
|
||||
}
|
||||
|
||||
fn get_remaining_str(
|
||||
&self,
|
||||
excluded_files: &[String],
|
||||
exclude_all_dirs: bool,
|
||||
k: usize,
|
||||
) -> String {
|
||||
let remaining_files: Vec<String> = self
|
||||
.files
|
||||
.iter()
|
||||
.filter(|f| !excluded_files.contains(*f))
|
||||
.cloned()
|
||||
.collect();
|
||||
let mut file_ext_str = get_file_ext_str(&remaining_files, k);
|
||||
if !file_ext_str.is_empty() {
|
||||
file_ext_str.push(' ');
|
||||
}
|
||||
let file_count = remaining_files.len();
|
||||
let dir_count = if exclude_all_dirs { 0 } else { self.dirs.len() };
|
||||
let indent = " ".repeat(self.depth + 1);
|
||||
format!("{indent}- [+{file_count} files {file_ext_str}& {dir_count} dirs]")
|
||||
}
|
||||
|
||||
fn is_expanded(&self) -> bool {
|
||||
self.children.is_some()
|
||||
}
|
||||
|
||||
fn expand_children(&mut self, contents: &HashMap<PathBuf, DirContents>) {
|
||||
if self.is_expanded() {
|
||||
return;
|
||||
}
|
||||
let mut children: HashMap<String, DirectoryNode> = HashMap::new();
|
||||
for dir in &self.dirs {
|
||||
let child_path = self.path.join(dir.trim_end_matches('/'));
|
||||
let dir_node = DirectoryNode::new(child_path, self.depth + 1, contents);
|
||||
children.insert(dir.clone(), dir_node);
|
||||
}
|
||||
self.children = Some(children);
|
||||
self.num_listed_files = std::cmp::min(3, self.files.len());
|
||||
}
|
||||
|
||||
fn unexpand_children(&mut self) {
|
||||
self.children = None;
|
||||
self.num_listed_files = 0;
|
||||
}
|
||||
|
||||
fn subitem_str(&self, subitem: &str) -> String {
|
||||
let indent = " ".repeat(self.depth + 1);
|
||||
format!("{indent}- {subitem}")
|
||||
}
|
||||
|
||||
fn get_complete_str(&self) -> String {
|
||||
assert!(self.is_expanded());
|
||||
let children = self.children.as_ref().unwrap();
|
||||
let mut remaining_subitems = self.dirs.clone();
|
||||
remaining_subitems.extend(self.files[0..self.num_listed_files].iter().cloned());
|
||||
remaining_subitems.sort_by_key(|s| s.to_lowercase());
|
||||
let mut curr_str = String::new();
|
||||
for subitem in remaining_subitems {
|
||||
curr_str.push_str(&self.subitem_str(&subitem));
|
||||
curr_str.push('\n');
|
||||
if let Some(child) = children.get(&subitem) {
|
||||
if child.is_expanded() {
|
||||
curr_str.push_str(&child.get_complete_str());
|
||||
curr_str.push('\n');
|
||||
} else {
|
||||
curr_str.push_str(&child.summary_str);
|
||||
curr_str.push('\n');
|
||||
}
|
||||
}
|
||||
}
|
||||
if self.files.len() > self.num_listed_files {
|
||||
let remaining_str =
|
||||
self.get_remaining_str(&self.files[0..self.num_listed_files], true, 3);
|
||||
curr_str.push_str(&remaining_str);
|
||||
}
|
||||
curr_str.trim_end_matches('\n').to_string()
|
||||
}
|
||||
}
|
||||
|
||||
/// Creates the project overview
|
||||
pub async fn list_contents(
|
||||
path: impl Into<PathBuf>,
|
||||
limits: ListContentsLimits,
|
||||
) -> Result<String, FsError> {
|
||||
let _timer = (); // instrumentation_timer noop (dev infra)
|
||||
let t_total = Instant::now();
|
||||
let path: PathBuf = path.into();
|
||||
|
||||
// Use this only for the printed header
|
||||
let path_head = {
|
||||
let s = path.to_string_lossy().replace('\\', "/");
|
||||
if s.ends_with('/') {
|
||||
s.to_owned()
|
||||
} else {
|
||||
format!("{s}/")
|
||||
}
|
||||
};
|
||||
|
||||
let lim_characters = limits.max_characters + path_head.len();
|
||||
|
||||
// Single walk to collect all directory contents
|
||||
let path_clone = path.clone();
|
||||
let max_depth = limits.max_depth;
|
||||
let max_dirs = limits.max_dirs_visited;
|
||||
let contents = {
|
||||
let _timer = (); // instrumentation_timer noop (dev infra)
|
||||
tokio::task::spawn_blocking(move || collect_all_contents(&path_clone, max_depth, max_dirs))
|
||||
.await
|
||||
.map_err(|e| FsError::Other(format!("walk join error: {e}")))?
|
||||
}?;
|
||||
|
||||
let t_walk = t_total.elapsed();
|
||||
|
||||
let (
|
||||
mut root_node,
|
||||
mut remaining_chars,
|
||||
to_fit_files,
|
||||
dirs_visited,
|
||||
max_depth_reached,
|
||||
depth_limit_hit,
|
||||
dirs_limit_hit,
|
||||
) = {
|
||||
let _timer = (); // instrumentation_timer noop (dev infra)
|
||||
|
||||
let mut root_node = DirectoryNode::new(path, 0, &contents);
|
||||
|
||||
let min_chars = path_head.len() + root_node.summary_str.len();
|
||||
if min_chars > lim_characters {
|
||||
return Err(FsError::Other(format!(
|
||||
"Minimum possible string is too long for character limit, {} > {}",
|
||||
min_chars, lim_characters
|
||||
)));
|
||||
}
|
||||
|
||||
let mut remaining_chars = lim_characters - min_chars;
|
||||
let mut to_fit_files = true;
|
||||
let mut dirs_visited: usize = 1; // Count root as visited
|
||||
let mut max_depth_reached: usize = 0;
|
||||
let mut depth_limit_hit = false;
|
||||
let mut dirs_limit_hit = false;
|
||||
let mut q: VecDeque<&mut DirectoryNode> = VecDeque::new();
|
||||
q.push_back(&mut root_node);
|
||||
while let Some(node) = q.pop_front() {
|
||||
// Check if we've hit the max directories limit
|
||||
if dirs_visited >= limits.max_dirs_visited {
|
||||
dirs_limit_hit = true;
|
||||
to_fit_files = false;
|
||||
break;
|
||||
}
|
||||
|
||||
remaining_chars += node.summary_str.len();
|
||||
node.expand_children(&contents);
|
||||
dirs_visited += node.dirs.len();
|
||||
max_depth_reached = max_depth_reached.max(node.depth);
|
||||
|
||||
let test_str = node.get_complete_str();
|
||||
let new_additional_len = test_str.replace('\n', "").len();
|
||||
if new_additional_len > remaining_chars {
|
||||
node.unexpand_children();
|
||||
to_fit_files = false;
|
||||
break;
|
||||
}
|
||||
remaining_chars -= new_additional_len;
|
||||
if let Some(children_map) = node.children.as_mut() {
|
||||
let mut child_values: Vec<&mut DirectoryNode> = children_map.values_mut().collect();
|
||||
child_values.sort_by_key(|node| node.path.to_string_lossy().to_lowercase());
|
||||
|
||||
// Filter out children that exceed max_depth
|
||||
for child in child_values {
|
||||
if child.depth > limits.max_depth {
|
||||
depth_limit_hit = true;
|
||||
continue;
|
||||
}
|
||||
q.push_back(child);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
(
|
||||
root_node,
|
||||
remaining_chars,
|
||||
to_fit_files,
|
||||
dirs_visited,
|
||||
max_depth_reached,
|
||||
depth_limit_hit,
|
||||
dirs_limit_hit,
|
||||
)
|
||||
};
|
||||
|
||||
{
|
||||
let _timer = (); // instrumentation_timer noop (dev infra)
|
||||
let mut q: VecDeque<&mut DirectoryNode> = VecDeque::new();
|
||||
if to_fit_files {
|
||||
q.push_back(&mut root_node);
|
||||
}
|
||||
let mut file_done = false;
|
||||
while let Some(node) = q.pop_front() {
|
||||
if !node.is_expanded() {
|
||||
continue;
|
||||
}
|
||||
let num_file_limit = node.files.len();
|
||||
for i in node.num_listed_files..num_file_limit {
|
||||
let new_additional_len = node.subitem_str(&node.files[i]).len();
|
||||
if new_additional_len <= remaining_chars {
|
||||
node.num_listed_files = i + 1;
|
||||
remaining_chars -= new_additional_len;
|
||||
} else {
|
||||
file_done = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if file_done {
|
||||
break;
|
||||
}
|
||||
if let Some(children_map) = node.children.as_mut() {
|
||||
let mut child_values: Vec<&mut DirectoryNode> = children_map.values_mut().collect();
|
||||
child_values.sort_by_key(|node| node.path.to_string_lossy().to_lowercase());
|
||||
q.extend(child_values);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let output = {
|
||||
let _timer = (); // instrumentation_timer noop (dev infra)
|
||||
let mut output = format!("{path_head}\n");
|
||||
if root_node.is_expanded() {
|
||||
output.push_str(&root_node.get_complete_str());
|
||||
} else {
|
||||
output.push_str(&root_node.summary_str);
|
||||
}
|
||||
output
|
||||
};
|
||||
|
||||
// Log warnings when limits are hit
|
||||
if depth_limit_hit {
|
||||
tracing::warn!(
|
||||
path = %path_head,
|
||||
max_depth = limits.max_depth,
|
||||
"list_contents: max_depth limit hit, some directories were not traversed"
|
||||
);
|
||||
}
|
||||
if dirs_limit_hit {
|
||||
tracing::warn!(
|
||||
path = %path_head,
|
||||
max_dirs_visited = limits.max_dirs_visited,
|
||||
dirs_visited = dirs_visited,
|
||||
"list_contents: max_dirs_visited limit hit, traversal stopped early"
|
||||
);
|
||||
}
|
||||
|
||||
tracing::debug!(
|
||||
path = %path_head,
|
||||
max_characters = limits.max_characters,
|
||||
max_depth = limits.max_depth,
|
||||
max_dirs_visited = limits.max_dirs_visited,
|
||||
dirs_visited = dirs_visited,
|
||||
max_depth_reached = max_depth_reached,
|
||||
depth_limit_hit = depth_limit_hit,
|
||||
dirs_limit_hit = dirs_limit_hit,
|
||||
output_len = output.len(),
|
||||
walk_ms = t_walk.as_millis() as u64,
|
||||
elapsed_ms = t_total.elapsed().as_millis() as u64,
|
||||
"list_contents complete"
|
||||
);
|
||||
Ok(output)
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
use kigi_paths::ToAbsPath;
|
||||
use std::io;
|
||||
use std::path::Path;
|
||||
use std::sync::Arc;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum FsError {
|
||||
#[error(transparent)]
|
||||
Io(#[from] io::Error),
|
||||
#[error("{0}")]
|
||||
Other(String),
|
||||
}
|
||||
|
||||
// TODO: handle atomic write
|
||||
#[async_trait::async_trait]
|
||||
pub trait AsyncFileSystem: Send + Sync {
|
||||
/// Get the root directory for this filesystem.
|
||||
///
|
||||
/// This is used to resolve relative paths via `ToAbsPath::to_abs_path(fs.root())`.
|
||||
fn root(&self) -> &Path;
|
||||
|
||||
async fn exists(&self, path: &Path) -> Result<bool, FsError>;
|
||||
|
||||
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, FsError>;
|
||||
|
||||
/// Read a file if it exists, returning `Ok(None)` when the file is not found.
|
||||
///
|
||||
/// The default implementation calls `exists()` then `read_file()` (two operations).
|
||||
/// Backends should override this to collapse both into a single operation —
|
||||
/// e.g. one ACP RPC or one syscall — to avoid a redundant round trip.
|
||||
async fn try_read_file(&self, path: &Path) -> Result<Option<Vec<u8>>, FsError> {
|
||||
if self.exists(path).await? {
|
||||
Ok(Some(self.read_file(path).await?))
|
||||
} else {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), FsError>;
|
||||
|
||||
/// Delete a file (for rewind functionality)
|
||||
async fn delete_file(&self, path: &Path) -> Result<(), FsError>;
|
||||
}
|
||||
|
||||
pub fn bytes_to_string(file_bytes: Vec<u8>) -> Result<String, FsError> {
|
||||
String::from_utf8(file_bytes).map_err(|e| FsError::Other(e.to_string()))
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// AsyncFsWrapper - Generic wrapper that accepts any path type
|
||||
// ============================================================================
|
||||
|
||||
/// A wrapper around `AsyncFileSystem` that accepts any path type implementing `ToAbsPath`.
|
||||
///
|
||||
/// This allows callers to pass `AbsPathBuf`, `RelPathBuf`, `&Path`, or `&PathBuf` directly,
|
||||
/// and the wrapper automatically resolves them to absolute paths using the filesystem's root.
|
||||
///
|
||||
/// # Example
|
||||
/// ```ignore
|
||||
/// let wrapper = AsyncFsWrapper::new(fs);
|
||||
///
|
||||
/// // All of these work:
|
||||
/// wrapper.read_to_string(&abs_path).await?;
|
||||
/// wrapper.read_to_string(&rel_path).await?;
|
||||
/// wrapper.read_to_string(Path::new("relative/path")).await?;
|
||||
/// ```
|
||||
#[derive(Clone)]
|
||||
pub struct AsyncFsWrapper {
|
||||
inner: Arc<dyn AsyncFileSystem>,
|
||||
}
|
||||
|
||||
impl AsyncFsWrapper {
|
||||
pub fn new(fs: Arc<dyn AsyncFileSystem>) -> Self {
|
||||
Self { inner: fs }
|
||||
}
|
||||
|
||||
/// Get a reference to the inner `Arc<dyn AsyncFileSystem>`.
|
||||
///
|
||||
/// This is useful when you need raw access to the underlying filesystem
|
||||
/// without the path conversion layer.
|
||||
pub fn inner(&self) -> &Arc<dyn AsyncFileSystem> {
|
||||
&self.inner
|
||||
}
|
||||
|
||||
/// Get the root directory for this filesystem.
|
||||
pub fn root(&self) -> &Path {
|
||||
self.inner.root()
|
||||
}
|
||||
|
||||
/// Check if a file exists.
|
||||
pub async fn exists<P: ToAbsPath>(&self, path: P) -> Result<bool, FsError> {
|
||||
self.inner.exists(&path.to_abs_path(self.root())).await
|
||||
}
|
||||
|
||||
/// Read a file as bytes.
|
||||
pub async fn read_file<P: ToAbsPath>(&self, path: P) -> Result<Vec<u8>, FsError> {
|
||||
self.inner.read_file(&path.to_abs_path(self.root())).await
|
||||
}
|
||||
|
||||
/// Read a file as a UTF-8 string.
|
||||
pub async fn read_to_string<P: ToAbsPath>(&self, path: P) -> Result<String, FsError> {
|
||||
let bytes = self.inner.read_file(&path.to_abs_path(self.root())).await?;
|
||||
bytes_to_string(bytes)
|
||||
}
|
||||
|
||||
/// Read a file as bytes if it exists, returning `Ok(None)` when not found.
|
||||
///
|
||||
/// Uses a single backend operation instead of separate `exists()` + `read_file()`.
|
||||
pub async fn try_read_file<P: ToAbsPath>(&self, path: P) -> Result<Option<Vec<u8>>, FsError> {
|
||||
self.inner
|
||||
.try_read_file(&path.to_abs_path(self.root()))
|
||||
.await
|
||||
}
|
||||
|
||||
/// Read a file as a UTF-8 string if it exists, returning `Ok(None)` when not found.
|
||||
///
|
||||
/// Uses a single backend operation instead of separate `exists()` + `read_to_string()`.
|
||||
pub async fn try_read_to_string<P: ToAbsPath>(
|
||||
&self,
|
||||
path: P,
|
||||
) -> Result<Option<String>, FsError> {
|
||||
match self
|
||||
.inner
|
||||
.try_read_file(&path.to_abs_path(self.root()))
|
||||
.await?
|
||||
{
|
||||
Some(bytes) => Ok(Some(bytes_to_string(bytes)?)),
|
||||
None => Ok(None),
|
||||
}
|
||||
}
|
||||
|
||||
/// Write data to a file.
|
||||
pub async fn write_file<P: ToAbsPath>(&self, path: P, data: &[u8]) -> Result<(), FsError> {
|
||||
self.inner
|
||||
.write_file(&path.to_abs_path(self.root()), data)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Delete a file.
|
||||
pub async fn delete_file<P: ToAbsPath>(&self, path: P) -> Result<(), FsError> {
|
||||
self.inner.delete_file(&path.to_abs_path(self.root())).await
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,479 @@
|
||||
use std::{
|
||||
path::{Path, PathBuf},
|
||||
sync::{
|
||||
Arc, Mutex,
|
||||
atomic::{AtomicBool, Ordering},
|
||||
mpsc::{RecvError, RecvTimeoutError, SyncSender, sync_channel},
|
||||
},
|
||||
thread::{self, JoinHandle},
|
||||
time::Duration,
|
||||
};
|
||||
|
||||
use ignore::{DirEntry, WalkBuilder, WalkState, overrides::OverrideBuilder};
|
||||
use nucleo::{
|
||||
Match, Matcher, Nucleo, Snapshot, Utf32String,
|
||||
pattern::{CaseMatching, MultiPattern, Normalization, Pattern},
|
||||
};
|
||||
|
||||
const NUM_NUCLEO_THREADS: usize = 2;
|
||||
const NUM_IGNORE_THREADS: usize = 8;
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct FuzzyMatchResult {
|
||||
// Path of the matched entry.
|
||||
pub path: Utf32String,
|
||||
/// Matcher score, higher is better.
|
||||
pub score: u32,
|
||||
/// Matched indices of characters.
|
||||
pub indices: Vec<u32>,
|
||||
/// Is it a directory.
|
||||
pub is_dir: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, Default)]
|
||||
pub struct FuzzyMatcherStatus {
|
||||
pub changed: bool,
|
||||
pub done: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
struct MatchEntry {
|
||||
pub is_dir: bool,
|
||||
}
|
||||
|
||||
/// A very fast fuzzy matcher that does ignore-walking. Both happen in background threads.
|
||||
pub struct FuzzyFileMatcher {
|
||||
root: PathBuf,
|
||||
query: String,
|
||||
nucleo: Nucleo<MatchEntry>,
|
||||
matcher: Matcher,
|
||||
walk_handle: Option<JoinHandle<()>>,
|
||||
cancel: Arc<AtomicBool>,
|
||||
top_entries: Vec<FuzzyMatchResult>,
|
||||
dirs: bool,
|
||||
}
|
||||
|
||||
impl FuzzyFileMatcher {
|
||||
/// Create a new matcher with default config focused on matching paths.
|
||||
pub fn new(root: &Path) -> Self {
|
||||
let matcher_config = nucleo::Config::DEFAULT.match_paths();
|
||||
// matcher_config.prefer_prefix = true; // yes or no? nucleo docs lean towards no
|
||||
|
||||
let mut nucleo = Nucleo::new(
|
||||
matcher_config.clone(),
|
||||
Arc::new(move || ()),
|
||||
Some(NUM_NUCLEO_THREADS),
|
||||
1,
|
||||
);
|
||||
nucleo.pattern = MultiPattern::new(1);
|
||||
|
||||
Self {
|
||||
root: root.to_owned(),
|
||||
nucleo,
|
||||
matcher: Matcher::new(matcher_config),
|
||||
walk_handle: None,
|
||||
cancel: Arc::new(AtomicBool::new(false)),
|
||||
query: String::new(),
|
||||
top_entries: Vec::new(),
|
||||
dirs: false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn query(&self) -> &str {
|
||||
&self.query
|
||||
}
|
||||
|
||||
/// Start a new walk and restart nucleo matcher.
|
||||
pub fn restart_walk_custom(
|
||||
&mut self,
|
||||
make_walker: impl FnOnce(&mut WalkBuilder) -> &mut WalkBuilder,
|
||||
) {
|
||||
// first, wait for previous walker to finish if it's up
|
||||
self.cancel.store(true, Ordering::Relaxed);
|
||||
if let Some(walk_handle) = self.walk_handle.take() {
|
||||
walk_handle.join().unwrap();
|
||||
}
|
||||
|
||||
// disconnect all injectors and clear snapshots and streams
|
||||
self.nucleo.restart(true);
|
||||
|
||||
// we're back in business
|
||||
self.cancel.store(false, Ordering::Relaxed);
|
||||
|
||||
// build the walker(s)
|
||||
let walker_builder = make_walker(
|
||||
WalkBuilder::new(&self.root)
|
||||
.threads(NUM_IGNORE_THREADS)
|
||||
.follow_links(false)
|
||||
.git_ignore(true)
|
||||
.git_global(true)
|
||||
.git_exclude(true)
|
||||
.ignore(true)
|
||||
.hidden(true)
|
||||
.require_git(false)
|
||||
.overrides(
|
||||
OverrideBuilder::new(&self.root)
|
||||
.add("!.git")
|
||||
.unwrap()
|
||||
.build()
|
||||
.unwrap(),
|
||||
),
|
||||
)
|
||||
.clone();
|
||||
|
||||
fn check_entry<'a>(entry: &'a DirEntry, root: &Path) -> Option<(&'a str, bool)> {
|
||||
let path = entry.path();
|
||||
if path != root
|
||||
&& let Some(file_type) = entry.file_type()
|
||||
&& (file_type.is_file() || file_type.is_dir())
|
||||
&& let Ok(path) = path.strip_prefix(root)
|
||||
&& let Some(path) = path.as_os_str().to_str()
|
||||
&& !path.is_empty()
|
||||
{
|
||||
Some((path, file_type.is_dir()))
|
||||
} else {
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
// we'll just do it in a blocking way here assuming it's super fast anyway
|
||||
let top_walker = walker_builder
|
||||
.clone()
|
||||
.max_depth(Some(1))
|
||||
.sort_by_file_name(|a, b| a.cmp(b))
|
||||
.build();
|
||||
let top_entries = top_walker
|
||||
.into_iter()
|
||||
.filter_map(|entry| {
|
||||
let entry = entry.ok()?;
|
||||
let (path, is_dir) = check_entry(&entry, &self.root)?;
|
||||
Some(FuzzyMatchResult {
|
||||
path: path.into(),
|
||||
score: 0,
|
||||
indices: Vec::new(),
|
||||
is_dir,
|
||||
})
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let injector = self.nucleo.injector();
|
||||
let root = self.root.clone();
|
||||
let cancel = self.cancel.clone();
|
||||
|
||||
// link walker threads with injectors and start it up
|
||||
let walker = walker_builder.build_parallel();
|
||||
let walk_handle = thread::spawn(move || {
|
||||
walker.run(|| {
|
||||
let injector = injector.clone();
|
||||
let root = root.clone();
|
||||
let cancel = cancel.clone();
|
||||
Box::new(move |entry| {
|
||||
if cancel.load(Ordering::Relaxed) {
|
||||
return WalkState::Quit;
|
||||
} else if let Ok(entry) = entry
|
||||
&& let Some((path, is_dir)) = check_entry(&entry, &root)
|
||||
{
|
||||
injector.push(MatchEntry { is_dir }, |_entry, columns| {
|
||||
columns[0] = path.into();
|
||||
});
|
||||
}
|
||||
WalkState::Continue
|
||||
})
|
||||
});
|
||||
});
|
||||
self.walk_handle = Some(walk_handle);
|
||||
self.top_entries = top_entries;
|
||||
|
||||
self.nucleo.tick(0);
|
||||
}
|
||||
|
||||
/// Restart the walk with default walker parameters.
|
||||
pub fn restart_walk(&mut self) {
|
||||
self.restart_walk_custom(|w| w);
|
||||
}
|
||||
|
||||
/// Set the query to a given string and trigger reparse.
|
||||
///
|
||||
/// It will be faster if the current query is a strict prefix of the new query.
|
||||
pub fn set_query(&mut self, mut query: &str, dirs: bool) {
|
||||
self.dirs = dirs;
|
||||
if dirs && query.ends_with('/') {
|
||||
query = &query[..query.len() - 1];
|
||||
}
|
||||
if query == self.query {
|
||||
return;
|
||||
}
|
||||
// see this re: backslash etc: https://github.com/helix-editor/nucleo/pull/87
|
||||
let append = query.as_bytes().starts_with(self.query.as_bytes())
|
||||
&& !query.ends_with('\\')
|
||||
&& !query
|
||||
.as_bytes()
|
||||
.last()
|
||||
.is_some_and(|ch| ch.is_ascii_whitespace());
|
||||
self.nucleo
|
||||
.pattern
|
||||
.reparse(0, query, CaseMatching::Smart, Normalization::Smart, append);
|
||||
self.nucleo.tick(0);
|
||||
self.query = query.to_owned();
|
||||
}
|
||||
|
||||
/// Sends a tick to nucleo matcher. Can be safely called at any frequency.
|
||||
pub fn tick(&mut self, tick_timeout_ms: u64) -> FuzzyMatcherStatus {
|
||||
if self.query.is_empty() {
|
||||
return FuzzyMatcherStatus {
|
||||
done: true,
|
||||
changed: false,
|
||||
};
|
||||
}
|
||||
let status = self.nucleo.tick(tick_timeout_ms);
|
||||
let done = self.nucleo.active_injectors() == 0 && !status.running;
|
||||
FuzzyMatcherStatus {
|
||||
done,
|
||||
changed: status.changed,
|
||||
}
|
||||
}
|
||||
|
||||
/// Total number of currently matched items in the snapshot.
|
||||
pub fn num_items(&self) -> usize {
|
||||
if self.query.is_empty() {
|
||||
self.top_entries.len()
|
||||
} else {
|
||||
self.nucleo.snapshot().item_count() as _
|
||||
}
|
||||
}
|
||||
|
||||
/// Get top `k` items from the snapshot and sort them by score, path length and path.
|
||||
pub fn get_top_k(&mut self, k: usize) -> Vec<FuzzyMatchResult> {
|
||||
// note: &mut only because we access self.matcher which has internal allocations
|
||||
|
||||
// rust is a bit dumb at times, we'll need this for sorting without cloning
|
||||
fn sort_by_key_hrtb<T, F, K, Q>(slice: &mut [T], f: F)
|
||||
where
|
||||
F: for<'a> Fn(&'a T) -> (Q, &'a K),
|
||||
K: Ord,
|
||||
Q: Ord,
|
||||
{
|
||||
slice.sort_by(|a, b| f(a).cmp(&f(b)))
|
||||
}
|
||||
|
||||
// special case: if query is empty, return top items only
|
||||
if self.query.is_empty() {
|
||||
return self
|
||||
.top_entries
|
||||
.iter()
|
||||
// dirs_only=true means only directories; dirs_only=false means both files and directories
|
||||
.filter(|e| !self.dirs || e.is_dir)
|
||||
.take(k)
|
||||
.cloned()
|
||||
.collect(); // should be already sorted
|
||||
}
|
||||
|
||||
// https://github.com/helix-editor/helix/blob/d79cce4e4bfc24dd204f1b294c899ed73f7e9453/helix-term/src/ui/completion.rs#L369
|
||||
// suggested min score = 7 * len + 14
|
||||
let len = self.query.chars().count() as u32;
|
||||
let min_score = 7 + len * 14;
|
||||
|
||||
let mut items = Vec::with_capacity(k);
|
||||
let pattern = self.nucleo.pattern.column_pattern(0);
|
||||
let snapshot = self.nucleo.snapshot();
|
||||
let mut iter = snapshot.matches().iter().peekable();
|
||||
|
||||
while items.len() < k
|
||||
&& let Some(m) = iter.next()
|
||||
// for empty queries, return everything; otherwise, apply heuristic min-score limit
|
||||
&& (self.query.is_empty() || m.score >= min_score)
|
||||
{
|
||||
fn extract_match(
|
||||
m: &Match,
|
||||
snapshot: &Snapshot<MatchEntry>,
|
||||
pattern: &Pattern,
|
||||
matcher: &mut Matcher,
|
||||
dirs_only: bool,
|
||||
) -> Option<FuzzyMatchResult> {
|
||||
let item = unsafe { snapshot.get_item_unchecked(m.idx) };
|
||||
// dirs_only=true means only directories; dirs_only=false means both files and directories
|
||||
if dirs_only && !item.data.is_dir {
|
||||
return None;
|
||||
}
|
||||
let path = item.matcher_columns[0].clone();
|
||||
let mut indices = Vec::new();
|
||||
if !pattern.atoms.is_empty() {
|
||||
pattern.indices(path.slice(..), matcher, &mut indices);
|
||||
}
|
||||
Some(FuzzyMatchResult {
|
||||
path,
|
||||
score: m.score,
|
||||
indices,
|
||||
is_dir: item.data.is_dir,
|
||||
})
|
||||
}
|
||||
|
||||
if !pattern.atoms.is_empty() {
|
||||
let start = items.len();
|
||||
items.extend(extract_match(
|
||||
m,
|
||||
snapshot,
|
||||
pattern,
|
||||
&mut self.matcher,
|
||||
self.dirs,
|
||||
));
|
||||
while iter.peek().is_some_and(|p| p.score == m.score) {
|
||||
let m = iter.next().unwrap();
|
||||
items.extend(extract_match(
|
||||
m,
|
||||
snapshot,
|
||||
pattern,
|
||||
&mut self.matcher,
|
||||
self.dirs,
|
||||
));
|
||||
}
|
||||
sort_by_key_hrtb(&mut items[start..], |m| (m.path.len(), &m.path));
|
||||
} else {
|
||||
items.extend(extract_match(
|
||||
m,
|
||||
snapshot,
|
||||
pattern,
|
||||
&mut self.matcher,
|
||||
self.dirs,
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
if items.len() > k {
|
||||
items.truncate(k);
|
||||
}
|
||||
|
||||
if pattern.atoms.is_empty() {
|
||||
sort_by_key_hrtb(&mut items, |m| (true, &m.path));
|
||||
}
|
||||
|
||||
items
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for FuzzyFileMatcher {
|
||||
fn drop(&mut self) {
|
||||
// note: walker threads *may* get detached for a little while but hopefully not for too long
|
||||
self.cancel.store(true, Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct FuzzyMatcherDaemonResults {
|
||||
pub topk: Arc<[FuzzyMatchResult]>,
|
||||
pub num_items: usize,
|
||||
pub status: FuzzyMatcherStatus,
|
||||
pub generation: usize,
|
||||
}
|
||||
|
||||
impl AsRef<[FuzzyMatchResult]> for FuzzyMatcherDaemonResults {
|
||||
fn as_ref(&self) -> &[FuzzyMatchResult] {
|
||||
self.topk.as_ref()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
enum FuzzyMatcherDaemonMessage {
|
||||
RestartWalk { hidden: bool },
|
||||
SetQuery { query: String, dirs: bool },
|
||||
Stop,
|
||||
}
|
||||
|
||||
pub struct FuzzyFileMatcherDaemon {
|
||||
results: Arc<Mutex<FuzzyMatcherDaemonResults>>,
|
||||
tx: SyncSender<FuzzyMatcherDaemonMessage>,
|
||||
_handle: JoinHandle<()>,
|
||||
}
|
||||
|
||||
impl FuzzyFileMatcherDaemon {
|
||||
pub fn new(mut matcher: FuzzyFileMatcher, topk: usize) -> Self {
|
||||
let results = Arc::new(Mutex::new(FuzzyMatcherDaemonResults::default()));
|
||||
let (tx, rx) = sync_channel(1024);
|
||||
|
||||
let res = results.clone();
|
||||
let handle = thread::spawn(move || {
|
||||
let results = res;
|
||||
let mut done = false;
|
||||
let mut generation = 0;
|
||||
loop {
|
||||
let msg = if !done {
|
||||
rx.recv_timeout(Duration::from_micros(250))
|
||||
} else {
|
||||
rx.recv().map_err(|e| match e {
|
||||
RecvError => RecvTimeoutError::Disconnected,
|
||||
})
|
||||
};
|
||||
match msg {
|
||||
Ok(FuzzyMatcherDaemonMessage::RestartWalk { hidden }) => {
|
||||
if !hidden {
|
||||
tracing::trace!("restarting normal walk");
|
||||
matcher.restart_walk();
|
||||
} else {
|
||||
tracing::trace!("restarting hidden walk");
|
||||
matcher.restart_walk_custom(|w| {
|
||||
w.hidden(false).ignore(false).git_ignore(false)
|
||||
});
|
||||
}
|
||||
generation += 1;
|
||||
*results.lock().unwrap() = FuzzyMatcherDaemonResults::default();
|
||||
done = false;
|
||||
}
|
||||
Ok(FuzzyMatcherDaemonMessage::SetQuery { query, dirs }) => {
|
||||
matcher.set_query(&query, dirs);
|
||||
generation += 1;
|
||||
done = false;
|
||||
}
|
||||
Ok(FuzzyMatcherDaemonMessage::Stop) | Err(RecvTimeoutError::Disconnected) => {
|
||||
break;
|
||||
}
|
||||
Err(RecvTimeoutError::Timeout) => {
|
||||
if !done {
|
||||
let status = matcher.tick(10);
|
||||
done = status.done;
|
||||
let num_items = matcher.num_items();
|
||||
let topk: Arc<[_]> = matcher.get_top_k(topk).into();
|
||||
*results.lock().unwrap() = FuzzyMatcherDaemonResults {
|
||||
topk,
|
||||
num_items,
|
||||
status,
|
||||
generation,
|
||||
};
|
||||
generation += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
Self {
|
||||
results,
|
||||
tx,
|
||||
_handle: handle,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn get(&self) -> FuzzyMatcherDaemonResults {
|
||||
self.results.lock().unwrap().clone()
|
||||
}
|
||||
|
||||
pub fn set_query(&self, query: impl AsRef<str>, dirs: bool) {
|
||||
let query = query.as_ref().to_owned();
|
||||
_ = self
|
||||
.tx
|
||||
.send(FuzzyMatcherDaemonMessage::SetQuery { query, dirs })
|
||||
.ok();
|
||||
}
|
||||
|
||||
pub fn restart_walk(&self, hidden: bool) {
|
||||
_ = self
|
||||
.tx
|
||||
.send(FuzzyMatcherDaemonMessage::RestartWalk { hidden })
|
||||
.ok();
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for FuzzyFileMatcherDaemon {
|
||||
fn drop(&mut self) {
|
||||
_ = self.tx.send(FuzzyMatcherDaemonMessage::Stop).ok();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,290 @@
|
||||
//! Generates a compact git status for the system prompt.
|
||||
//!
|
||||
//! Uses the git CLI for performance — libgit2's status is 5-10x slower than
|
||||
//! the native git binary on large repos due to inefficient index refresh.
|
||||
//! Output is prioritized by change type and limited to ~1k characters.
|
||||
|
||||
use crate::file_system::FsError;
|
||||
use std::fmt::Write;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
/// Gets a compact git status for the system prompt using the git CLI.
|
||||
///
|
||||
/// Output includes:
|
||||
/// 1. Branch name
|
||||
/// 2. Upstream ahead/behind status
|
||||
/// 3. Staged files (if any)
|
||||
///
|
||||
/// Total output is capped at ~1k characters.
|
||||
pub async fn git_status(working_directory: impl Into<PathBuf>) -> Result<String, FsError> {
|
||||
let working_directory = working_directory.into();
|
||||
|
||||
tokio::task::spawn_blocking(move || git_status_impl(&working_directory))
|
||||
.await
|
||||
.map_err(|e| FsError::Other(format!("git status task failed: {}", e)))?
|
||||
}
|
||||
|
||||
/// Matches Node's default `execFile` `maxBuffer` (1 MiB). This cap is
|
||||
/// load-bearing: `git status` output at or above it makes the spawn throw, so
|
||||
/// the repo is dropped from `<git_status>` entirely (never truncated).
|
||||
/// Oversized output is treated as an error -- the caller maps `Err` to a
|
||||
/// dropped section.
|
||||
const GIT_STATUS_BUFFER_LIMIT: usize = 1024 * 1024;
|
||||
|
||||
/// Whether `git status` stdout is large enough that the repo is dropped
|
||||
/// (`>= 1 MiB`). Extracted as a pure predicate so it is unit-testable
|
||||
/// without spawning git.
|
||||
fn git_status_exceeds_buffer(stdout_len: usize) -> bool {
|
||||
stdout_len >= GIT_STATUS_BUFFER_LIMIT
|
||||
}
|
||||
|
||||
/// Collapse runs of 2+ spaces to a single space.
|
||||
///
|
||||
/// The `<git_status>` body collapses consecutive spaces, so the
|
||||
/// porcelain two-column status renders with a single separator: `A staged.txt`
|
||||
/// (index-added, clean worktree) becomes `A staged.txt`, `M mod.txt` becomes
|
||||
/// `M mod.txt`, `R old -> new` becomes `R old -> new`. A single leading space
|
||||
/// (e.g. ` M file`, worktree-modified) and the rename ` -> ` separator are
|
||||
/// preserved because they are runs of length one.
|
||||
/// Newlines are never touched.
|
||||
fn collapse_status_spaces(s: &str) -> String {
|
||||
let mut out = String::with_capacity(s.len());
|
||||
let mut prev_space = false;
|
||||
for ch in s.chars() {
|
||||
if ch == ' ' {
|
||||
if prev_space {
|
||||
continue;
|
||||
}
|
||||
prev_space = true;
|
||||
} else {
|
||||
prev_space = false;
|
||||
}
|
||||
out.push(ch);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Short git status for the templated user message.
|
||||
///
|
||||
/// Runs `git status --short --branch` and returns its output with consecutive
|
||||
/// spaces collapsed via [`collapse_status_spaces`]: a leading `## <branch>`
|
||||
/// line followed by the file change list (or just `## <branch>` on a clean
|
||||
/// tree). This matches the body embedded in the `<git_status>` block
|
||||
/// byte-for-byte.
|
||||
pub async fn git_status_short(working_directory: impl Into<PathBuf>) -> Result<String, FsError> {
|
||||
let working_directory = working_directory.into();
|
||||
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let output = kigi_tty_utils::git_command()
|
||||
.args(["status", "--short", "--branch"])
|
||||
.current_dir(&working_directory)
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.output()
|
||||
.map_err(|e| FsError::Other(format!("git status --short --branch failed: {}", e)))?;
|
||||
|
||||
if !output.status.success() {
|
||||
return Err(FsError::Other(format!(
|
||||
"git status --short --branch exited with code {:?}",
|
||||
output.status.code()
|
||||
)));
|
||||
}
|
||||
|
||||
// Output >= 1 MiB is dropped entirely, not truncated. Render-time
|
||||
// truncation handles the < 1 MiB case.
|
||||
if git_status_exceeds_buffer(output.stdout.len()) {
|
||||
return Err(FsError::Other(
|
||||
"git status --short --branch output exceeded 1 MiB buffer".to_string(),
|
||||
));
|
||||
}
|
||||
|
||||
// Consecutive spaces in the status body are collapsed so staged
|
||||
// entries (`A file` -> `A file`) match the wire format.
|
||||
Ok(collapse_status_spaces(&String::from_utf8_lossy(
|
||||
&output.stdout,
|
||||
)))
|
||||
})
|
||||
.await
|
||||
.map_err(|e| FsError::Other(format!("git status --short --branch task failed: {}", e)))?
|
||||
}
|
||||
|
||||
fn git_status_impl(working_directory: &Path) -> Result<String, FsError> {
|
||||
let _timer = /* instrumentation_timer */ () ; // dev macro; noop stub ("git_status.impl")
|
||||
let max_status_chars = 1000;
|
||||
let mut output = String::with_capacity(max_status_chars);
|
||||
|
||||
// Get branch name
|
||||
let branch_name = {
|
||||
let _timer = /* instrumentation_timer */ () ; // dev macro; noop stub ("git_status.branch_info")
|
||||
run_git(working_directory, &["rev-parse", "--abbrev-ref", "HEAD"])
|
||||
};
|
||||
|
||||
match &branch_name {
|
||||
Some(branch) if branch == "HEAD" => {
|
||||
// Detached HEAD — get short commit hash
|
||||
if let Some(hash) = run_git(working_directory, &["rev-parse", "--short", "HEAD"]) {
|
||||
let _ = writeln!(output, "HEAD detached at {}", hash);
|
||||
}
|
||||
}
|
||||
Some(branch) => {
|
||||
let _ = writeln!(output, "On branch {}", branch);
|
||||
}
|
||||
None => {
|
||||
return Err(FsError::Other("not a git repository".to_string()));
|
||||
}
|
||||
}
|
||||
|
||||
// Get upstream ahead/behind
|
||||
{
|
||||
let _timer = (); // instrumentation_timer noop stub
|
||||
if let Some(upstream_name) = run_git(
|
||||
working_directory,
|
||||
&["rev-parse", "--abbrev-ref", "@{upstream}"],
|
||||
) && let Some(counts) = run_git(
|
||||
working_directory,
|
||||
&["rev-list", "--count", "--left-right", "@{upstream}...HEAD"],
|
||||
) {
|
||||
let parts: Vec<&str> = counts.split_whitespace().collect();
|
||||
if let (Some(behind_str), Some(ahead_str)) = (parts.first(), parts.get(1)) {
|
||||
let behind: usize = behind_str.parse().unwrap_or(0);
|
||||
let ahead: usize = ahead_str.parse().unwrap_or(0);
|
||||
|
||||
let status_msg = match (ahead, behind) {
|
||||
(0, 0) => {
|
||||
format!("Your branch is up to date with '{}'.", upstream_name)
|
||||
}
|
||||
(a, 0) => format!(
|
||||
"Your branch is ahead of '{}' by {} commit{}.",
|
||||
upstream_name,
|
||||
a,
|
||||
if a == 1 { "" } else { "s" }
|
||||
),
|
||||
(0, b) => format!(
|
||||
"Your branch is behind '{}' by {} commit{}.",
|
||||
upstream_name,
|
||||
b,
|
||||
if b == 1 { "" } else { "s" }
|
||||
),
|
||||
(a, b) => format!(
|
||||
"Your branch and '{}' have diverged ({} ahead, {} behind).",
|
||||
upstream_name, a, b
|
||||
),
|
||||
};
|
||||
let _ = writeln!(output, "{}", status_msg);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Get staged changes (index vs HEAD) — fast, no workdir scan
|
||||
let staged_output = {
|
||||
let _timer = /* instrumentation_timer */ () ; // dev macro; noop stub ("git_status.staged")
|
||||
run_git(
|
||||
working_directory,
|
||||
&["diff", "--cached", "--name-status", "HEAD"],
|
||||
)
|
||||
};
|
||||
|
||||
let mut staged: Vec<String> = Vec::new();
|
||||
if let Some(ref diff_output) = staged_output {
|
||||
for line in diff_output.lines() {
|
||||
let mut parts = line.splitn(2, '\t');
|
||||
let status_char = parts.next().unwrap_or("");
|
||||
let path = parts.next().unwrap_or("");
|
||||
if path.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let formatted = match status_char.chars().next() {
|
||||
Some('A') => format!("\tnew file: {}", path),
|
||||
Some('M') => format!("\tmodified: {}", path),
|
||||
Some('D') => format!("\tdeleted: {}", path),
|
||||
Some('R') => format!("\trenamed: {}", path),
|
||||
_ => format!("\t{}: {}", status_char, path),
|
||||
};
|
||||
staged.push(formatted);
|
||||
}
|
||||
}
|
||||
|
||||
// Check if clean
|
||||
if staged.is_empty() {
|
||||
let _ = writeln!(output, "\nnothing to commit, working tree clean");
|
||||
return Ok(output);
|
||||
}
|
||||
|
||||
// Reserve space for truncation message
|
||||
let reserve_for_truncation = 50;
|
||||
let char_budget = max_status_chars - reserve_for_truncation;
|
||||
|
||||
// Write staged files
|
||||
if !staged.is_empty() && output.len() < char_budget {
|
||||
let _ = writeln!(output, "\nChanges to be committed:");
|
||||
for (shown, line) in staged.iter().enumerate() {
|
||||
if output.len() + line.len() + 1 > char_budget {
|
||||
let remaining = staged.len() - shown;
|
||||
if remaining > 0 {
|
||||
let _ = writeln!(output, "\t... and {} more staged", remaining);
|
||||
}
|
||||
break;
|
||||
}
|
||||
let _ = writeln!(output, "{}", line);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(output)
|
||||
}
|
||||
|
||||
/// Run a read-only git command and return its stdout, trimmed.
|
||||
/// Returns None on failure.
|
||||
///
|
||||
/// Uses `--no-optional-locks` to avoid creating `index.lock` for stat-cache
|
||||
/// refreshes. This function is called from background tasks (system prompt
|
||||
/// generation) and must never contend with foreground git operations.
|
||||
fn run_git(cwd: &Path, args: &[&str]) -> Option<String> {
|
||||
let output = kigi_tty_utils::git_command()
|
||||
.args(args)
|
||||
.current_dir(cwd)
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.output()
|
||||
.ok()?;
|
||||
|
||||
if !output.status.success() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let stdout = String::from_utf8_lossy(&output.stdout).trim().to_string();
|
||||
if stdout.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(stdout)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn git_status_buffer_cap_matches_spec() {
|
||||
assert!(!git_status_exceeds_buffer(0));
|
||||
assert!(!git_status_exceeds_buffer(GIT_STATUS_BUFFER_LIMIT - 1));
|
||||
// At or above 1 MiB -> dropped.
|
||||
assert!(git_status_exceeds_buffer(GIT_STATUS_BUFFER_LIMIT));
|
||||
assert!(git_status_exceeds_buffer(GIT_STATUS_BUFFER_LIMIT + 1));
|
||||
}
|
||||
|
||||
/// Staged entries collapse the porcelain double space, while leading
|
||||
/// single spaces and ` -> ` are preserved.
|
||||
#[test]
|
||||
fn collapse_status_spaces_matches_spec() {
|
||||
let raw = "## main...origin/main\n M committed.txt\nA staged.txt\nM mod.txt\nR old.txt -> new.txt\n?? untracked.txt\n";
|
||||
let want = "## main...origin/main\n M committed.txt\nA staged.txt\nM mod.txt\nR old.txt -> new.txt\n?? untracked.txt\n";
|
||||
assert_eq!(collapse_status_spaces(raw), want);
|
||||
}
|
||||
|
||||
/// Newlines are never collapsed (blank lines preserved).
|
||||
#[test]
|
||||
fn collapse_status_spaces_preserves_newlines() {
|
||||
assert_eq!(collapse_status_spaces("a\n\n\nb"), "a\n\n\nb");
|
||||
assert_eq!(collapse_status_spaces(""), "");
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,85 @@
|
||||
//! Compact jj status for the system prompt.
|
||||
|
||||
use std::fmt::Write;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Command;
|
||||
|
||||
use crate::file_system::FsError;
|
||||
|
||||
/// jj template: change ID, commit ID, description, bookmarks.
|
||||
const JJ_LOG_TEMPLATE: &str = r#"separate("\n",
|
||||
"Change: " ++ change_id.shortest(8),
|
||||
"Commit: " ++ commit_id.shortest(8),
|
||||
if(description,
|
||||
"Description: " ++ description.first_line(),
|
||||
"Description: (no description set)"),
|
||||
if(bookmarks,
|
||||
"Bookmarks: " ++ bookmarks.join(", "),
|
||||
"")
|
||||
)"#;
|
||||
|
||||
/// Compact jj status for the system prompt (~1k chars max).
|
||||
pub async fn jj_status(working_directory: impl Into<PathBuf>) -> Result<String, FsError> {
|
||||
let working_directory = working_directory.into();
|
||||
tokio::task::spawn_blocking(move || jj_status_impl(&working_directory))
|
||||
.await
|
||||
.map_err(|e| FsError::Other(format!("jj status task failed: {e}")))?
|
||||
}
|
||||
|
||||
fn jj_status_impl(cwd: &Path) -> Result<String, FsError> {
|
||||
let max_chars = 1000;
|
||||
let mut out = String::with_capacity(max_chars);
|
||||
|
||||
let log = run_jj(
|
||||
cwd,
|
||||
&["log", "--no-graph", "-r", "@", "-T", JJ_LOG_TEMPLATE],
|
||||
)
|
||||
.ok_or_else(|| FsError::Other("not a jujutsu repository".into()))?;
|
||||
|
||||
for line in log.lines().filter(|l| !l.is_empty()) {
|
||||
let _ = writeln!(out, "{line}");
|
||||
}
|
||||
|
||||
match run_jj(cwd, &["st"]) {
|
||||
Some(st) if st.contains("The working copy is clean") || st.is_empty() => {
|
||||
let _ = writeln!(out, "\nWorking copy is clean");
|
||||
}
|
||||
Some(st) => {
|
||||
let _ = writeln!(out);
|
||||
let budget = max_chars - 50;
|
||||
for (i, line) in st.lines().enumerate() {
|
||||
if out.len() + line.len() + 1 > budget {
|
||||
let remaining = st.lines().count() - i;
|
||||
if remaining > 0 {
|
||||
let _ = writeln!(out, "... and {remaining} more lines");
|
||||
}
|
||||
break;
|
||||
}
|
||||
let _ = writeln!(out, "{line}");
|
||||
}
|
||||
}
|
||||
None => {}
|
||||
}
|
||||
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Run a jj command synchronously, returning trimmed stdout or `None` on failure.
|
||||
fn run_jj(cwd: &Path, args: &[&str]) -> Option<String> {
|
||||
let mut cmd = Command::new("jj");
|
||||
cmd.arg("--ignore-working-copy")
|
||||
.args(args)
|
||||
.current_dir(cwd)
|
||||
.stdout(std::process::Stdio::piped())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.stdin(std::process::Stdio::null());
|
||||
kigi_tools::util::detach_std_command(&mut cmd);
|
||||
let output = cmd.output().ok()?;
|
||||
|
||||
if !output.status.success() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let stdout = String::from_utf8_lossy(&output.stdout).trim().to_string();
|
||||
(!stdout.is_empty()).then_some(stdout)
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
use std::path::{Path, PathBuf};
|
||||
use tokio::fs;
|
||||
|
||||
use crate::file_system::{AsyncFileSystem, FsError};
|
||||
|
||||
pub struct LocalFs {
|
||||
root: PathBuf,
|
||||
}
|
||||
|
||||
impl LocalFs {
|
||||
pub fn new(root: PathBuf) -> Self {
|
||||
Self { root }
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl AsyncFileSystem for LocalFs {
|
||||
fn root(&self) -> &Path {
|
||||
&self.root
|
||||
}
|
||||
|
||||
async fn exists(&self, path: &Path) -> Result<bool, FsError> {
|
||||
Ok(fs::try_exists(path).await?)
|
||||
}
|
||||
|
||||
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, FsError> {
|
||||
Ok(fs::read(path).await?)
|
||||
}
|
||||
|
||||
async fn try_read_file(&self, path: &Path) -> Result<Option<Vec<u8>>, FsError> {
|
||||
match fs::read(path).await {
|
||||
Ok(bytes) => Ok(Some(bytes)),
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
|
||||
Err(e) => Err(e.into()),
|
||||
}
|
||||
}
|
||||
|
||||
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), FsError> {
|
||||
if let Some(dir) = path.parent() {
|
||||
fs::create_dir_all(dir).await?;
|
||||
}
|
||||
fs::write(path, data).await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn delete_file(&self, path: &Path) -> Result<(), FsError> {
|
||||
fs::remove_file(path).await?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
use std::collections::HashMap;
|
||||
use std::io;
|
||||
use std::path::{Path, PathBuf};
|
||||
use tokio::sync::RwLock;
|
||||
|
||||
use crate::file_system::{AsyncFileSystem, FsError};
|
||||
|
||||
pub struct MockFs {
|
||||
root: PathBuf,
|
||||
files: RwLock<HashMap<PathBuf, Vec<u8>>>,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl AsyncFileSystem for MockFs {
|
||||
fn root(&self) -> &Path {
|
||||
&self.root
|
||||
}
|
||||
|
||||
async fn exists(&self, path: &Path) -> Result<bool, FsError> {
|
||||
let map = self.files.read().await;
|
||||
Ok(map.contains_key(path))
|
||||
}
|
||||
|
||||
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, FsError> {
|
||||
let map = self.files.read().await;
|
||||
if let Some(bytes) = map.get(path) {
|
||||
Ok(bytes.clone())
|
||||
} else {
|
||||
Err(io::Error::new(io::ErrorKind::NotFound, "File not found").into())
|
||||
}
|
||||
}
|
||||
|
||||
async fn try_read_file(&self, path: &Path) -> Result<Option<Vec<u8>>, FsError> {
|
||||
let map = self.files.read().await;
|
||||
Ok(map.get(path).cloned())
|
||||
}
|
||||
|
||||
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), FsError> {
|
||||
let mut map = self.files.write().await;
|
||||
map.insert(path.to_path_buf(), data.to_vec());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn delete_file(&self, path: &Path) -> Result<(), FsError> {
|
||||
let mut map = self.files.write().await;
|
||||
map.remove(path);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl MockFs {
|
||||
pub fn new(root: PathBuf) -> Self {
|
||||
Self {
|
||||
root,
|
||||
files: RwLock::new(HashMap::new()),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,360 @@
|
||||
mod acp_fs;
|
||||
pub use acp_fs::AcpSessionFs;
|
||||
|
||||
mod ext_fs;
|
||||
pub use ext_fs::{
|
||||
FsDeleteFileReq, FsExistsData, FsExistsReq, FsListData, FsListNode, FsListReq, FsReadFileData,
|
||||
FsReadFileReq, FsWriteFileReq,
|
||||
};
|
||||
|
||||
// Client-facing read-only fs ops (`workspace.client_fs_*`). Not re-exported:
|
||||
// its wire types live in `kigi_workspace_types::rpc::fs` (the `ClientFs*`
|
||||
// types) and would collide with the shell-facing `ext_fs` names above.
|
||||
pub(crate) mod client_fs;
|
||||
|
||||
// Shared filesystem core: paginated listing + binary-safe ranged reads,
|
||||
// used by `client_fs`, `ext_fs`, and the shell-local `session::file_system`.
|
||||
mod walk;
|
||||
pub use walk::{
|
||||
ChunkPayload, ListOptions, ListPage, ListedEntry, MAX_LIST_COLLECT, MAX_READ_BYTES,
|
||||
clamp_read_length, encode_chunk, list_directory_paged, read_range,
|
||||
};
|
||||
// Re-exported so shell-side fs ops can name the shared read encoding.
|
||||
pub use kigi_workspace_types::rpc::fs::FsReadEncoding;
|
||||
|
||||
pub mod adapter;
|
||||
pub use adapter::AcpFsAdapter;
|
||||
|
||||
mod codebase_index;
|
||||
pub use codebase_index::CodebaseIndexManager;
|
||||
|
||||
mod fs;
|
||||
pub use fs::{AsyncFileSystem, AsyncFsWrapper, FsError, bytes_to_string};
|
||||
|
||||
mod local_fs;
|
||||
pub use local_fs::LocalFs;
|
||||
|
||||
mod mock_fs;
|
||||
pub use mock_fs::MockFs;
|
||||
|
||||
mod file_tree;
|
||||
pub use file_tree::{ListContentsLimits, list_contents};
|
||||
|
||||
mod git_status;
|
||||
pub use git_status::{git_status, git_status_short};
|
||||
|
||||
mod jj_status;
|
||||
pub use jj_status::jj_status;
|
||||
|
||||
mod attach_file;
|
||||
pub use attach_file::{FileReference, render_embedded_resource, render_file_reference};
|
||||
|
||||
mod fuzzy;
|
||||
pub use fuzzy::{
|
||||
FuzzyFileMatcher, FuzzyFileMatcherDaemon, FuzzyMatchResult, FuzzyMatcherDaemonResults,
|
||||
FuzzyMatcherStatus,
|
||||
};
|
||||
|
||||
mod index;
|
||||
pub use index::{FileEntry, FileIndex, FileIndexDelta, SegmentId, StringInterner, WalkOptions};
|
||||
|
||||
mod content;
|
||||
pub use content::{
|
||||
ContentMatch, ContentMatchFile, ContentSearchBatch, ContentSearchData, ContentSearchParams,
|
||||
content_search_streaming,
|
||||
};
|
||||
|
||||
use serde::Serialize;
|
||||
use std::{
|
||||
collections::HashMap,
|
||||
path::{Path, PathBuf},
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
use uuid::Uuid;
|
||||
|
||||
// Canonical in kigi-workspace-types; re-exported for existing paths.
|
||||
pub use kigi_workspace_types::rpc::search::{ClientId, ContentSearchRequest, TargetClientId};
|
||||
|
||||
impl From<ContentSearchRequest> for ContentSearchParams {
|
||||
fn from(req: ContentSearchRequest) -> Self {
|
||||
let pattern = if req.is_regex {
|
||||
req.pattern
|
||||
} else if req.whole_word {
|
||||
format!("\\b{}\\b", regex::escape(&req.pattern))
|
||||
} else {
|
||||
req.pattern
|
||||
};
|
||||
|
||||
let literal = !req.is_regex && !req.whole_word;
|
||||
let globs: Vec<String> = req
|
||||
.include_globs
|
||||
.into_iter()
|
||||
.chain(req.exclude_globs.into_iter().map(|g| format!("!{g}")))
|
||||
.collect();
|
||||
|
||||
Self {
|
||||
pattern,
|
||||
case_insensitive: req.case_insensitive,
|
||||
literal,
|
||||
globs,
|
||||
max_files: req.max_files,
|
||||
max_matches: req.max_matches,
|
||||
respect_gitignore: req.respect_gitignore,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const DEFAULT_SEARCH_TIMEOUT_SECS: u64 = 30;
|
||||
const DEFAULT_TOP_K: usize = 1000;
|
||||
|
||||
pub type FuzzySearchId = String;
|
||||
|
||||
impl Serialize for FuzzyMatchResult {
|
||||
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
|
||||
where
|
||||
S: serde::Serializer,
|
||||
{
|
||||
use serde::ser::SerializeStruct;
|
||||
use std::borrow::Cow;
|
||||
|
||||
let path_str = self.path.to_string();
|
||||
let node_type = if self.is_dir { "directory" } else { "file" };
|
||||
let name: Cow<str> = std::path::Path::new(&path_str)
|
||||
.file_name()
|
||||
.map(|s| s.to_string_lossy())
|
||||
.unwrap_or(Cow::Borrowed(&path_str));
|
||||
|
||||
let mut state = serializer.serialize_struct("FuzzyMatchResult", 5)?;
|
||||
state.serialize_field("name", &name)?;
|
||||
state.serialize_field("type", node_type)?;
|
||||
state.serialize_field("path", &path_str)?;
|
||||
state.serialize_field("score", &self.score)?;
|
||||
state.serialize_field("indices", &self.indices)?;
|
||||
state.end()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct FuzzySearchData {
|
||||
pub matches: Vec<FuzzyMatchResult>,
|
||||
pub total: usize,
|
||||
pub done: bool,
|
||||
pub generation: usize,
|
||||
}
|
||||
|
||||
/// Result of one fuzzy-search poll tick (see [`WorkspaceHandle::fuzzy_poll`]).
|
||||
///
|
||||
/// Consumed in-process by the shell's notification driver, so it carries the
|
||||
/// (non-`Deserialize`) match results directly rather than going over RPC.
|
||||
///
|
||||
/// [`WorkspaceHandle::fuzzy_poll`]: crate::handle::WorkspaceHandle::fuzzy_poll
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum FuzzyPollOutcome {
|
||||
/// The query was superseded by a newer change — stop polling.
|
||||
Stale,
|
||||
/// The search no longer exists — stop polling.
|
||||
Closed,
|
||||
/// The search exists but produced no new results this tick — keep polling.
|
||||
Pending,
|
||||
/// New results, with paths already absolutized against the search root.
|
||||
Update(FuzzySearchData),
|
||||
}
|
||||
|
||||
pub struct FuzzySearchContext {
|
||||
pub daemon: FuzzyFileMatcherDaemon,
|
||||
pub created_at: Instant,
|
||||
pub last_activity: Instant,
|
||||
pub hidden: bool,
|
||||
pub min_generation: usize,
|
||||
pub has_query: bool,
|
||||
pub query_version: usize,
|
||||
/// The root path for this search (used to convert relative paths to absolute).
|
||||
pub root: PathBuf,
|
||||
/// Session ID for routing notifications.
|
||||
/// Used by the relay to route notifications to session subscribers.
|
||||
pub session_id: Option<String>,
|
||||
/// Target client ID for routing notifications.
|
||||
/// Extracted from `_meta.clientId` in the open request.
|
||||
pub target_client_id: TargetClientId,
|
||||
}
|
||||
|
||||
impl FuzzySearchContext {
|
||||
pub fn new(
|
||||
root: &Path,
|
||||
hidden: bool,
|
||||
session_id: Option<String>,
|
||||
target_client_id: TargetClientId,
|
||||
) -> Self {
|
||||
let matcher = FuzzyFileMatcher::new(root);
|
||||
let daemon = FuzzyFileMatcherDaemon::new(matcher, DEFAULT_TOP_K);
|
||||
daemon.restart_walk(hidden);
|
||||
|
||||
Self {
|
||||
daemon,
|
||||
created_at: Instant::now(),
|
||||
last_activity: Instant::now(),
|
||||
hidden,
|
||||
min_generation: 0,
|
||||
has_query: false,
|
||||
query_version: 0,
|
||||
root: root.to_path_buf(),
|
||||
session_id,
|
||||
target_client_id,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_stale(&self, timeout: Duration) -> bool {
|
||||
self.last_activity.elapsed() > timeout
|
||||
}
|
||||
}
|
||||
|
||||
pub struct FuzzySearchManager {
|
||||
searches: HashMap<FuzzySearchId, FuzzySearchContext>,
|
||||
timeout: Duration,
|
||||
}
|
||||
|
||||
impl FuzzySearchManager {
|
||||
pub fn new(timeout: Duration) -> Self {
|
||||
Self {
|
||||
searches: HashMap::new(),
|
||||
timeout,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn open(
|
||||
&mut self,
|
||||
root: &Path,
|
||||
request_id: Option<String>,
|
||||
hidden: bool,
|
||||
session_id: Option<String>,
|
||||
target_client_id: TargetClientId,
|
||||
) -> FuzzySearchId {
|
||||
self.cleanup_stale();
|
||||
let search_id = request_id.unwrap_or_else(|| Uuid::now_v7().to_string());
|
||||
|
||||
let context = FuzzySearchContext::new(root, hidden, session_id, target_client_id);
|
||||
self.searches.insert(search_id.clone(), context);
|
||||
search_id
|
||||
}
|
||||
|
||||
/// Get the session ID for a search, if one was set.
|
||||
/// Used for routing notifications to session subscribers.
|
||||
pub fn get_session_id(&self, search_id: &str) -> Option<String> {
|
||||
self.searches
|
||||
.get(search_id)
|
||||
.and_then(|ctx| ctx.session_id.clone())
|
||||
}
|
||||
|
||||
/// Get the target client ID for a search, if one was set.
|
||||
/// Used for routing notifications to the correct client via relay.
|
||||
pub fn get_target_client_id(&self, search_id: &str) -> TargetClientId {
|
||||
self.searches
|
||||
.get(search_id)
|
||||
.map(|ctx| ctx.target_client_id.clone())
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Get the root path for a search.
|
||||
/// Used to convert relative paths to absolute paths in results.
|
||||
pub fn get_root(&self, search_id: &str) -> Option<PathBuf> {
|
||||
self.searches.get(search_id).map(|ctx| ctx.root.clone())
|
||||
}
|
||||
|
||||
pub fn change(
|
||||
&mut self,
|
||||
search_id: &str,
|
||||
query: &str,
|
||||
dirs_only: bool,
|
||||
) -> Option<(usize, bool, usize)> {
|
||||
let ctx = self.searches.get_mut(search_id)?;
|
||||
ctx.last_activity = Instant::now();
|
||||
|
||||
// Rewalk on empty query to refresh index when picker opens.
|
||||
if query.is_empty() {
|
||||
ctx.daemon.restart_walk(ctx.hidden);
|
||||
}
|
||||
|
||||
ctx.daemon.set_query(query, dirs_only);
|
||||
ctx.min_generation += 1;
|
||||
ctx.has_query = !query.is_empty();
|
||||
ctx.query_version += 1;
|
||||
Some((ctx.min_generation, ctx.has_query, ctx.query_version))
|
||||
}
|
||||
|
||||
pub fn is_current_query(&self, search_id: &str, query_version: usize) -> bool {
|
||||
self.searches
|
||||
.get(search_id)
|
||||
.is_some_and(|ctx| ctx.query_version == query_version)
|
||||
}
|
||||
|
||||
pub fn get_results(&mut self, search_id: &str) -> Option<FuzzySearchData> {
|
||||
let ctx = self.searches.get_mut(search_id)?;
|
||||
ctx.last_activity = Instant::now();
|
||||
|
||||
let results = ctx.daemon.get();
|
||||
|
||||
Some(FuzzySearchData {
|
||||
matches: results.topk.to_vec(),
|
||||
total: results.num_items,
|
||||
done: results.status.done,
|
||||
generation: results.generation,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn get_results_filtered(
|
||||
&mut self,
|
||||
search_id: &str,
|
||||
min_gen: usize,
|
||||
has_query: bool,
|
||||
) -> Option<FuzzySearchData> {
|
||||
let ctx = self.searches.get_mut(search_id)?;
|
||||
ctx.last_activity = Instant::now();
|
||||
|
||||
let results = ctx.daemon.get();
|
||||
|
||||
if results.generation < min_gen {
|
||||
return None;
|
||||
}
|
||||
|
||||
// Skip intermediate states: empty results or unscored defaults while scanning
|
||||
if has_query && !results.status.done {
|
||||
if results.topk.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let all_unscored = results
|
||||
.topk
|
||||
.iter()
|
||||
.all(|m| m.score == 0 && m.indices.is_empty());
|
||||
if all_unscored {
|
||||
return None;
|
||||
}
|
||||
}
|
||||
|
||||
Some(FuzzySearchData {
|
||||
matches: results.topk.to_vec(),
|
||||
total: results.num_items,
|
||||
done: results.status.done,
|
||||
generation: results.generation,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn close(&mut self, search_id: &str) -> bool {
|
||||
self.searches.remove(search_id).is_some()
|
||||
}
|
||||
|
||||
pub fn cleanup_stale(&mut self) {
|
||||
let timeout = self.timeout;
|
||||
self.searches.retain(|_, ctx| !ctx.is_stale(timeout));
|
||||
}
|
||||
|
||||
pub fn active_count(&self) -> usize {
|
||||
self.searches.len()
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for FuzzySearchManager {
|
||||
fn default() -> Self {
|
||||
Self::new(Duration::from_secs(DEFAULT_SEARCH_TIMEOUT_SECS))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
//! Shared filesystem core for the fs list/read ops.
|
||||
//!
|
||||
//! Owns the `ignore::WalkBuilder` configuration, glob overrides, the
|
||||
//! dirs-first paginated listing ([`list_directory_paged`]), and the
|
||||
//! binary-safe ranged-read primitives ([`read_range`], [`encode_chunk`])
|
||||
//! used by all three fs surfaces — the shell-local
|
||||
//! `session::file_system`, the shell-facing
|
||||
//! [`ext_fs`](super::ext_fs) `workspace.fs_*`, and the client-facing
|
||||
//! [`client_fs`](super::client_fs) `workspace.client_fs_*` — so walk and
|
||||
//! read fixes apply to every consumer. Each consumer maps the neutral
|
||||
//! [`ListedEntry`] / [`ChunkPayload`] to its own wire shape (absolute vs
|
||||
//! root-relative paths, RFC 3339 vs epoch-ms timestamps, MIME vs
|
||||
//! text/binary type tags).
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::SystemTime;
|
||||
|
||||
use base64::Engine;
|
||||
use ignore::{WalkBuilder, overrides::OverrideBuilder};
|
||||
use kigi_workspace_types::rpc::fs::FsReadEncoding;
|
||||
|
||||
/// Hard cap on entries collected per list call before sorting. A
|
||||
/// pathological directory truncates (`truncated = true`) instead of
|
||||
/// ballooning memory. Shared by every fs surface.
|
||||
pub const MAX_LIST_COLLECT: usize = 50_000;
|
||||
|
||||
/// Server-side cap on a single ranged read's effective byte budget
|
||||
/// (`min(length, max_bytes)`). 4 MiB raw (≈ 5.3 MiB base64) stays under
|
||||
/// the server's 8 MiB frame cap. Shared by every fs read surface.
|
||||
pub const MAX_READ_BYTES: u64 = 4 * 1024 * 1024;
|
||||
|
||||
/// Resolve a ranged read's effective byte budget, shared by every fs read
|
||||
/// surface so the clamp policy can't drift between them. An absent `length`
|
||||
/// means "to EOF", but the result is always capped at the caller's
|
||||
/// `max_bytes` and the hard [`MAX_READ_BYTES`] server limit — so a short
|
||||
/// read is expected, and callers detect "more data" by comparing the
|
||||
/// returned bytes (at `offset`) against the file `size`.
|
||||
pub fn clamp_read_length(length: Option<u64>, max_bytes: u64) -> u64 {
|
||||
length
|
||||
.unwrap_or(u64::MAX)
|
||||
.min(max_bytes)
|
||||
.min(MAX_READ_BYTES)
|
||||
}
|
||||
|
||||
/// Walk configuration. Field semantics mirror the `x.ai/fs/list` request.
|
||||
pub(super) struct FsWalk<'a> {
|
||||
pub depth: usize,
|
||||
pub follow_symlinks: bool,
|
||||
pub respect_git_ignore: bool,
|
||||
pub include_hidden: bool,
|
||||
pub include_globs: &'a [String],
|
||||
pub exclude_globs: &'a [String],
|
||||
/// When set, symlink entries whose canonical target leaves this
|
||||
/// canonical root are excluded — and not descended into — so a walk
|
||||
/// of a confined tree cannot enumerate paths outside it. `None`
|
||||
/// preserves the shell's unconfined semantics.
|
||||
pub confine_to_canonical_root: Option<PathBuf>,
|
||||
}
|
||||
|
||||
/// One raw walk entry; `metadata` follows symlinks (like `fs::metadata`).
|
||||
pub(super) struct RawFsEntry {
|
||||
pub path: PathBuf,
|
||||
pub name: String,
|
||||
pub is_symlink: bool,
|
||||
pub metadata: std::fs::Metadata,
|
||||
}
|
||||
|
||||
/// Walk `abs_dir` per `opts`, collecting up to `max_entries` entries (the
|
||||
/// root itself is skipped; unreadable entries are skipped without counting).
|
||||
/// Returns `(entries, hit_cap)` where `hit_cap` means the walk stopped at
|
||||
/// the cap with entries left over.
|
||||
pub(super) fn walk_fs_entries(
|
||||
abs_dir: &Path,
|
||||
opts: FsWalk<'_>,
|
||||
max_entries: usize,
|
||||
) -> (Vec<RawFsEntry>, bool) {
|
||||
let overrides = build_glob_overrides(abs_dir, opts.include_globs, opts.exclude_globs);
|
||||
let mut builder = WalkBuilder::new(abs_dir);
|
||||
builder
|
||||
.max_depth(Some(opts.depth))
|
||||
.follow_links(opts.follow_symlinks)
|
||||
.same_file_system(true)
|
||||
.standard_filters(true)
|
||||
.git_ignore(opts.respect_git_ignore)
|
||||
.git_global(opts.respect_git_ignore)
|
||||
.git_exclude(opts.respect_git_ignore)
|
||||
.hidden(!opts.include_hidden)
|
||||
.overrides(overrides);
|
||||
if let Some(canonical_root) = opts.confine_to_canonical_root {
|
||||
builder.filter_entry(move |dent| symlink_stays_in_root(dent.path(), &canonical_root));
|
||||
}
|
||||
|
||||
let mut entries: Vec<RawFsEntry> = Vec::new();
|
||||
let mut hit_cap = false;
|
||||
for dent in builder.build() {
|
||||
let Ok(entry) = dent else { continue };
|
||||
if entry.depth() == 0 {
|
||||
continue;
|
||||
}
|
||||
if entries.len() >= max_entries {
|
||||
hit_cap = true;
|
||||
break;
|
||||
}
|
||||
let path = entry.path().to_path_buf();
|
||||
let is_symlink = std::fs::symlink_metadata(&path)
|
||||
.map(|m| m.file_type().is_symlink())
|
||||
.unwrap_or(false);
|
||||
let metadata = match std::fs::metadata(&path) {
|
||||
Ok(m) => m,
|
||||
Err(_) => continue,
|
||||
};
|
||||
entries.push(RawFsEntry {
|
||||
name: entry.file_name().to_string_lossy().into_owned(),
|
||||
path,
|
||||
is_symlink,
|
||||
metadata,
|
||||
});
|
||||
}
|
||||
(entries, hit_cap)
|
||||
}
|
||||
|
||||
/// `true` when `path` is not a symlink, or is a symlink whose canonical
|
||||
/// target stays under `canonical_root`. Unverifiable symlinks (e.g.
|
||||
/// dangling) are excluded — confinement fails closed.
|
||||
fn symlink_stays_in_root(path: &Path, canonical_root: &Path) -> bool {
|
||||
let is_symlink = std::fs::symlink_metadata(path)
|
||||
.map(|m| m.file_type().is_symlink())
|
||||
.unwrap_or(false);
|
||||
if !is_symlink {
|
||||
return true;
|
||||
}
|
||||
dunce::canonicalize(path)
|
||||
.map(|c| c.starts_with(canonical_root))
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
pub(super) fn build_glob_overrides(
|
||||
base: &Path,
|
||||
include: &[String],
|
||||
exclude: &[String],
|
||||
) -> ignore::overrides::Override {
|
||||
let mut ob = OverrideBuilder::new(base);
|
||||
for pat in include {
|
||||
let patt = if pat.starts_with('!') {
|
||||
pat.clone()
|
||||
} else {
|
||||
format!("!{}", pat)
|
||||
};
|
||||
let _ = ob.add(&patt);
|
||||
}
|
||||
for pat in exclude {
|
||||
let _ = ob.add(pat);
|
||||
}
|
||||
ob.build().unwrap_or_else(|_| {
|
||||
OverrideBuilder::new(base)
|
||||
.build()
|
||||
.expect("override build fallback")
|
||||
})
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Paginated listing
|
||||
// =========================================================================
|
||||
|
||||
/// One listed node in neutral form (no wire serialization). Consumers map
|
||||
/// this to their own node shape.
|
||||
pub struct ListedEntry {
|
||||
/// File name (final path component).
|
||||
pub name: String,
|
||||
/// Absolute path on the workspace/host filesystem.
|
||||
pub abs_path: PathBuf,
|
||||
/// Whether the entry is a directory.
|
||||
pub is_dir: bool,
|
||||
/// Whether the entry itself is a symlink.
|
||||
pub is_symlink: bool,
|
||||
/// Size in bytes (files only).
|
||||
pub size: Option<u64>,
|
||||
/// Modification time, when readable.
|
||||
pub modified: Option<SystemTime>,
|
||||
}
|
||||
|
||||
/// One page of a directory listing.
|
||||
pub struct ListPage {
|
||||
/// The page slice `[offset, offset + limit)` after the dirs-first sort.
|
||||
pub entries: Vec<ListedEntry>,
|
||||
/// `true` when more entries exist beyond this page, or the collection
|
||||
/// cap was hit before the walk finished.
|
||||
pub truncated: bool,
|
||||
}
|
||||
|
||||
/// Options for [`list_directory_paged`].
|
||||
pub struct ListOptions<'a> {
|
||||
pub depth: usize,
|
||||
pub follow_symlinks: bool,
|
||||
pub respect_git_ignore: bool,
|
||||
pub include_hidden: bool,
|
||||
pub include_globs: &'a [String],
|
||||
pub exclude_globs: &'a [String],
|
||||
/// Pagination offset applied after the sort.
|
||||
pub offset: u64,
|
||||
/// Page size (already clamped by the caller as appropriate).
|
||||
pub limit: usize,
|
||||
/// When set, mid-walk symlink escapes outside this canonical root are
|
||||
/// excluded; `None` keeps the shell's unconfined semantics.
|
||||
pub confine_to_canonical_root: Option<PathBuf>,
|
||||
}
|
||||
|
||||
/// Whether more entries exist than this page returned. True when entries
|
||||
/// remain beyond the page (`end < total`) OR the walk hit the collection cap
|
||||
/// AND the caller has not yet paged past the collected window
|
||||
/// (`hit_cap && start < total`). Gating the cap term on `start < total` is
|
||||
/// what makes a `while truncated { offset += limit }` loop terminate: once a
|
||||
/// client has consumed every collected entry the flag drops to false instead
|
||||
/// of reporting the (unreachable) over-cap remainder forever.
|
||||
fn page_truncated(start: usize, end: usize, total: usize, hit_cap: bool) -> bool {
|
||||
end < total || (hit_cap && start < total)
|
||||
}
|
||||
|
||||
/// Walk `abs_dir`, sort directories-first / case-insensitive (with exact
|
||||
/// name as a deterministic tiebreak), then return the stable slice
|
||||
/// `[offset, offset + limit)`. See [`page_truncated`] for the `truncated`
|
||||
/// semantics (incomplete listing OR more pages, but terminating).
|
||||
pub fn list_directory_paged(abs_dir: &Path, opts: ListOptions<'_>, max_collect: usize) -> ListPage {
|
||||
let (raw, hit_cap) = walk_fs_entries(
|
||||
abs_dir,
|
||||
FsWalk {
|
||||
depth: opts.depth,
|
||||
follow_symlinks: opts.follow_symlinks,
|
||||
respect_git_ignore: opts.respect_git_ignore,
|
||||
include_hidden: opts.include_hidden,
|
||||
include_globs: opts.include_globs,
|
||||
exclude_globs: opts.exclude_globs,
|
||||
confine_to_canonical_root: opts.confine_to_canonical_root,
|
||||
},
|
||||
max_collect,
|
||||
);
|
||||
|
||||
let mut entries: Vec<ListedEntry> = raw
|
||||
.into_iter()
|
||||
.map(|e| ListedEntry {
|
||||
is_dir: e.metadata.is_dir(),
|
||||
size: e.metadata.is_file().then_some(e.metadata.len()),
|
||||
modified: e.metadata.modified().ok(),
|
||||
is_symlink: e.is_symlink,
|
||||
abs_path: e.path,
|
||||
name: e.name,
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Directories first, then case-insensitive by name; exact name as a
|
||||
// tiebreak so page boundaries are deterministic.
|
||||
entries.sort_by_cached_key(|n| (!n.is_dir, n.name.to_lowercase(), n.name.clone()));
|
||||
|
||||
let total = entries.len();
|
||||
let start = usize::try_from(opts.offset)
|
||||
.unwrap_or(usize::MAX)
|
||||
.min(total);
|
||||
let end = start.saturating_add(opts.limit).min(total);
|
||||
let truncated = page_truncated(start, end, total, hit_cap);
|
||||
entries.truncate(end);
|
||||
let page = entries.split_off(start);
|
||||
ListPage {
|
||||
entries: page,
|
||||
truncated,
|
||||
}
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// Ranged, binary-safe reads
|
||||
// =========================================================================
|
||||
|
||||
/// A read chunk in the requested transfer encoding.
|
||||
pub enum ChunkPayload {
|
||||
/// Valid UTF-8 text (caller places it in a `content` field).
|
||||
Text(String),
|
||||
/// Base64 of the raw bytes (caller places it in a `contentBase64`
|
||||
/// field). Used when `base64` is requested or the bytes are not UTF-8.
|
||||
Base64(String),
|
||||
}
|
||||
|
||||
/// Encode `bytes` per `encoding`, returning the payload and whether the
|
||||
/// bytes were valid UTF-8 (`is_text`). One UTF-8 validation pass; on the
|
||||
/// `Utf8` request the error hands the bytes back for base64 fallback.
|
||||
pub fn encode_chunk(bytes: Vec<u8>, encoding: FsReadEncoding) -> (ChunkPayload, bool) {
|
||||
let b64 = |b: &[u8]| base64::engine::general_purpose::STANDARD.encode(b);
|
||||
match (encoding, String::from_utf8(bytes)) {
|
||||
(FsReadEncoding::Utf8, Ok(text)) => (ChunkPayload::Text(text), true),
|
||||
(FsReadEncoding::Utf8, Err(e)) => (ChunkPayload::Base64(b64(e.as_bytes())), false),
|
||||
(FsReadEncoding::Base64, Ok(text)) => (ChunkPayload::Base64(b64(text.as_bytes())), true),
|
||||
(FsReadEncoding::Base64, Err(e)) => (ChunkPayload::Base64(b64(e.as_bytes())), false),
|
||||
}
|
||||
}
|
||||
|
||||
/// Read only `[offset, offset + length)` of `abs` (no hashing).
|
||||
pub async fn read_range(abs: &Path, offset: u64, length: u64) -> std::io::Result<Vec<u8>> {
|
||||
use tokio::io::{AsyncReadExt, AsyncSeekExt};
|
||||
|
||||
let mut f = tokio::fs::File::open(abs).await?;
|
||||
if offset > 0 {
|
||||
f.seek(std::io::SeekFrom::Start(offset)).await?;
|
||||
}
|
||||
let mut chunk = Vec::new();
|
||||
f.take(length).read_to_end(&mut chunk).await?;
|
||||
Ok(chunk)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn clamp_read_length_caps_at_max_bytes_and_hard_limit() {
|
||||
// Absent length -> capped at max_bytes.
|
||||
assert_eq!(clamp_read_length(None, 1024), 1024);
|
||||
// Explicit length above max_bytes -> max_bytes wins.
|
||||
assert_eq!(clamp_read_length(Some(8192), 1024), 1024);
|
||||
// Explicit length below max_bytes -> length wins.
|
||||
assert_eq!(clamp_read_length(Some(512), 1024), 512);
|
||||
// max_bytes above the hard server limit -> hard limit wins.
|
||||
assert_eq!(clamp_read_length(None, u64::MAX), MAX_READ_BYTES);
|
||||
assert_eq!(clamp_read_length(Some(u64::MAX), u64::MAX), MAX_READ_BYTES);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn page_truncated_signals_more_pages_within_collected_set() {
|
||||
// 100 collected, no cap, page [0,10) -> more remain.
|
||||
assert!(page_truncated(0, 10, 100, false));
|
||||
// Last page [90,100) -> nothing remains.
|
||||
assert!(!page_truncated(90, 100, 100, false));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn page_truncated_terminates_when_paging_past_collection_cap() {
|
||||
// Cap hit, total clamped to 50 collected, limit 10.
|
||||
// Populated pages stay truncated (listing is incomplete)...
|
||||
assert!(page_truncated(0, 10, 50, true));
|
||||
assert!(page_truncated(40, 50, 50, true));
|
||||
// ...but once the client pages past the collected window the flag
|
||||
// drops, so `while truncated { offset += limit }` terminates instead
|
||||
// of fetching empty pages forever.
|
||||
assert!(!page_truncated(50, 50, 50, true));
|
||||
assert!(!page_truncated(60, 50, 50, true));
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user