M0: compilable skeleton — Kigi 0.1.0 fork surgery

Hard fork of xai-org/grok-build (Apache-2.0) re-targeted as Kigi, an
unofficial Kimi Code CLI community build.

Rename & identity
- 72 xai-*/xai-grok-* crates -> kigi-* (explicit: xai-grok-pager-bin ->
  kigi-bin [binary `kigi`], xai-grok-pager -> kigi-tui; rest mechanical);
  ptyctl, ptyctl-cli, third_party/ unchanged; proto package
  xai.grok.tools.v1 -> kigi.tools.v1
- Config home ~/.kigi (KIGI_SHARE_DIR override), env prefix GROK_* ->
  KIGI_*, `kigi --version` carries the unofficial-community-build notice
- clap identity, help text, startup banner, prompt templates rebranded
  (templates re-encrypted)

Deletions (PRD removal list #5/#6/#7/#9/#10)
- voice input (xai-grok-voice) and all TUI wiring
- telemetry: Mixpanel client, external OTel stream, Sentry, OTLP layers,
  trace/GCS/S3 upload queues (kigi-file-utils halved), workspace upload
  module & dc_log, heap-profile uploader, auth-diagnostics uploader,
  session-analytics halves of feedback; local zero-egress observability
  preserved in new kigi-log crate (unified log, --debug firehose,
  subsystem file logs, opt-in instrumentation)
- announcements (crate, remote-settings fields, TUI surfaces)
- plugin marketplace (crate, sources/browse/CTA/extensions-modal tab);
  direct plugin install/uninstall/update via kigi-agent git_install kept
- relay/gateway/assets endpoints and features (agent relay, headless
  relay transport, gateway bridge, LeaderEnvUrls); leader IPC socket now
  ~/.kigi/leader.sock + KIGI_LEADER_SOCKET, no ws-url derivation
- functional types rehomed instead of deleted: PermissionMode ->
  kigi-config-types, McpInitStrategy -> kigi-mcp, PrCreationSource ->
  session signals, TerminalDiagnostics -> kigi-pager-render, agent_id ->
  shell util

Endpoints
- kigi-env rewritten: single production KigiEndpoints {coding_api_base_url
  https://api.kimi.com/coding/v1 (KIGI_CODE_BASE_URL), oauth_host
  https://auth.kimi.com (KIGI_OAUTH_HOST), update_base_url (GitHub
  Releases API), upgrade_page_url}; GrokBuildEnvironment enum deleted

Toolchain & workspace hygiene
- Rust 1.97.0 pinned; edition 2024; full cargo update; git2 hoisted to
  workspace at 0.21 (Option->Result API migration), quick-xml 0.41
- Root Cargo.toml hand-maintained (PRD §8.1): version 0.1.0 inherited by
  all members, members sorted, unused deps pruned
- cargo-deny advisories gate (deny.toml with documented transitive
  exceptions); CI workflow (check/clippy/fmt/deny/test, macOS+Linux)
- cross-crate test seams re-gated behind `test-support` cargo feature;
  insta snapshot baselines renamed to the kigi_tui prefix
- clippy --workspace --all-targets: zero warnings; fmt clean

Fixes surfaced by the port
- updater probe/installer divergence (bin/kigi vs bin/grok symlink set)
- idle model-metadata refresh dead under KIGI_CODE_BASE_URL override
  (new is_effective_coding_endpoint_url, loopback+override aware)
- macOS symlinked-TMPDIR fixture canonicalization (foreign_sessions,
  fast-worktree); RSS measurement tests serialized via serial_test

Docs & legal (Apache §4)
- NOTICE added (upstream attribution + change statement); THIRD-PARTY
  notices sustained; kigi-tools ported-code notices extended; README,
  CONTRIBUTING, SECURITY, AGENTS.md rewritten

Out of scope for M0 (tracked): Kimi auth/inference (M1), search/fetch,
command parity, config import (M2), Computer Hub excision & final
brand-token sweep (M2), distribution & self-update rewrite (M3).
This commit is contained in:
2026-07-17 05:31:01 -04:00
commit d6c20fc13f
2612 changed files with 1353757 additions and 0 deletions
@@ -0,0 +1,179 @@
use crate::file_system::{AsyncFileSystem, FsError};
use agent_client_protocol as acp;
use kigi_acp_lib::AcpAgentGatewaySender as GatewaySender;
use std::path::{Path, PathBuf};
pub struct AcpSessionFs {
root: PathBuf,
gateway: GatewaySender,
session_id: acp::SessionId,
/// When set, any path under `display_cwd` is rewritten to `root` before
/// being sent to the extension. This is the defense-in-depth guard for
/// AB overlay isolation: if a tool accidentally passes the display path
/// (e.g., `/testbed/project/foo.rs`) instead of the overlay path
/// (`~/.kigi/worktrees/.../b-overlay/foo.rs`), the adapter rewrites it
/// so the extension reads/writes to the correct overlay location.
display_cwd: Option<PathBuf>,
}
impl AcpSessionFs {
pub fn new(root: PathBuf, session_id: acp::SessionId, gateway: GatewaySender) -> Self {
Self {
root,
session_id,
gateway,
display_cwd: None,
}
}
/// Set the display CWD for path rewriting.
///
/// When AB FS isolation is active, the model sees `display_cwd`
/// (e.g., `/testbed/project`) but writes should go to `root`
/// (the overlay path). Any path under `display_cwd` is rewritten
/// to the equivalent path under `root`.
pub fn with_display_cwd(mut self, display_cwd: PathBuf) -> Self {
self.display_cwd = Some(display_cwd);
self
}
/// Rewrite a display path to the overlay path if needed.
fn resolve_path(&self, path: &Path) -> PathBuf {
if let Some(ref display) = self.display_cwd
&& let Ok(suffix) = path.strip_prefix(display)
{
let resolved = self.root.join(suffix);
tracing::debug!(
display_path = %path.display(),
resolved_path = %resolved.display(),
"AcpSessionFs: rewrote display path to overlay path"
);
return resolved;
}
path.to_path_buf()
}
}
#[async_trait::async_trait]
impl AsyncFileSystem for AcpSessionFs {
fn root(&self) -> &Path {
&self.root
}
async fn exists(&self, path: &Path) -> Result<bool, FsError> {
let resolved = self.resolve_path(path);
let read_req = acp::ReadTextFileRequest::new(self.session_id.clone(), resolved).limit(0);
match self.gateway.send(read_req).await {
Ok(_) => Ok(true),
Err(e) if e.code == acp::ErrorCode::ResourceNotFound => Ok(false),
Err(e) => Err(FsError::Other(e.to_string())),
}
}
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, FsError> {
let resolved = self.resolve_path(path);
let read_req = acp::ReadTextFileRequest::new(self.session_id.clone(), resolved);
let response = self
.gateway
.send(read_req)
.await
.map_err(|e| FsError::Other(e.to_string()))?;
Ok(response.content.into_bytes())
}
async fn try_read_file(&self, path: &Path) -> Result<Option<Vec<u8>>, FsError> {
let resolved = self.resolve_path(path);
let read_req = acp::ReadTextFileRequest::new(self.session_id.clone(), resolved);
match self.gateway.send(read_req).await {
Ok(response) => Ok(Some(response.content.into_bytes())),
Err(e) if e.code == acp::ErrorCode::ResourceNotFound => Ok(None),
Err(e) => Err(FsError::Other(e.to_string())),
}
}
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), FsError> {
let resolved = self.resolve_path(path);
let write_req = acp::WriteTextFileRequest::new(
self.session_id.clone(),
resolved,
String::from_utf8(data.to_vec()).map_err(|e| FsError::Other(e.to_string()))?,
);
self.gateway
.send(write_req)
.await
.map_err(|e| FsError::Other(e.to_string()))?;
Ok(())
}
async fn delete_file(&self, path: &Path) -> Result<(), FsError> {
// ACP protocol doesn't support file deletion yet
// For now, we'll log a warning and return Ok (no-op)
tracing::warn!(?path, "ACP filesystem does not support file deletion");
Err(FsError::Other(
"File deletion not supported via ACP".to_string(),
))
}
}
#[cfg(test)]
mod tests {
use super::*;
// resolve_path only uses self.root and self.display_cwd — extract
// the logic into a standalone test helper that doesn't need a gateway.
fn test_resolve(root: &str, display_cwd: Option<&str>, input: &str) -> PathBuf {
let root = PathBuf::from(root);
let display = display_cwd.map(PathBuf::from);
// Inline the same logic as resolve_path
if let Some(ref display) = display
&& let Ok(suffix) = Path::new(input).strip_prefix(display)
{
return root.join(suffix);
}
PathBuf::from(input)
}
#[test]
fn resolve_path_rewrites_display_to_overlay() {
let result = test_resolve(
"/root/.kigi/worktrees/proj/ab-123-b-overlay",
Some("/testbed/proj"),
"/testbed/proj/src/main.rs",
);
assert_eq!(
result,
PathBuf::from("/root/.kigi/worktrees/proj/ab-123-b-overlay/src/main.rs")
);
}
#[test]
fn resolve_path_passes_through_overlay_path() {
let overlay_path = "/root/.kigi/worktrees/proj/ab-123-b-overlay/src/main.rs";
let result = test_resolve(
"/root/.kigi/worktrees/proj/ab-123-b-overlay",
Some("/testbed/proj"),
overlay_path,
);
assert_eq!(result, PathBuf::from(overlay_path));
}
#[test]
fn resolve_path_no_display_cwd_passthrough() {
let result = test_resolve(
"/root/.kigi/worktrees/proj/ab-123-b-overlay",
None,
"/testbed/proj/src/main.rs",
);
assert_eq!(result, PathBuf::from("/testbed/proj/src/main.rs"));
}
#[test]
fn resolve_path_relative_path_passthrough() {
let result = test_resolve(
"/root/.kigi/worktrees/proj/ab-123-b-overlay",
Some("/testbed/proj"),
"src/main.rs",
);
assert_eq!(result, PathBuf::from("src/main.rs"));
}
}
@@ -0,0 +1,89 @@
//! AcpFsAdapter: implements `kigi-tools::AsyncFileSystem` using ACP gateway calls.
//!
//! This adapter enables file tool execution over ACP (remote filesystem).
//! It translates kigi-tools' `AsyncFileSystem` trait into ACP protocol calls:
//! `read_file()` → read_text_file
//! `write_file()` → write_text_file
//! `delete_file()` → not supported by ACP (returns error)
//!
//! Mirrors the pattern of `AcpTerminalAdapter` for terminal execution.
use std::path::Path;
use agent_client_protocol as acp;
use kigi_acp_lib::AcpAgentGatewaySender as GatewaySender;
use kigi_tools::computer::types::{AsyncFileSystem, ComputerError};
/// Wraps kigi-shell's ACP gateway to satisfy kigi-tools' AsyncFileSystem.
///
/// When a client advertises `clientCapabilities.fs.readTextFile` and `writeTextFile`,
/// file operations from tools (read_file, search_replace, etc.) are routed through
/// the ACP gateway back to the client instead of hitting the local disk directly.
pub struct AcpFsAdapter {
gateway: GatewaySender,
session_id: acp::SessionId,
}
impl AcpFsAdapter {
pub fn new(gateway: GatewaySender, session_id: acp::SessionId) -> Self {
Self {
gateway,
session_id,
}
}
}
#[async_trait::async_trait]
impl AsyncFileSystem for AcpFsAdapter {
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, ComputerError> {
let read_req = acp::ReadTextFileRequest::new(self.session_id.clone(), path.to_path_buf());
let response = self
.gateway
.send(read_req)
.await
.map_err(acp_error_to_computer_error)?;
Ok(response.content.into_bytes())
}
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), ComputerError> {
let content =
String::from_utf8(data.to_vec()).map_err(|e| ComputerError::io(e.to_string()))?;
let write_req =
acp::WriteTextFileRequest::new(self.session_id.clone(), path.to_path_buf(), content);
self.gateway
.send(write_req)
.await
.map_err(acp_error_to_computer_error)?;
Ok(())
}
async fn delete_file(&self, path: &Path) -> Result<(), ComputerError> {
// ACP protocol doesn't support file deletion yet
tracing::warn!(?path, "ACP filesystem does not support file deletion");
Err(ComputerError::io("File deletion not supported via ACP"))
}
}
fn acp_error_to_computer_error(err: acp::Error) -> ComputerError {
match acp_error_to_io_kind(&err) {
Some(kind) => ComputerError::io_with_kind(err.to_string(), kind),
None => ComputerError::io(err.to_string()),
}
}
fn acp_error_to_io_kind(err: &acp::Error) -> Option<std::io::ErrorKind> {
let msg_lower = err.message.to_ascii_lowercase();
if err.code == acp::ErrorCode::ResourceNotFound {
Some(std::io::ErrorKind::NotFound)
} else if msg_lower.contains("permission denied") {
Some(std::io::ErrorKind::PermissionDenied)
} else {
None
}
}
@@ -0,0 +1,540 @@
//! Contains utility functions to attach file content and render it according to the
//! training format we have been using
use agent_client_protocol::{BlobResourceContents, EmbeddedResource, EmbeddedResourceResource};
use base64::{Engine as _, engine::general_purpose};
use kigi_tools::util::truncate::estimate_tokens;
use regex::Regex;
use sha2::{Digest, Sha256};
use std::path::PathBuf;
use tracing::warn;
#[cfg(test)]
mod persistence {
use std::path::PathBuf;
pub fn session_dir(_suffix: &str) -> PathBuf {
super::session_scratch_root()
}
}
/// Maximum number of estimated tokens for a file to be included inline.
/// Files exceeding this limit are represented as a metadata-only stub so the
/// model knows the file exists without blowing up the context window.
const MAX_FILE_TOKENS: usize = 5_000;
/// 8-char content hash for dedup + collision avoidance.
fn content_hash(content: &[u8]) -> String {
format!("{:x}", Sha256::digest(content))[..8].to_string()
}
/// Parsed file reference with optional line range.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct FileReference {
pub path: PathBuf,
/// 1-indexed start line (inclusive)
pub start_line: Option<usize>,
/// 1-indexed end line (inclusive)
pub end_line: Option<usize>,
}
impl FileReference {
/// Parse a file reference string in the format `@{file_path}` or `@{file_path}:L?{start_line}-L?{end_line}`.
pub fn parse(input: &str) -> Option<Self> {
let re = Regex::new(r"^@?([^@].*?)(?::L?(\d+)-L?(\d+))?$").ok()?;
let caps = re.captures(input)?;
let path = PathBuf::from(caps.get(1)?.as_str());
let start_line: Option<usize> = caps.get(2).and_then(|m| m.as_str().parse().ok());
let end_line: Option<usize> = caps.get(3).and_then(|m| m.as_str().parse().ok());
Some(Self {
path,
start_line,
end_line,
})
}
}
/// Render file content from a `FileReference`.
///
/// When `is_cursor` is true, renders `<code_selection path="..." lines="X-Y">` format.
/// When `is_cursor` is false, renders the original `<file_contents path="..." startLine/endLine/isFullFile>` format.
///
/// If the rendered content exceeds [`MAX_FILE_TOKENS`] estimated tokens the
/// full body is omitted and a metadata-only stub is returned instead so the
/// model still knows the file exists.
pub async fn render_file_reference(file_ref: FileReference, is_cursor: bool) -> Option<String> {
let read_file = tokio::fs::read(&file_ref.path).await;
let file_content = if let Ok(read_file_output) = read_file {
String::from_utf8(read_file_output).ok()
} else {
None
};
let path = file_ref.path.to_string_lossy();
let start_line = file_ref.start_line;
let end_line = file_ref.end_line;
file_content
.map(|file_content| {
let lines: Vec<&str> = file_content.lines().collect();
let line_offset = start_line.unwrap_or(1);
let start_idx = (line_offset.saturating_sub(1)).min(lines.len());
let end_idx = end_line.unwrap_or(lines.len()).min(lines.len());
let sliced_lines = &lines[start_idx..end_idx];
let file_content = sliced_lines
.iter()
.enumerate()
.map(|(line_number, content)| {
format!("{}{content}", line_offset + line_number)
})
.collect::<Vec<_>>()
.join("\n");
let _ = is_cursor;
let attrs = match (start_line, end_line) {
(Some(s), Some(e)) => format!(r#"startLine="{}" endLine="{}""#, s, e),
_ => r#"isFullFile="true""#.to_string(),
};
if estimate_tokens(&file_content) > MAX_FILE_TOKENS {
return format!(
r#"<file_contents path="{path}" {attrs} skipped="true" reason="file too large (~{} estimated tokens, limit {MAX_FILE_TOKENS}). Use read_file tool to read specific sections."/>"#,
estimate_tokens(& file_content),
);
}
format!(
r#"<file_contents path="{path}" {attrs}>
{file_content}
</file_contents>"#
)
})
}
const FILE_REGEX: &str = r"^(?:file://)?([^#]+)(?:#L(\d+)-L?(\d+))?$";
/// Render an ACP EmbeddedResource.
///
/// When `is_cursor` is true, renders `<code_selection>` tags. Otherwise uses `<file_contents>`.
/// Parses URIs in the format: `file://[path]#L[start]-[end]` or `file://[path]#L[start]-L[end]`
///
/// When content exceeds [`MAX_FILE_TOKENS`], the text is written to
/// `~/.kigi/sessions/{cwd}/{session_id}/pasted/` so the model can `read_file`
/// specific sections instead of receiving the full content inline.
///
/// Binary blob resources are written to `attachments/` and a path hint is returned.
pub async fn render_embedded_resource(
resource: &EmbeddedResource,
is_cursor: bool,
) -> Option<String> {
match &resource.resource {
EmbeddedResourceResource::TextResourceContents(text_resource)
if text_resource.mime_type.as_deref() == Some("text/x-diff") =>
{
render_diff_resource(text_resource).await
}
EmbeddedResourceResource::TextResourceContents(text_resource) => {
render_text_resource(text_resource, is_cursor).await
}
EmbeddedResourceResource::BlobResourceContents(blob) => render_blob_attachment(blob).await,
_ => None,
}
}
async fn render_text_resource(
text_resource: &agent_client_protocol::TextResourceContents,
is_cursor: bool,
) -> Option<String> {
let re = Regex::new(FILE_REGEX).ok()?;
let caps = re.captures(&text_resource.uri)?;
let path = caps.get(1)?.as_str();
let start_line: Option<usize> = caps.get(2).and_then(|m| m.as_str().parse().ok());
let end_line: Option<usize> = caps.get(3).and_then(|m| m.as_str().parse().ok());
let line_offset = start_line.unwrap_or(1);
let file_content = text_resource
.text
.lines()
.enumerate()
.map(|(line_number, content)| format!("{}{content}", line_offset + line_number))
.collect::<Vec<_>>()
.join("\n");
let attrs = match (start_line, end_line) {
(Some(s), Some(e)) => format!(r#"startLine="{}" endLine="{}""#, s, e),
_ => r#"isFullFile="true""#.to_string(),
};
let (tag, attrs_str) = ("file_contents", attrs);
let _ = is_cursor;
if estimate_tokens(&file_content) > MAX_FILE_TOKENS {
let dest = write_to_session_subdir("pasted", path, text_resource.text.as_bytes()).await;
let display_path = dest
.as_ref()
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|| path.to_string());
return Some(format!(
r#"<{tag} path="{display_path}" {attrs_str} skipped="true" reason="file too large (~{} estimated tokens, limit {MAX_FILE_TOKENS}). Use read_file tool to read specific sections."/>"#,
estimate_tokens(&file_content),
));
}
Some(format!(
r#"<{tag} path="{path}" {attrs_str}>
{file_content}
</{tag}>"#
))
}
/// Render a diff citation as `<diff_contents>`.
///
/// The text is passed through verbatim (no line-number rewriting) since it
/// represents a change, not a file snapshot.
async fn render_diff_resource(
text_resource: &agent_client_protocol::TextResourceContents,
) -> Option<String> {
let re = Regex::new(FILE_REGEX).ok()?;
let caps = re.captures(&text_resource.uri)?;
let path = caps.get(1)?.as_str();
let start_line: Option<usize> = caps.get(2).and_then(|m| m.as_str().parse().ok());
let end_line: Option<usize> = caps.get(3).and_then(|m| m.as_str().parse().ok());
let mut attrs = Vec::new();
if let Some(s) = start_line {
attrs.push(format!(r#"startLine="{s}""#));
}
if let Some(e) = end_line {
attrs.push(format!(r#"endLine="{e}""#));
}
let attrs_str = if attrs.is_empty() {
String::new()
} else {
format!(" {}", attrs.join(" "))
};
if estimate_tokens(&text_resource.text) > MAX_FILE_TOKENS {
let dest = write_to_session_subdir("pasted", path, text_resource.text.as_bytes()).await;
let display_path = dest
.as_ref()
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|| path.to_string());
return Some(format!(
r#"<diff_contents path="{display_path}"{attrs_str} skipped="true" reason="diff too large (~{} estimated tokens, limit {MAX_FILE_TOKENS}). Use read_file tool to read specific sections."/>"#,
estimate_tokens(&text_resource.text),
));
}
Some(format!(
r#"<diff_contents path="{path}"{attrs_str}>
{text}
</diff_contents>"#,
text = text_resource.text,
))
}
/// Decode base64 blob, write to `attachments/`, return `<file_contents type="binary">` hint.
async fn render_blob_attachment(blob: &BlobResourceContents) -> Option<String> {
let raw_name = blob.uri.strip_prefix("file://").unwrap_or(&blob.uri);
let filename = PathBuf::from(raw_name)
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| "attachment".to_string());
let bytes = match general_purpose::STANDARD.decode(&blob.blob) {
Ok(b) => b,
Err(e) => {
warn!("binary attachment {filename}: base64 decode failed: {e}");
return None;
}
};
let size = bytes.len();
let mime = blob
.mime_type
.as_deref()
.unwrap_or("application/octet-stream");
let dest = write_to_session_subdir("attachments", &filename, &bytes).await?;
let path = dest.to_string_lossy();
Some(format!(
r#"<file_contents type="binary" path="{path}" mime_type="{mime}" size="{size}"/>"#
))
}
/// Base directory for Phase-1 session-scoped scratch files. Namespaced by PID so
/// concurrent test processes (repeated or parallel CI test runs that share
/// `/tmp`) never collide on identical content-hash paths and
/// race each other's cleanup. The real session_dir lives in shell persistence.
fn session_scratch_root() -> PathBuf {
std::env::temp_dir().join(format!("grok-test-sessions-{}", std::process::id()))
}
/// Write content to session subdir, return absolute path on success.
/// Uses content hash prefix for dedup: identical content → same path, different content → unique path.
async fn write_to_session_subdir(subdir: &str, filename: &str, content: &[u8]) -> Option<PathBuf> {
let dir = session_scratch_root().join(subdir);
if let Err(e) = tokio::fs::create_dir_all(&dir).await {
warn!("failed to create {subdir} directory: {e}");
return None;
}
let basename = PathBuf::from(filename)
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| filename.to_string());
let hash = content_hash(content);
let dest = dir.join(format!("{hash}-{basename}"));
if dest.exists() {
return Some(dest);
}
if let Err(e) = tokio::fs::write(&dest, content).await {
warn!("failed to write to {}: {e}", dest.display());
return None;
}
Some(dest)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_file_url_regex() {
let expected = vec![
(
"file://Users/alice/first.txt#L10-20",
"Users/alice/first.txt",
"10",
"20",
),
(
"file://Users/alice/second.txt#L12-L44",
"Users/alice/second.txt",
"12",
"44",
),
];
let re = Regex::new(FILE_REGEX).unwrap();
for (path, expected_path, expected_start, expected_end) in expected {
let captures = re.captures(path).unwrap();
assert_eq!(captures.get(1).unwrap().as_str(), expected_path);
assert_eq!(captures.get(2).unwrap().as_str(), expected_start);
assert_eq!(captures.get(3).unwrap().as_str(), expected_end);
}
}
#[test]
fn test_file_path_regex() {
let path = "/Users/test/019c6024-aef0-7472-89ec-65b62c577c09/prompt_3.txt#L929-L933";
let re = Regex::new(FILE_REGEX).unwrap();
let captures = re.captures(path).unwrap();
assert_eq!(
captures.get(1).unwrap().as_str(),
"/Users/test/019c6024-aef0-7472-89ec-65b62c577c09/prompt_3.txt"
);
assert_eq!(captures.get(2).unwrap().as_str(), "929");
assert_eq!(captures.get(3).unwrap().as_str(), "933");
}
fn file_reference(
path: &str,
start_line: Option<usize>,
end_line: Option<usize>,
) -> Option<FileReference> {
Some(FileReference {
path: PathBuf::from(path),
start_line,
end_line,
})
}
#[test]
fn test_parse_file_references() {
let data: Vec<(&str, Option<FileReference>)> = vec![
("", None),
("@", None),
("@@foo", None),
("foo", file_reference("foo", None, None)),
("@foo", file_reference("foo", None, None)),
(
"@Users/test/bar",
file_reference("Users/test/bar", None, None),
),
(
"@Users/test/bar:1-12",
file_reference("Users/test/bar", Some(1), Some(12)),
),
(
"@/asdf/asdf/asdf/asdf/asdf:L1-12",
file_reference("/asdf/asdf/asdf/asdf/asdf", Some(1), Some(12)),
),
(
"@ssasdf/asdf/dsa/fsda/f/sdf/:L1-L12",
file_reference("ssasdf/asdf/dsa/fsda/f/sdf/", Some(1), Some(12)),
),
(
"/home/user/project/src/main.rs",
file_reference("/home/user/project/src/main.rs", None, None),
),
(
"src/lib.rs:10-20",
file_reference("src/lib.rs", Some(10), Some(20)),
),
(
"@my.project/src/file.test.rs:L100-L200",
file_reference("my.project/src/file.test.rs", Some(100), Some(200)),
),
("@foo.rs:L5-L5", file_reference("foo.rs", Some(5), Some(5))),
];
for (input, expected) in data {
let reference = FileReference::parse(input);
assert_eq!(reference, expected, "Failed for input: {input:?}");
}
}
#[test]
fn test_estimate_tokens() {
assert_eq!(estimate_tokens(""), 0);
assert_eq!(estimate_tokens("abcd"), 1);
assert_eq!(estimate_tokens("abcdefgh"), 2);
assert_eq!(estimate_tokens(&"x".repeat(20_000)), 5_000);
}
fn test_info(suffix: &str) -> String {
format!("test-session-{suffix}")
}
#[tokio::test]
async fn test_render_embedded_resource_large_file_skipped() {
let large_content = "x".repeat(80).repeat(300);
let _info = test_info("large-file");
let resource = EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
agent_client_protocol::TextResourceContents::new(
large_content.clone(),
"file:///project/huge.rs",
),
));
let rendered = render_embedded_resource(&resource, true).await.unwrap();
assert!(rendered.contains("skipped=\"true\""));
assert!(!rendered.contains("xxxxxxxx"));
assert!(rendered.contains("pasted"));
assert!(rendered.contains("huge.rs"));
let hash = content_hash(large_content.as_bytes());
let expected_path = persistence::session_dir(&_info).join(format!("pasted/{hash}-huge.rs"));
assert!(
expected_path.exists(),
"expected {}",
expected_path.display()
);
let _ = std::fs::remove_file(&expected_path);
}
#[tokio::test]
async fn test_render_embedded_resource_small_file_included() {
let _info = test_info("small-file");
let resource = EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
agent_client_protocol::TextResourceContents::new(
"fn main() {}\n",
"file:///project/small.rs",
),
));
let rendered = render_embedded_resource(&resource, true).await.unwrap();
assert!(rendered.contains("fn main()"));
assert!(!rendered.contains("skipped"));
}
#[tokio::test]
async fn test_render_blob_attachment_written_to_disk() {
use base64::{Engine as _, engine::general_purpose};
let _info = test_info("blob-write");
let content = b"fake pdf bytes";
let encoded = general_purpose::STANDARD.encode(content);
let resource = EmbeddedResource::new(EmbeddedResourceResource::BlobResourceContents(
agent_client_protocol::BlobResourceContents::new(encoded.clone(), "file://report.pdf")
.mime_type(Some("application/pdf".to_string())),
));
let rendered = render_embedded_resource(&resource, false).await.unwrap();
assert!(rendered.contains("file_contents"), "got: {rendered}");
assert!(rendered.contains(r#"type="binary""#), "got: {rendered}");
assert!(rendered.contains("report.pdf"), "got: {rendered}");
assert!(rendered.contains("application/pdf"), "got: {rendered}");
assert!(
rendered.contains(&content.len().to_string()),
"got: {rendered}"
);
assert!(
!rendered.contains(&encoded),
"blob leaked into hint: {rendered}"
);
let hash = content_hash(content);
let expected =
persistence::session_dir(&_info).join(format!("attachments/{hash}-report.pdf"));
assert!(
expected.exists(),
"attachment not written to disk at {}",
expected.display()
);
assert_eq!(std::fs::read(&expected).unwrap(), content);
let _ = std::fs::remove_file(&expected);
}
#[tokio::test]
async fn test_render_blob_attachment_bad_base64_returns_none() {
let _info = test_info("blob-bad-b64");
let resource = EmbeddedResource::new(EmbeddedResourceResource::BlobResourceContents(
agent_client_protocol::BlobResourceContents::new(
"!!! not valid base64 !!!",
"file://bad.pdf",
),
));
let result = render_embedded_resource(&resource, false).await;
assert!(
result.is_none(),
"expected None for bad base64, got: {result:?}"
);
}
fn diff_resource(uri: &str, text: &str) -> EmbeddedResource {
EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
agent_client_protocol::TextResourceContents::new(text, uri)
.mime_type(Some("text/x-diff".to_string())),
))
}
#[tokio::test]
async fn test_render_diff_resource_uses_diff_contents_tag() {
let _info = test_info("diff-small");
let resource = diff_resource(
"file:///project/main.rs#L10-L12",
"+ new line\n context\n- old line",
);
let rendered = render_embedded_resource(&resource, true).await.unwrap();
assert!(rendered.contains("<diff_contents"), "got: {rendered}");
assert!(rendered.contains("</diff_contents>"), "got: {rendered}");
assert!(
rendered.contains(r#"path="/project/main.rs""#),
"got: {rendered}"
);
assert!(rendered.contains(r#"startLine="10""#), "got: {rendered}");
assert!(rendered.contains(r#"endLine="12""#), "got: {rendered}");
assert!(rendered.contains("+ new line"), "got: {rendered}");
assert!(rendered.contains("- old line"), "got: {rendered}");
assert!(!rendered.contains("code_selection"), "got: {rendered}");
}
#[tokio::test]
async fn test_render_diff_resource_without_line_range() {
let _info = test_info("diff-no-range");
let resource = diff_resource("file:///project/lib.rs", " unchanged line");
let rendered = render_embedded_resource(&resource, true).await.unwrap();
assert!(rendered.contains("<diff_contents"), "got: {rendered}");
assert!(
rendered.contains(r#"path="/project/lib.rs""#),
"got: {rendered}"
);
assert!(!rendered.contains("startLine"), "got: {rendered}");
assert!(!rendered.contains("endLine"), "got: {rendered}");
}
#[tokio::test]
async fn test_render_diff_resource_large_skipped() {
let large_diff = "x".repeat(80).repeat(300);
let _info = test_info("diff-large");
let resource = diff_resource("file:///project/big.rs#L1-L999", &large_diff);
let rendered = render_embedded_resource(&resource, true).await.unwrap();
assert!(rendered.contains("<diff_contents"), "got: {rendered}");
assert!(rendered.contains("skipped=\"true\""), "got: {rendered}");
assert!(
!rendered.contains("xxxxxxxx"),
"full content leaked: {rendered}"
);
assert!(rendered.contains("pasted"), "got: {rendered}");
let hash = content_hash(large_diff.as_bytes());
let expected = persistence::session_dir(&_info).join(format!("pasted/{hash}-big.rs"));
let _ = std::fs::remove_file(&expected);
}
#[tokio::test]
async fn test_grok_render_embedded_resource_uses_file_contents_tag() {
let _info = test_info("grok-text");
let resource = EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
agent_client_protocol::TextResourceContents::new(
"const x = 1;\nconst y = 2;\n",
"file:///project/app.ts#L5-L6",
),
));
let rendered = render_embedded_resource(&resource, false).await.unwrap();
assert!(rendered.contains("<file_contents"), "got: {rendered}");
assert!(rendered.contains("</file_contents>"), "got: {rendered}");
assert!(rendered.contains(r#"startLine="5""#), "got: {rendered}");
assert!(rendered.contains(r#"endLine="6""#), "got: {rendered}");
assert!(!rendered.contains("code_selection"), "got: {rendered}");
}
#[tokio::test]
async fn test_grok_render_embedded_resource_full_file_uses_is_full_file() {
let _info = test_info("grok-full-file");
let resource = EmbeddedResource::new(EmbeddedResourceResource::TextResourceContents(
agent_client_protocol::TextResourceContents::new(
"fn main() {}\n",
"file:///project/main.rs",
),
));
let rendered = render_embedded_resource(&resource, false).await.unwrap();
assert!(rendered.contains("<file_contents"), "got: {rendered}");
assert!(rendered.contains(r#"isFullFile="true""#), "got: {rendered}");
assert!(!rendered.contains("code_selection"), "got: {rendered}");
}
}
@@ -0,0 +1,775 @@
//! Read-only filesystem helpers backing the client-facing
//! `workspace.client_fs_*` RPCs (the grok.com conversation-files UI,
//! tunneled through the server).
//!
//! Deliberately separate from the shell-facing ext ops in
//! [`ext_fs`](super::ext_fs): every path here is workspace-root-relative
//! and resolves through the root-confinement helper
//! (`WorkspaceHandle::resolve_service_path`), the list walk excludes
//! symlinks that resolve outside the root (and never descends into
//! them), listings paginate with stable post-sort slices, and reads are
//! binary-safe (base64 chunks).
//!
//! Wire types live in `kigi_workspace_types::rpc::fs` (the
//! `ClientFs*` types), shared with the backend caller.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use kigi_workspace_types::rpc::fs::{
ClientFsListNode as FsListNode, ClientFsListReq as FsListReq, ClientFsListRes as FsListRes,
ClientFsReadFileReq as FsReadFileReq, ClientFsReadFileRes as FsReadFileRes,
ClientFsStatReq as FsStatReq, ClientFsStatRes as FsStatRes, FsContentType, FsNodeType,
};
use crate::error::{WorkspaceError, WorkspaceResult};
use crate::handle::WorkspaceHandle;
/// Hard cap on entries collected per list call before sorting (shared
/// across all fs surfaces; see [`super::walk::MAX_LIST_COLLECT`]).
const MAX_LIST_COLLECT: usize = super::walk::MAX_LIST_COLLECT;
/// Server-side cap on `FsListReq::limit`.
const MAX_LIST_LIMIT: u32 = 1000;
/// Server-side cap on a single read's effective byte budget (shared
/// across all fs surfaces; see [`super::walk::MAX_READ_BYTES`]). Only
/// referenced by tests now that the clamp lives in `walk::clamp_read_length`.
#[cfg(test)]
const MAX_READ_BYTES: u64 = super::walk::MAX_READ_BYTES;
/// Bound on memoized hashes; the memo is cleared (not LRU-evicted) when
/// full — entries simply re-hash on next use.
const HASH_MEMO_CAPACITY: usize = 4096;
// =========================================================================
// (path, size, mtime_ms) → hash memo
// =========================================================================
#[derive(Debug, Clone)]
struct MemoEntry {
size: u64,
mtime_ms: i64,
hash: String,
}
/// Memo of full-content SHA-256 digests keyed by absolute path and
/// validated against `(size, mtime_ms)`, so unchanged files hash once
/// instead of on every `client_fs_stat`. The memo only avoids redundant
/// hashing — it never substitutes mtime for content addressing: a
/// `(size, mtime_ms)` mismatch is a miss and the caller re-hashes.
#[derive(Debug, Default)]
pub(crate) struct FileHashMemo {
entries: parking_lot::Mutex<HashMap<PathBuf, MemoEntry>>,
}
impl FileHashMemo {
/// Return the memoized hash when `(size, mtime_ms)` still match.
pub(crate) fn lookup(&self, path: &Path, size: u64, mtime_ms: i64) -> Option<String> {
let entries = self.entries.lock();
let entry = entries.get(path)?;
(entry.size == size && entry.mtime_ms == mtime_ms).then(|| entry.hash.clone())
}
/// Record a freshly computed hash, replacing any stale entry for the
/// same path. Clears the whole memo when inserting a new path would
/// exceed [`HASH_MEMO_CAPACITY`].
pub(crate) fn store(&self, path: &Path, size: u64, mtime_ms: i64, hash: String) {
let mut entries = self.entries.lock();
if !entries.contains_key(path) && entries.len() >= HASH_MEMO_CAPACITY {
entries.clear();
}
entries.insert(
path.to_path_buf(),
MemoEntry {
size,
mtime_ms,
hash,
},
);
}
}
// =========================================================================
// Path resolution
// =========================================================================
/// Resolve a root-relative request path through the workspace's
/// root-confinement helper, returning the resolved path together with the
/// canonical root it was checked against. `""` and `"."` mean the
/// workspace root; absolute paths, `..` escapes, and symlink escapes are
/// rejected there.
async fn resolve_with_root(
ws: &WorkspaceHandle,
path: &str,
) -> WorkspaceResult<(PathBuf, PathBuf)> {
let rel = if path.is_empty() { "." } else { path };
let canonical_root = ws.canonical_root().await?;
let abs = ws.resolve_service_path(rel, &canonical_root).await?;
Ok((abs, canonical_root))
}
/// [`resolve_with_root`] for callers that don't need the canonical root.
async fn resolve(ws: &WorkspaceHandle, path: &str) -> WorkspaceResult<PathBuf> {
resolve_with_root(ws, path).await.map(|(abs, _)| abs)
}
fn system_time_ms(st: std::time::SystemTime) -> i64 {
match st.duration_since(std::time::UNIX_EPOCH) {
Ok(d) => i64::try_from(d.as_millis()).unwrap_or(i64::MAX),
Err(e) => -i64::try_from(e.duration().as_millis()).unwrap_or(i64::MAX),
}
}
// =========================================================================
// list
// =========================================================================
/// List `req.path` (workspace-root-relative) with stable pagination:
/// collect the full walk (bounded by [`MAX_LIST_COLLECT`]), sort
/// directories-first / case-insensitive by name, then slice
/// `[offset, offset + limit)`. Symlinks resolving outside the workspace
/// root are excluded from the walk (and never descended into).
pub(crate) async fn list(ws: &WorkspaceHandle, req: &FsListReq) -> WorkspaceResult<FsListRes> {
let (abs, canonical_root) = resolve_with_root(ws, &req.path).await?;
let root = ws.root_cwd()?;
let req = req.clone();
// The walk does synchronous traversal + metadata syscalls; run it off
// the async executor (matching the ext_fs ops).
tokio::task::spawn_blocking(move || {
list_blocking(&abs, &root, &canonical_root, &req, MAX_LIST_COLLECT)
})
.await
.map_err(|e| WorkspaceError::JoinError(e.to_string()))?
}
fn list_blocking(
abs_dir: &Path,
root: &Path,
canonical_root: &Path,
req: &FsListReq,
max_collect: usize,
) -> WorkspaceResult<FsListRes> {
// Root confinement also holds mid-walk: a symlink inside the root
// pointing outside must not enumerate outside metadata.
let page = super::walk::list_directory_paged(
abs_dir,
super::walk::ListOptions {
depth: req.depth as usize,
follow_symlinks: req.follow_symlinks,
respect_git_ignore: req.respect_git_ignore,
include_hidden: req.include_hidden,
include_globs: &req.include_globs,
exclude_globs: &req.exclude_globs,
offset: req.offset,
limit: req.limit.min(MAX_LIST_LIMIT) as usize,
confine_to_canonical_root: Some(canonical_root.to_path_buf()),
},
max_collect,
);
let nodes: Vec<FsListNode> = page
.entries
.into_iter()
.map(|e| FsListNode {
node_type: if e.is_dir {
FsNodeType::Directory
} else {
FsNodeType::File
},
size: e.size,
mtime_ms: e.modified.map(system_time_ms),
is_symlink: e.is_symlink.then_some(true),
// Root-relative path (divergent from the shell's absolute path).
// A walk under a symlinked root yields canonical-root-spelled
// entries, so strip either spelling.
path: e
.abs_path
.strip_prefix(root)
.or_else(|_| e.abs_path.strip_prefix(canonical_root))
.unwrap_or(&e.abs_path)
.to_string_lossy()
.into_owned(),
name: e.name,
})
.collect();
Ok(FsListRes {
nodes,
truncated: page.truncated,
})
}
// =========================================================================
// stat
// =========================================================================
/// Stat `req.path`: existence, kind, size, mtime, and — for files — a
/// full-content SHA-256 served through the workspace hash memo.
pub(crate) async fn stat(ws: &WorkspaceHandle, req: &FsStatReq) -> WorkspaceResult<FsStatRes> {
let abs = resolve(ws, &req.path).await?;
let md = match tokio::fs::metadata(&abs).await {
Ok(md) => md,
// NotADirectory: a *file* sits mid-path (e.g. `a.txt/sub`) — for an
// existence probe that is a miss, not an RPC error.
Err(e)
if e.kind() == std::io::ErrorKind::NotFound
|| e.kind() == std::io::ErrorKind::NotADirectory =>
{
return Ok(FsStatRes {
exists: false,
node_type: None,
size: None,
mtime_ms: None,
hash: None,
});
}
Err(e) => {
return Err(WorkspaceError::HubError(format!(
"stat failed for {}: {e}",
req.path
)));
}
};
let mtime_ms = md.modified().ok().map(system_time_ms);
if md.is_dir() {
return Ok(FsStatRes {
exists: true,
node_type: Some(FsNodeType::Directory),
size: None,
mtime_ms,
hash: None,
});
}
let size = md.len();
let memo = &ws.shared.client_fs_hash_memo;
let hash = match mtime_ms.and_then(|m| memo.lookup(&abs, size, m)) {
Some(hash) => hash,
None => {
let (hash, _, _) = crate::handle::stream_hash_and_range(&abs, 0, 0)
.await
.map_err(|e| {
WorkspaceError::HubError(format!("hash failed for {}: {e}", req.path))
})?;
if let Some(m) = mtime_ms {
memo.store(&abs, size, m, hash.clone());
}
hash
}
};
Ok(FsStatRes {
exists: true,
node_type: Some(FsNodeType::File),
size: Some(size),
mtime_ms,
hash: Some(hash),
})
}
// =========================================================================
// read_file
// =========================================================================
/// Read a byte range of `req.path` (binary-safe, capped at
/// `min(req.max_bytes, MAX_READ_BYTES)`) together with the full-file
/// SHA-256. When the hash is memoized for the current `(size, mtime)`
/// only the requested range is read; otherwise the whole file streams
/// once (via the shared [`crate::handle::stream_hash_and_range`]) to
/// hash it.
pub(crate) async fn read_file(
ws: &WorkspaceHandle,
req: &FsReadFileReq,
) -> WorkspaceResult<FsReadFileRes> {
let abs = resolve(ws, &req.path).await?;
let read_err =
|e: std::io::Error| WorkspaceError::HubError(format!("read failed for {}: {e}", req.path));
let md = tokio::fs::metadata(&abs).await.map_err(read_err)?;
if md.is_dir() {
return Err(WorkspaceError::HubError(format!(
"not a file: {}",
req.path
)));
}
let size = md.len();
let mtime_ms = md.modified().ok().map(system_time_ms);
let offset = req.offset.unwrap_or(0);
// Server-side clamp: a hostile/buggy caller cannot lift the per-chunk
// budget past MAX_READ_BYTES regardless of `maxBytes`.
let length = super::walk::clamp_read_length(req.length, req.max_bytes);
let memo = &ws.shared.client_fs_hash_memo;
let (hash, chunk, size) = match mtime_ms.and_then(|m| memo.lookup(&abs, size, m)) {
Some(hash) => {
let chunk = super::walk::read_range(&abs, offset, length)
.await
.map_err(read_err)?;
(hash, chunk, size)
}
None => {
let (hash, chunk, streamed) =
crate::handle::stream_hash_and_range(&abs, offset, length)
.await
.map_err(read_err)?;
if let Some(m) = mtime_ms {
memo.store(&abs, streamed, m, hash.clone());
}
(hash, chunk, streamed)
}
};
// Shared encoder keeps the paired wire fields coherent with one UTF-8
// validation pass; `type` is text iff the bytes were valid UTF-8.
let (payload, is_text) = super::walk::encode_chunk(chunk, req.encoding);
let (content, content_base64) = match payload {
super::walk::ChunkPayload::Text(t) => (Some(t), None),
super::walk::ChunkPayload::Base64(b) => (None, Some(b)),
};
let content_type = if is_text {
FsContentType::Text
} else {
FsContentType::Binary
};
Ok(FsReadFileRes {
content,
content_base64,
size,
hash,
content_type,
})
}
#[cfg(test)]
mod tests {
use base64::Engine;
use kigi_workspace_types::rpc::fs::FsReadEncoding;
use super::*;
use crate::handle::tests::make_handle;
fn list_req(path: &str) -> FsListReq {
FsListReq {
path: path.to_owned(),
depth: 1,
include_hidden: true,
limit: 1000,
offset: 0,
follow_symlinks: true,
respect_git_ignore: false,
include_globs: vec![],
exclude_globs: vec![],
}
}
/// Fixture: root with files `b.txt`, `A.txt`, `c.txt` and dirs
/// `Zeta`, `alpha`. Expected order: dirs first case-insensitive
/// (`alpha`, `Zeta`), then files (`A.txt`, `b.txt`, `c.txt`).
fn populate(root: &Path) {
std::fs::write(root.join("b.txt"), b"bb").unwrap();
std::fs::write(root.join("A.txt"), b"a").unwrap();
std::fs::write(root.join("c.txt"), b"ccc").unwrap();
std::fs::create_dir(root.join("Zeta")).unwrap();
std::fs::create_dir(root.join("alpha")).unwrap();
}
/// `list_blocking` against `dir` as both walk root and workspace root
/// (canonicalized for the confinement check, like production).
fn list_dir(dir: &Path, req: &FsListReq, max_collect: usize) -> FsListRes {
let canonical = dunce::canonicalize(dir).unwrap();
list_blocking(dir, dir, &canonical, req, max_collect).unwrap()
}
#[test]
fn list_sorts_dirs_first_case_insensitive() {
let dir = tempfile::tempdir().unwrap();
populate(dir.path());
let res = list_dir(dir.path(), &list_req(""), MAX_LIST_COLLECT);
let names: Vec<&str> = res.nodes.iter().map(|n| n.name.as_str()).collect();
assert_eq!(names, ["alpha", "Zeta", "A.txt", "b.txt", "c.txt"]);
assert!(!res.truncated);
assert_eq!(res.nodes[0].node_type, FsNodeType::Directory);
assert_eq!(res.nodes[2].node_type, FsNodeType::File);
assert_eq!(res.nodes[2].size, Some(1));
assert!(res.nodes[2].mtime_ms.is_some());
// Paths are workspace-root-relative.
assert_eq!(res.nodes[2].path, "A.txt");
}
/// Pagination slices the *sorted* listing, so consecutive pages have
/// stable boundaries and concatenate to the full listing.
#[test]
fn list_paginates_post_sort_with_stable_boundaries() {
let dir = tempfile::tempdir().unwrap();
populate(dir.path());
let full = list_dir(dir.path(), &list_req(""), MAX_LIST_COLLECT);
let mut paged = Vec::new();
for page_start in [0u64, 2, 4] {
let req = FsListReq {
limit: 2,
offset: page_start,
..list_req("")
};
let page = list_dir(dir.path(), &req, MAX_LIST_COLLECT);
// truncated while more entries remain past this page.
assert_eq!(page.truncated, page_start + 2 < full.nodes.len() as u64);
paged.extend(page.nodes);
}
assert_eq!(paged, full.nodes);
// Offset past the end yields an empty, non-truncated page.
let req = FsListReq {
offset: 100,
..list_req("")
};
let page = list_dir(dir.path(), &req, MAX_LIST_COLLECT);
assert!(page.nodes.is_empty());
assert!(!page.truncated);
}
/// The collection cap marks the result truncated even when the page
/// itself is not full.
#[test]
fn list_collection_cap_truncates() {
let dir = tempfile::tempdir().unwrap();
populate(dir.path());
let res = list_dir(dir.path(), &list_req(""), 2);
assert_eq!(res.nodes.len(), 2);
assert!(res.truncated);
}
#[test]
fn list_caps_limit_at_server_max() {
let dir = tempfile::tempdir().unwrap();
populate(dir.path());
let req = FsListReq {
limit: u32::MAX,
..list_req("")
};
// Must not panic / overflow; the page is everything (< 1000).
let res = list_dir(dir.path(), &req, MAX_LIST_COLLECT);
assert_eq!(res.nodes.len(), 5);
}
/// Regression: the list walk must not traverse — or
/// even surface — in-root symlinks that resolve outside the workspace
/// root, while symlinks staying inside the root keep working.
#[test]
#[cfg(unix)]
fn list_excludes_symlink_escapes_mid_walk() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let outside = tempfile::tempdir().unwrap();
std::fs::write(outside.path().join("secret.txt"), b"secret").unwrap();
// Escaping symlink: root/escape_link -> <outside>.
std::os::unix::fs::symlink(outside.path(), root.join("escape_link")).unwrap();
// In-root symlink: root/good_link -> root/real_dir.
std::fs::create_dir(root.join("real_dir")).unwrap();
std::fs::write(root.join("real_dir/inner.txt"), b"inner").unwrap();
std::os::unix::fs::symlink(root.join("real_dir"), root.join("good_link")).unwrap();
let req = FsListReq {
depth: 2,
follow_symlinks: true,
..list_req("")
};
let res = list_dir(root, &req, MAX_LIST_COLLECT);
let paths: Vec<&str> = res.nodes.iter().map(|n| n.path.as_str()).collect();
assert!(
!paths.iter().any(|p| p.contains("escape_link")),
"escaping symlink (and its subtree) must be excluded: {paths:?}"
);
assert!(
!paths.iter().any(|p| p.contains("secret.txt")),
"outside entries must not be enumerated: {paths:?}"
);
// Confinement must not over-filter: in-root symlinks survive,
// including descent through them.
assert!(paths.contains(&"good_link"), "{paths:?}");
assert!(paths.contains(&"good_link/inner.txt"), "{paths:?}");
assert!(paths.contains(&"real_dir/inner.txt"), "{paths:?}");
let good = res.nodes.iter().find(|n| n.path == "good_link").unwrap();
assert_eq!(good.is_symlink, Some(true));
}
#[test]
fn memo_lookup_hits_and_invalidates_on_mismatch() {
let memo = FileHashMemo::default();
let path = Path::new("/ws/a.txt");
memo.store(path, 10, 1000, "h1".into());
assert_eq!(memo.lookup(path, 10, 1000).as_deref(), Some("h1"));
// Size change ⇒ miss.
assert_eq!(memo.lookup(path, 11, 1000), None);
// Mtime change ⇒ miss.
assert_eq!(memo.lookup(path, 10, 2000), None);
// Re-store replaces the stale entry.
memo.store(path, 11, 2000, "h2".into());
assert_eq!(memo.lookup(path, 11, 2000).as_deref(), Some("h2"));
assert_eq!(memo.lookup(path, 10, 1000), None);
}
/// `stat` consults the memo (no re-hash for an unchanged file) and
/// recomputes when `(size, mtime)` no longer match.
#[tokio::test]
async fn stat_uses_memo_until_file_changes() {
let ws = make_handle();
let root = ws.root_cwd().unwrap();
std::fs::write(root.join("data.txt"), b"hello world").unwrap();
let req = FsStatReq {
path: "data.txt".into(),
};
let first = stat(&ws, &req).await.unwrap();
assert!(first.exists);
assert_eq!(first.node_type, Some(FsNodeType::File));
assert_eq!(first.size, Some(11));
let real_hash = first.hash.clone().expect("hash for files");
// Plant a sentinel hash for the file's current (size, mtime). A
// second stat must return the sentinel — proof it did not re-hash.
let abs = root.join("data.txt");
let md = std::fs::metadata(&abs).unwrap();
let mtime = system_time_ms(md.modified().unwrap());
ws.shared
.client_fs_hash_memo
.store(&abs, md.len(), mtime, "sentinel".into());
let memoized = stat(&ws, &req).await.unwrap();
assert_eq!(memoized.hash.as_deref(), Some("sentinel"));
// A size change invalidates the memo entry and re-hashes.
std::fs::write(&abs, b"hello brave new world").unwrap();
let rehashed = stat(&ws, &req).await.unwrap();
let new_hash = rehashed.hash.expect("hash for files");
assert_ne!(new_hash, "sentinel");
assert_ne!(new_hash, real_hash);
}
/// A path with a *file* as an intermediate component (`ENOTDIR`) is an
/// existence miss, not an RPC error.
#[tokio::test]
async fn stat_enotdir_intermediate_reports_not_exists() {
let ws = make_handle();
let root = ws.root_cwd().unwrap();
std::fs::write(root.join("file.txt"), b"x").unwrap();
let res = stat(
&ws,
&FsStatReq {
path: "file.txt/nested".into(),
},
)
.await
.unwrap();
assert!(!res.exists);
assert_eq!(res.node_type, None);
assert_eq!(res.hash, None);
}
#[tokio::test]
async fn stat_missing_path_reports_not_exists() {
let ws = make_handle();
let res = stat(
&ws,
&FsStatReq {
path: "nope.txt".into(),
},
)
.await
.unwrap();
assert!(!res.exists);
assert_eq!(res.node_type, None);
assert_eq!(res.hash, None);
}
#[tokio::test]
async fn read_file_chunks_are_binary_safe_and_capped() {
let ws = make_handle();
let root = ws.root_cwd().unwrap();
// Non-UTF-8 payload: every byte value once.
let payload: Vec<u8> = (0u8..=255).collect();
std::fs::write(root.join("blob.bin"), &payload).unwrap();
let req = FsReadFileReq {
path: "blob.bin".into(),
// Bytes 200..210 are bare continuation bytes — never valid UTF-8.
offset: Some(200),
length: Some(50),
max_bytes: 10, // cap below the requested length
encoding: FsReadEncoding::Base64,
};
let res = read_file(&ws, &req).await.unwrap();
assert_eq!(res.size, 256);
assert_eq!(res.content, None);
assert_eq!(res.content_type, FsContentType::Binary);
let bytes = base64::engine::general_purpose::STANDARD
.decode(res.content_base64.unwrap())
.unwrap();
assert_eq!(bytes, payload[200..210], "maxBytes caps the chunk");
// Full-file hash regardless of the requested range.
use sha2::{Digest, Sha256};
assert_eq!(res.hash, format!("{:x}", Sha256::digest(&payload)));
// Memoized second read (range-only fast path) returns the
// identical chunk + hash.
let again = read_file(&ws, &req).await.unwrap();
assert_eq!(again.hash, res.hash);
let again_bytes = base64::engine::general_purpose::STANDARD
.decode(again.content_base64.unwrap())
.unwrap();
assert_eq!(again_bytes, payload[200..210]);
}
/// `maxBytes` is server-capped at [`MAX_READ_BYTES`]:
/// a caller-supplied huge budget cannot make the workspace buffer the
/// whole file.
#[tokio::test]
async fn read_file_server_caps_max_bytes() {
let ws = make_handle();
let root = ws.root_cwd().unwrap();
let payload = vec![0u8; (MAX_READ_BYTES + 100) as usize];
std::fs::write(root.join("big.bin"), &payload).unwrap();
let res = read_file(
&ws,
&FsReadFileReq {
path: "big.bin".into(),
offset: None,
length: None,
max_bytes: u64::MAX,
encoding: FsReadEncoding::Base64,
},
)
.await
.unwrap();
assert_eq!(res.size, payload.len() as u64);
let bytes = base64::engine::general_purpose::STANDARD
.decode(res.content_base64.unwrap())
.unwrap();
assert_eq!(
bytes.len() as u64,
MAX_READ_BYTES,
"clamped to the server cap"
);
}
#[tokio::test]
async fn read_file_utf8_default_and_binary_fallback() {
let ws = make_handle();
let root = ws.root_cwd().unwrap();
std::fs::write(root.join("text.txt"), "héllo").unwrap();
std::fs::write(root.join("bin.dat"), [0xff, 0xfe, 0x00]).unwrap();
let text = read_file(
&ws,
&FsReadFileReq {
path: "text.txt".into(),
offset: None,
length: None,
max_bytes: 1_048_576,
encoding: FsReadEncoding::Utf8,
},
)
.await
.unwrap();
assert_eq!(text.content.as_deref(), Some("héllo"));
assert_eq!(text.content_base64, None);
assert_eq!(text.content_type, FsContentType::Text);
// Invalid UTF-8 under the utf8 default degrades to base64.
let bin = read_file(
&ws,
&FsReadFileReq {
path: "bin.dat".into(),
offset: None,
length: None,
max_bytes: 1_048_576,
encoding: FsReadEncoding::Utf8,
},
)
.await
.unwrap();
assert_eq!(bin.content, None);
assert_eq!(bin.content_type, FsContentType::Binary);
let bytes = base64::engine::general_purpose::STANDARD
.decode(bin.content_base64.unwrap())
.unwrap();
assert_eq!(bytes, [0xff, 0xfe, 0x00]);
}
#[tokio::test]
async fn resolve_rejects_escapes() {
let ws = make_handle();
for path in ["/etc/passwd", "../escape.txt"] {
let err = stat(
&ws,
&FsStatReq {
path: path.to_owned(),
},
)
.await
.expect_err("escape must be rejected");
assert!(matches!(err, WorkspaceError::HubError(_)), "{err:?}");
}
}
/// An absolute path *inside* the workspace root is accepted and stats
/// the same file as its root-relative form.
#[tokio::test]
async fn resolve_accepts_absolute_within_root() {
let ws = make_handle();
let root = ws.root_cwd().unwrap();
std::fs::write(root.join("data.txt"), b"hello").unwrap();
let rel = stat(
&ws,
&FsStatReq {
path: "data.txt".into(),
},
)
.await
.unwrap();
assert!(rel.exists);
let abs_path = root.join("data.txt").to_string_lossy().into_owned();
let abs = stat(&ws, &FsStatReq { path: abs_path }).await.unwrap();
assert!(abs.exists);
assert_eq!(abs.node_type, rel.node_type);
assert_eq!(abs.size, rel.size);
assert_eq!(abs.hash, rel.hash);
}
#[tokio::test]
#[cfg(unix)]
async fn resolve_rejects_symlink_escape() {
let ws = make_handle();
let root = ws.root_cwd().unwrap();
let outside = tempfile::tempdir().unwrap();
std::fs::write(outside.path().join("secret.txt"), b"secret").unwrap();
std::os::unix::fs::symlink(outside.path(), root.join("escape_link")).unwrap();
let err = read_file(
&ws,
&FsReadFileReq {
path: "escape_link/secret.txt".into(),
offset: None,
length: None,
max_bytes: 1_048_576,
encoding: FsReadEncoding::Base64,
},
)
.await
.expect_err("symlink escape must be rejected");
assert!(
err.to_string().contains("symlink escape"),
"unexpected error: {err}"
);
}
#[tokio::test]
async fn list_empty_path_lists_root() {
let ws = make_handle();
let root = ws.root_cwd().unwrap();
std::fs::write(root.join("rooted.txt"), b"x").unwrap();
let res = list(&ws, &list_req("")).await.unwrap();
assert!(res.nodes.iter().any(|n| n.name == "rooted.txt"));
}
}
@@ -0,0 +1,306 @@
//! Codebase Index Manager
//!
//! Manages code graph indexes for code navigation features (go-to-definition, go-to-references).
//! Indexes are shared across sessions with the same cwd to avoid duplicate work.
//!
//! ## Deduplication
//!
//! Deduplication happens at two levels:
//! 1. **Process-level**: `IndexManager::spawn()` ensures at most one manager per workspace per process
//! 2. **Cross-process**: File-based locking prevents duplicate background operations
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Weak};
use kigi_codebase_graph::{IndexManager, IndexManagerConfig, IndexManagerHandle};
use kigi_tools::util::kigi_home::kigi_home;
/// Get the cache path for a cwd's index.
///
/// Cache is stored in: `~/.kigi/indexes/{url_encoded_cwd}/goto_index.bin`
pub fn get_index_cache_path(cwd: &Path) -> PathBuf {
let encoded = urlencoding::encode(&cwd.to_string_lossy()).into_owned();
kigi_home()
.join("indexes")
.join(encoded)
.join("goto_index.bin")
}
/// Manages code graph indexes across sessions.
///
/// Wraps `IndexManager::spawn()` with cache-path config and cross-session
/// handle reuse. Keeps only `Weak` refs — sessions hold the strong `Arc`s,
/// so the actor is reaped when the last session in a git-root closes.
pub struct CodebaseIndexManager {
indexes: HashMap<PathBuf, Weak<IndexManagerHandle>>,
}
impl Default for CodebaseIndexManager {
fn default() -> Self {
Self::new()
}
}
impl CodebaseIndexManager {
pub fn new() -> Self {
Self {
indexes: HashMap::new(),
}
}
/// Get or create the index for `cwd`. Returns `(handle, was_newly_spawned)`.
/// Caller must hold the `Arc` to keep the index alive.
pub fn get_or_create(&mut self, cwd: PathBuf) -> (Arc<IndexManagerHandle>, bool) {
self.indexes.retain(|_, weak| weak.strong_count() > 0);
if let Some(handle) = self.indexes.get(&cwd).and_then(Weak::upgrade) {
tracing::info!(
cwd = %cwd.display(),
event = "index_reused",
"Codebase index already running — reusing shared handle"
);
return (handle, false);
}
// Create cache directory if needed
let cache_path = get_index_cache_path(&cwd);
if let Some(parent) = cache_path.parent()
&& let Err(e) = std::fs::create_dir_all(parent)
{
tracing::warn!(
path = %parent.display(),
error = %e,
"Failed to create cache directory"
);
}
tracing::info!(
cwd = %cwd.display(),
cache_path = %cache_path.display(),
event = "index_lazy_started",
"Codebase index lazy-started — spawning IndexManager actor"
);
// IndexManager::spawn() handles global deduplication.
let config = IndexManagerConfig::new(cwd.clone()).with_cache_path(cache_path);
let handle = IndexManager::spawn(config);
self.indexes.insert(cwd, Arc::downgrade(&handle));
(handle, true)
}
/// Get the running index for `cwd`, or `None` if not started / already reaped.
pub fn get(&self, cwd: &Path) -> Option<Arc<IndexManagerHandle>> {
self.indexes.get(cwd).and_then(Weak::upgrade)
}
/// Returns the number of currently-live indexes (test helper).
#[cfg(test)]
pub(crate) fn active_count(&self) -> usize {
self.indexes
.values()
.filter(|weak| weak.strong_count() > 0)
.count()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_cache_path_encoding() {
let cwd = Path::new("/Users/test/my project");
let cache_path = get_index_cache_path(cwd);
// Should contain URL-encoded path
assert!(cache_path.to_string_lossy().contains("%2F"));
assert!(cache_path.to_string_lossy().ends_with("goto_index.bin"));
}
// =========================================================================
// Lazy-start mechanic tests
//
// These tests verify the core lazy-start behavior:
// `CodebaseIndexManager::get()` returns None before the index is created,
// which maps to `x.ai/code/status` reporting `reason: notStarted`.
// `get_or_create()` is the lazy-start entry point called by
// `MvpAgent::start_codebase_index_for_code_nav` on the first code-nav
// request for an eligible session.
// =========================================================================
/// An empty CodebaseIndexManager returns None for any path.
///
/// This is the steady-state before ANY code-nav request has been made.
/// In `x.ai/code/status`, `resolve_index_handle()` calls
/// `agent.get_codebase_index(cwd)` which calls `mgr.get(cwd)`.
/// When this returns None the status reports `reason: notStarted` — the
/// key non-starting guarantee from the plan.
#[test]
fn test_get_returns_none_before_any_index_created() {
let mgr = CodebaseIndexManager::new();
assert!(
mgr.get(Path::new("/some/repo")).is_none(),
"get() must return None before get_or_create() is called — this is \
the CodebaseIndexManager state that causes code/status to report notStarted"
);
}
/// get() returns None even after another path was indexed.
///
/// Proves path-scoped isolation: a different cwd's index does not
/// satisfy a lookup for an unrelated path.
#[test]
fn test_get_returns_none_for_different_cwd() {
let temp = tempfile::tempdir().unwrap();
let root_a = temp.path().join("repo_a");
let root_b = temp.path().join("repo_b");
std::fs::create_dir_all(&root_a).unwrap();
let mut mgr = CodebaseIndexManager::new();
let (_handle, _) = mgr.get_or_create(root_a.clone());
// root_b was never indexed — must return None.
assert!(
mgr.get(&root_b).is_none(),
"get() for an un-indexed path must return None (path-scoped isolation)"
);
}
/// After get_or_create() the same path is found by get().
///
/// This is the lazy-start path: `get_or_create` is called by
/// `start_codebase_index_for_code_nav` on the first eligible code-nav
/// request, after which `get_codebase_index` (used by `resolve_index_handle`
/// in `code_status`) returns `Some`.
#[test]
fn test_get_finds_handle_after_get_or_create() {
let temp = tempfile::tempdir().unwrap();
let root = temp.path().to_path_buf();
let mut mgr = CodebaseIndexManager::new();
// Before lazy-start: get() returns None.
assert!(mgr.get(&root).is_none(), "no index before get_or_create");
// Lazy-start: this is what start_codebase_index_for_code_nav calls.
let (handle, was_new) = mgr.get_or_create(root.clone());
assert!(was_new, "first get_or_create must report newly spawned");
assert_eq!(mgr.active_count(), 1);
let found = mgr
.get(&root)
.expect("index must be visible after get_or_create");
assert!(
Arc::ptr_eq(&handle, &found),
"get() must return the same Arc as get_or_create() — dedup / reuse"
);
}
/// The index is reaped once the last strong handle drops.
///
/// This is the eviction guarantee: the manager holds only a `Weak`, so when
/// the last session pinning a git-root is torn down (here simulated by
/// dropping the sole `Arc`), `get()` stops returning the handle and the
/// entry no longer counts as active — no per-repo accumulation in a
/// long-lived leader process.
#[test]
fn test_index_evicted_when_last_strong_ref_dropped() {
let temp = tempfile::tempdir().unwrap();
let root = temp.path().to_path_buf();
let mut mgr = CodebaseIndexManager::new();
let (handle, _) = mgr.get_or_create(root.clone());
assert_eq!(mgr.active_count(), 1, "index live while strong ref held");
assert!(mgr.get(&root).is_some());
// The agent (here: this test) was the only strong owner.
drop(handle);
assert!(
mgr.get(&root).is_none(),
"index must be released once the last strong ref drops"
);
assert_eq!(
mgr.active_count(),
0,
"a reaped index must not count as active"
);
}
/// Shared index across sessions: reaped only after the last strong ref drops.
#[test]
fn test_shared_index_evicted_only_after_last_session_drops() {
let temp = tempfile::tempdir().unwrap();
let root = temp.path().to_path_buf();
let mut mgr = CodebaseIndexManager::new();
let (session_a, was_new_a) = mgr.get_or_create(root.clone());
let (session_b, was_new_b) = mgr.get_or_create(root.clone());
assert!(was_new_a, "first session must spawn the index");
assert!(!was_new_b, "second session must reuse the shared index");
assert!(
Arc::ptr_eq(&session_a, &session_b),
"both sessions must share one index handle (Arc::ptr_eq)"
);
assert_eq!(
mgr.active_count(),
1,
"two sessions in one git-root back exactly one index"
);
// First session ends: the index stays warm for the surviving session.
drop(session_a);
assert!(
mgr.get(&root).is_some(),
"index must survive while another session still pins it"
);
assert_eq!(
mgr.active_count(),
1,
"index still live after only the first session drops"
);
// Last session ends: now — and only now — the index is reaped.
drop(session_b);
assert!(
mgr.get(&root).is_none(),
"index must be released once the LAST session drops its ref"
);
assert_eq!(
mgr.active_count(),
0,
"a reaped shared index must not count as active"
);
}
/// A second get_or_create() for the same path reuses the existing handle.
///
/// This corresponds to "subsequent code-nav requests reuse the same shared
/// handle" from the plan: once the index is running, all subsequent
/// get_or_create calls for the same path return the existing Arc.
#[test]
fn test_get_or_create_is_idempotent_for_same_path() {
let temp = tempfile::tempdir().unwrap();
let root = temp.path().to_path_buf();
let mut mgr = CodebaseIndexManager::new();
let (h1, was_new_1) = mgr.get_or_create(root.clone());
let (h2, was_new_2) = mgr.get_or_create(root.clone());
assert!(was_new_1, "first call must report newly spawned");
assert!(
!was_new_2,
"second call must report reused (not newly spawned)"
);
assert!(
Arc::ptr_eq(&h1, &h2),
"second get_or_create must return the same Arc as the first (index reuse)"
);
assert_eq!(mgr.active_count(), 1, "only one index for one path");
}
}
@@ -0,0 +1,271 @@
use std::path::Path;
use std::process::Stdio;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Instant;
use tokio::io::{AsyncBufReadExt, BufReader};
use tokio::process::Command;
// Canonical in kigi-workspace-types; re-exported for existing paths.
pub use kigi_workspace_types::rpc::search::{ContentMatch, ContentMatchFile, ContentSearchData};
#[derive(Debug, Clone, Default)]
pub struct ContentSearchParams {
pub pattern: String,
pub case_insensitive: bool,
pub literal: bool,
pub globs: Vec<String>,
pub max_files: Option<usize>,
pub max_matches: Option<usize>,
pub respect_gitignore: bool,
}
/// Batch of results sent during streaming search.
#[derive(Debug, Clone, Default)]
pub struct ContentSearchBatch {
pub files: Vec<ContentMatchFile>,
pub total_matches: usize,
pub total_files: usize,
pub done: bool,
pub truncated: bool,
}
const BATCH_INTERVAL_MS: u64 = 50;
const DEFAULT_MAX_FILES: usize = 100;
const DEFAULT_MAX_MATCHES: usize = 1000;
fn build_ripgrep_command(root: &Path, params: &ContentSearchParams) -> Command {
let rg_path = crate::util::ripgrep::rg_path();
let mut cmd = Command::new(&rg_path);
cmd.current_dir(root);
cmd.stdout(Stdio::piped());
cmd.stderr(Stdio::null());
cmd.stdin(Stdio::null());
kigi_tools::util::detach_command(&mut cmd);
cmd.arg("--json");
cmd.arg("--line-number");
const DEFAULT_EXCLUSIONS: &[&str] = &["!.git/**", "!submodules/**", "!vendor/**"];
for glob in DEFAULT_EXCLUSIONS {
cmd.arg("--glob").arg(glob);
}
cmd.arg("--max-filesize").arg("1M");
cmd.arg("--max-count").arg("50");
cmd.arg("--max-columns").arg("500");
cmd.arg("--max-columns-preview");
if params.case_insensitive {
cmd.arg("--ignore-case");
}
if params.literal {
cmd.arg("--fixed-strings");
}
if !params.respect_gitignore {
cmd.arg("--no-ignore");
}
for glob in &params.globs {
cmd.arg("--glob").arg(glob);
}
cmd.arg("-e").arg(&params.pattern);
cmd.arg(".");
cmd
}
fn extract_match_positions(data: &serde_json::Value) -> (Option<usize>, Option<usize>) {
data.get("submatches")
.and_then(|s| s.as_array())
.and_then(|arr| arr.first())
.map(|first| {
let start = first
.get("start")
.and_then(|s| s.as_u64())
.map(|s| s as usize);
let end = first
.get("end")
.and_then(|e| e.as_u64())
.map(|e| e as usize);
(start, end)
})
.unwrap_or((None, None))
}
fn parse_match_from_json(data: &serde_json::Value) -> Option<ContentMatch> {
let line_number = data.get("line_number").and_then(|l| l.as_u64())? as usize;
let content = data
.get("lines")
.and_then(|l| l.get("text"))
.and_then(|t| t.as_str())
.unwrap_or("")
.trim_end_matches('\n')
.to_string();
let (match_start, match_end) = extract_match_positions(data);
Some(ContentMatch {
line: line_number,
content,
match_start,
match_end,
})
}
fn parse_file_path_from_json(root: &Path, json: &serde_json::Value) -> Option<String> {
let path = json
.get("data")
.and_then(|d| d.get("path"))
.and_then(|p| p.get("text"))
.and_then(|t| t.as_str())?;
let normalized = path.strip_prefix("./").unwrap_or(path);
if Path::new(normalized).is_absolute() {
return Some(normalized.to_string());
}
Some(root.join(normalized).to_string_lossy().to_string())
}
/// Streaming content search with batched status notifications.
/// Set `cancel` to true to abort the search early.
pub async fn content_search_streaming<F>(
root: &Path,
params: &ContentSearchParams,
cancel: Arc<AtomicBool>,
on_status: F,
) -> anyhow::Result<ContentSearchData>
where
F: Fn(ContentSearchBatch) + Send + 'static,
{
let max_files = params.max_files.unwrap_or(DEFAULT_MAX_FILES);
let max_matches = params.max_matches.unwrap_or(DEFAULT_MAX_MATCHES);
let mut cmd = build_ripgrep_command(root, params);
let mut child = cmd
.spawn()
.map_err(|e| anyhow::anyhow!("Failed to spawn ripgrep: {}", e))?;
let stdout = child
.stdout
.take()
.ok_or_else(|| anyhow::anyhow!("Failed to capture ripgrep stdout"))?;
let mut reader = BufReader::new(stdout).lines();
let mut files: Vec<ContentMatchFile> = Vec::new();
let mut current_file: Option<ContentMatchFile> = None;
let mut total_matches = 0usize;
let mut pending_files: Vec<ContentMatchFile> = Vec::new();
let mut last_notify = Instant::now();
let mut hit_limit = false;
while let Ok(Some(line)) = reader.next_line().await {
if cancel.load(Ordering::Relaxed) {
break;
}
if line.is_empty() {
continue;
}
let json: serde_json::Value = match serde_json::from_str(&line) {
Ok(v) => v,
Err(_) => continue,
};
match json.get("type").and_then(|t| t.as_str()) {
Some("begin") => {
if let Some(file) = current_file.take()
&& !file.matches.is_empty()
{
pending_files.push(file.clone());
files.push(file);
}
if let Some(path) = parse_file_path_from_json(root, &json) {
current_file = Some(ContentMatchFile::new(path));
}
}
Some("match") => {
if let Some(ref mut file) = current_file
&& let Some(data) = json.get("data")
&& let Some(m) = parse_match_from_json(data)
{
file.matches.push(m);
total_matches += 1;
}
}
Some("end") => {
if let Some(file) = current_file.take()
&& !file.matches.is_empty()
{
pending_files.push(file.clone());
files.push(file);
}
}
_ => {}
}
if files.len() >= max_files || total_matches >= max_matches {
hit_limit = true;
break;
}
let should_notify = !pending_files.is_empty()
&& last_notify.elapsed().as_millis() >= BATCH_INTERVAL_MS as u128;
if should_notify {
on_status(ContentSearchBatch {
files: std::mem::take(&mut pending_files),
total_matches,
total_files: files.len(),
done: false,
truncated: false,
});
tokio::task::yield_now().await;
last_notify = Instant::now();
}
}
let cancelled = cancel.load(Ordering::Relaxed);
if hit_limit || cancelled {
let _ = child.kill().await;
}
let _ = child.wait().await;
if cancelled {
let total_files = files.len();
return Ok(ContentSearchData {
files,
total_matches,
total_files,
truncated: false,
});
}
if let Some(file) = current_file
&& !file.matches.is_empty()
&& files.len() < max_files
{
pending_files.push(file.clone());
files.push(file);
}
let truncated = hit_limit;
let total_files = files.len();
on_status(ContentSearchBatch {
files: pending_files,
total_matches,
total_files,
done: true,
truncated,
});
tokio::task::yield_now().await;
Ok(ContentSearchData {
files,
total_matches,
total_files,
truncated,
})
}
@@ -0,0 +1,560 @@
//! Filesystem extension ops (`workspace.fs_*`) — the server-proxied backing
//! for the shell's `x.ai/fs/*` ACP extension methods.
//!
//! These mirror the pure functions that previously lived only in the
//! shell (`kigi-shell/src/session/file_system.rs`) so that, in proxy
//! mode, a `x.ai/fs/*` request executes on the *remote* workspace server
//! instead of the agent host. Each request type implements
//! [`WorkspaceOp`], so it runs in-process for local sessions and routes
//! over the server `workspace_rpc` tool for proxy sessions — identical wire
//! output either way.
//!
//! Path resolution: an absolute `path` is used directly; a relative
//! `path` is joined onto `cwd` (the per-session cwd the shell resolves
//! and sends) or, when absent, the workspace root.
use std::path::{Path, PathBuf};
use async_trait::async_trait;
use base64::Engine;
use chrono::Utc;
use crate::error::{WorkspaceError, WorkspaceResult};
use crate::handle::WorkspaceHandle;
use crate::workspace_ops::WorkspaceOp;
// Canonical in kigi-workspace-types; re-exported for existing paths.
use kigi_workspace_types::rpc::fs::FsReadEncoding;
pub use kigi_workspace_types::rpc::fs::{
FsDeleteFileReq, FsExistsData, FsExistsReq, FsListData, FsListNode, FsListReq, FsReadFileData,
FsReadFileReq, FsWriteFileReq,
};
/// Resolve a request `path` to an absolute path. Absolute paths are used
/// directly; relative paths join `cwd` (the shell-resolved per-session
/// cwd) or, when absent, the workspace root.
fn resolve_abs(
path: &str,
cwd: &Option<PathBuf>,
ws: &WorkspaceHandle,
) -> WorkspaceResult<PathBuf> {
let p = Path::new(path);
if p.is_absolute() {
return Ok(p.to_path_buf());
}
let base = match cwd {
Some(c) => c.clone(),
None => ws.root_cwd()?,
};
Ok(base.join(p))
}
#[async_trait]
impl WorkspaceOp for FsListReq {
async fn execute(
&self,
ws: &WorkspaceHandle,
_session_id: Option<&str>,
) -> WorkspaceResult<Self::Response> {
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
let (abs, confine_root) = ws.confine_to_workspace_root(&abs_unconfined).await?;
// Off-executor: `list` does synchronous walk + metadata syscalls.
let req = self.clone();
tokio::task::spawn_blocking(move || list(&abs, &req, confine_root))
.await
.map_err(|e| WorkspaceError::HubError(e.to_string()))?
}
}
#[async_trait]
impl WorkspaceOp for FsExistsReq {
async fn execute(
&self,
ws: &WorkspaceHandle,
_session_id: Option<&str>,
) -> WorkspaceResult<Self::Response> {
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
let (abs, _) = ws.confine_to_workspace_root(&abs_unconfined).await?;
let exists = tokio::fs::try_exists(&abs).await.unwrap_or(false);
Ok(FsExistsData { exists })
}
}
#[async_trait]
impl WorkspaceOp for FsReadFileReq {
async fn execute(
&self,
ws: &WorkspaceHandle,
_session_id: Option<&str>,
) -> WorkspaceResult<Self::Response> {
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
let (abs, _) = ws.confine_to_workspace_root(&abs_unconfined).await?;
// Legacy full-file read path: preserves the pre-range wire output
// (auto utf8/base64 detect, MIME `type`, `lineCount`).
let ranged = self.offset.is_some()
|| self.length.is_some()
|| self.encoding == FsReadEncoding::Base64;
if !ranged {
let bytes = tokio::fs::read(&abs)
.await
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
return Ok(build_file_entry(&bytes));
}
// Binary-safe ranged read: `size` is the full file size, the
// chunk is `[offset, offset + min(length, max_bytes, cap))`.
let md = tokio::fs::metadata(&abs)
.await
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
if md.is_dir() {
return Err(WorkspaceError::HubError(format!(
"not a file: {}",
self.path
)));
}
// Best-effort snapshot: a concurrent truncate/grow between here and
// read_range can make `size` inconsistent with the returned chunk.
let size = md.len();
let offset = self.offset.unwrap_or(0);
let length = super::walk::clamp_read_length(self.length, self.max_bytes);
let chunk = super::walk::read_range(&abs, offset, length)
.await
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
Ok(build_ranged_entry(chunk, size, self.encoding))
}
}
#[async_trait]
impl WorkspaceOp for FsWriteFileReq {
async fn execute(
&self,
ws: &WorkspaceHandle,
_session_id: Option<&str>,
) -> WorkspaceResult<Self::Response> {
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
let (abs, _) = ws.confine_to_workspace_root(&abs_unconfined).await?;
let content = self.content.clone();
let create_dirs = self.create_dirs;
tokio::task::spawn_blocking(move || -> std::io::Result<()> {
if create_dirs && let Some(parent) = abs.parent() {
std::fs::create_dir_all(parent)?;
}
std::fs::write(&abs, content.as_bytes())
})
.await
.map_err(|e| WorkspaceError::HubError(e.to_string()))?
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
Ok(())
}
}
#[async_trait]
impl WorkspaceOp for FsDeleteFileReq {
async fn execute(
&self,
ws: &WorkspaceHandle,
_session_id: Option<&str>,
) -> WorkspaceResult<Self::Response> {
let abs_unconfined = resolve_abs(&self.path, &self.cwd, ws)?;
let (abs, _) = ws.confine_to_workspace_root(&abs_unconfined).await?;
tokio::fs::remove_file(&abs)
.await
.map_err(|e| WorkspaceError::HubError(e.to_string()))?;
Ok(())
}
}
// =========================================================================
// Pure helpers — ported verbatim from the shell so output is identical.
// =========================================================================
fn list(
abs_path: &Path,
req: &FsListReq,
confine_to_canonical_root: Option<PathBuf>,
) -> WorkspaceResult<FsListData> {
// Confined to the canonical root when set (escaping symlinks not enumerated);
// `None` (the default) walks unconfined.
let page = super::walk::list_directory_paged(
abs_path,
super::walk::ListOptions {
depth: req.depth,
follow_symlinks: req.follow_symlinks,
respect_git_ignore: req.respect_git_ignore,
include_hidden: req.include_hidden,
include_globs: &req.include_globs,
exclude_globs: &req.exclude_globs,
offset: req.offset,
limit: req.limit,
confine_to_canonical_root,
},
super::walk::MAX_LIST_COLLECT,
);
let nodes: Vec<FsListNode> = page
.entries
.into_iter()
.map(|e| FsListNode {
node_type: if e.is_dir { "directory" } else { "file" }.to_string(),
size: e.size,
modified_at: e.modified.map(|st| {
let dt: chrono::DateTime<Utc> = st.into();
dt.to_rfc3339()
}),
is_symlink: e.is_symlink.then_some(true),
path: e.abs_path.to_string_lossy().into_owned(),
name: e.name,
})
.collect();
Ok(FsListData {
nodes,
truncated: page.truncated,
})
}
/// Map a binary-safe ranged chunk to the shell-facing `FsReadFileData`.
/// `size` is the full file size; `lineCount` is omitted for ranged reads
/// and the MIME `type` is a coarse text/binary tag (mid-file chunks make
/// magic-byte sniffing meaningless).
fn build_ranged_entry(chunk: Vec<u8>, size: u64, encoding: FsReadEncoding) -> FsReadFileData {
let (payload, is_text) = super::walk::encode_chunk(chunk, encoding);
let (content, content_base64) = match payload {
super::walk::ChunkPayload::Text(t) => (t, None),
super::walk::ChunkPayload::Base64(b) => (String::new(), Some(b)),
};
FsReadFileData {
content,
content_base64,
size,
line_count: None,
content_type: if is_text {
"text/plain".to_string()
} else {
"application/octet-stream".to_string()
},
}
}
fn build_file_entry(bytes: &[u8]) -> FsReadFileData {
let size = bytes.len() as u64;
let inferred = infer::get(bytes).map(|t| t.mime_type().to_string());
match String::from_utf8(bytes.to_vec()) {
Ok(text) => FsReadFileData {
line_count: Some(text.lines().count() as u64),
content: text,
content_base64: None,
size,
content_type: inferred.unwrap_or_else(|| "text/plain".to_string()),
},
Err(_) => FsReadFileData {
content: String::new(),
content_base64: Some(base64::engine::general_purpose::STANDARD.encode(bytes)),
size,
line_count: None,
content_type: inferred.unwrap_or_else(|| "application/octet-stream".to_string()),
},
}
}
// =========================================================================
// Tests for the pure helpers (no `WorkspaceHandle` required).
// =========================================================================
#[cfg(test)]
mod tests {
use super::*;
/// Build an `FsListReq` for a temp dir test. `path`/`cwd` are unused by
/// `list` (it takes the resolved abs path directly); `respect_git_ignore`
/// is off so the temp dir's location can't filter out our fixtures.
fn list_req(limit: usize) -> FsListReq {
FsListReq {
path: String::new(),
cwd: None,
depth: 1,
limit,
offset: 0,
include_hidden: true,
follow_symlinks: true,
respect_git_ignore: false,
include_globs: Vec::new(),
exclude_globs: Vec::new(),
}
}
#[test]
fn build_file_entry_utf8_sets_content_and_line_count() {
let bytes = b"line one\nline two\n";
let entry = build_file_entry(bytes);
assert_eq!(entry.content, "line one\nline two\n");
assert!(entry.content_base64.is_none());
assert_eq!(entry.line_count, Some(2));
assert_eq!(entry.size, bytes.len() as u64);
}
#[test]
fn build_file_entry_invalid_utf8_uses_base64() {
let bytes: &[u8] = &[0xff, 0xfe, 0x00];
let entry = build_file_entry(bytes);
assert!(entry.content.is_empty());
assert_eq!(
entry.content_base64,
Some(base64::engine::general_purpose::STANDARD.encode(bytes)),
);
assert!(entry.line_count.is_none());
assert_eq!(entry.size, bytes.len() as u64);
}
#[test]
fn list_enumerates_nodes_without_truncation() {
let dir = tempfile::tempdir().expect("tempdir");
let root = dir.path();
std::fs::write(root.join("a.txt"), b"a").expect("write a");
std::fs::write(root.join("b.txt"), b"bb").expect("write b");
std::fs::create_dir(root.join("sub")).expect("mkdir sub");
let data = list(root, &list_req(1000), Some(root.to_path_buf())).expect("list");
let names: Vec<&str> = data.nodes.iter().map(|n| n.name.as_str()).collect();
assert_eq!(data.nodes.len(), 3, "names: {names:?}");
assert!(names.contains(&"a.txt"));
assert!(names.contains(&"b.txt"));
assert!(names.contains(&"sub"));
assert!(!data.truncated);
// Directories sort ahead of files.
assert_eq!(data.nodes[0].name, "sub");
assert_eq!(data.nodes[0].node_type, "directory");
}
#[test]
fn list_marks_truncated_when_limit_reached() {
let dir = tempfile::tempdir().expect("tempdir");
let root = dir.path();
std::fs::write(root.join("a.txt"), b"a").expect("write a");
std::fs::write(root.join("b.txt"), b"bb").expect("write b");
std::fs::create_dir(root.join("sub")).expect("mkdir sub");
let data = list(root, &list_req(1), Some(root.to_path_buf())).expect("list");
assert_eq!(data.nodes.len(), 1);
assert!(data.truncated);
}
#[test]
fn list_paginates_with_offset() {
let dir = tempfile::tempdir().expect("tempdir");
let root = dir.path();
for n in ["a.txt", "b.txt", "c.txt", "d.txt"] {
std::fs::write(root.join(n), b"x").expect("write");
}
let names = |d: FsListData| d.nodes.into_iter().map(|n| n.name).collect::<Vec<_>>();
let mut req = list_req(2);
let p0 = list(root, &req, Some(root.to_path_buf())).expect("list");
assert!(p0.truncated);
assert_eq!(names(p0), vec!["a.txt", "b.txt"]);
req.offset = 2;
let p1 = list(root, &req, Some(root.to_path_buf())).expect("list");
assert!(!p1.truncated, "last page is not truncated");
assert_eq!(names(p1), vec!["c.txt", "d.txt"]);
// Offset past the end yields an empty, non-truncated page.
req.offset = 10;
let p2 = list(root, &req, Some(root.to_path_buf())).expect("list");
assert!(!p2.truncated);
assert!(p2.nodes.is_empty());
}
#[test]
fn build_ranged_entry_encodes_utf8_and_binary() {
// UTF-8 chunk under the default encoding → `content`, text/plain,
// full `size` echoed, no line count.
let e = build_ranged_entry(b"hello".to_vec(), 100, FsReadEncoding::Utf8);
assert_eq!(e.content, "hello");
assert!(e.content_base64.is_none());
assert_eq!(e.size, 100);
assert!(e.line_count.is_none());
assert_eq!(e.content_type, "text/plain");
// Explicit base64 of valid UTF-8 stays text/plain but travels in
// `contentBase64`.
let e = build_ranged_entry(b"hi".to_vec(), 2, FsReadEncoding::Base64);
assert!(e.content.is_empty());
assert_eq!(
e.content_base64,
Some(base64::engine::general_purpose::STANDARD.encode(b"hi")),
);
assert_eq!(e.content_type, "text/plain");
// Non-UTF-8 bytes fall back to base64 + octet-stream.
let raw = vec![0xff_u8, 0x00, 0xfe];
let e = build_ranged_entry(raw.clone(), 3, FsReadEncoding::Utf8);
assert!(e.content.is_empty());
assert_eq!(
e.content_base64,
Some(base64::engine::general_purpose::STANDARD.encode(&raw)),
);
assert_eq!(e.content_type, "application/octet-stream");
}
// Confinement (WorkspaceOp::execute) — covers both local and proxy dispatch.
#[tokio::test]
async fn read_write_within_root_ok() {
let ws = crate::handle::tests::make_confining_handle();
let root = ws.root_cwd().unwrap();
FsWriteFileReq {
path: "sub/data.txt".into(),
cwd: Some(root.clone()),
content: "hello".into(),
create_dirs: true,
}
.execute(&ws, None)
.await
.expect("in-root write must succeed");
let data = FsReadFileReq {
path: "sub/data.txt".into(),
cwd: Some(root.clone()),
offset: None,
length: None,
max_bytes: 1 << 20,
encoding: FsReadEncoding::Utf8,
}
.execute(&ws, None)
.await
.expect("in-root read must succeed");
assert_eq!(data.content, "hello");
}
#[tokio::test]
async fn read_file_rejects_absolute_escape() {
let ws = crate::handle::tests::make_confining_handle();
let err = FsReadFileReq {
path: "/etc/passwd".into(),
cwd: None,
offset: None,
length: None,
max_bytes: 1 << 20,
encoding: FsReadEncoding::Utf8,
}
.execute(&ws, None)
.await
.expect_err("absolute escape must be rejected");
assert!(
err.to_string().contains("workspace root"),
"unexpected error: {err}"
);
}
#[tokio::test]
#[cfg(unix)]
async fn read_file_rejects_symlink_escape() {
use std::os::unix::fs::symlink;
let ws = crate::handle::tests::make_confining_handle();
let root = ws.root_cwd().unwrap();
let outside = tempfile::tempdir().unwrap();
std::fs::write(outside.path().join("secret.txt"), b"secret").unwrap();
symlink(outside.path(), root.join("escape_link")).unwrap();
let err = FsReadFileReq {
path: "escape_link/secret.txt".into(),
cwd: Some(root.clone()),
offset: None,
length: None,
max_bytes: 1 << 20,
encoding: FsReadEncoding::Utf8,
}
.execute(&ws, None)
.await
.expect_err("symlink escape must be rejected");
assert!(
err.to_string().contains("workspace root"),
"unexpected error: {err}"
);
}
#[tokio::test]
#[cfg(unix)]
async fn write_file_rejects_symlink_escape() {
use std::os::unix::fs::symlink;
let ws = crate::handle::tests::make_confining_handle();
let root = ws.root_cwd().unwrap();
let outside = tempfile::tempdir().unwrap();
symlink(outside.path(), root.join("escape_link")).unwrap();
let err = FsWriteFileReq {
path: "escape_link/injected.txt".into(),
cwd: Some(root.clone()),
content: "x".into(),
create_dirs: true,
}
.execute(&ws, None)
.await
.expect_err("symlink escape write must be rejected");
assert!(
err.to_string().contains("workspace root"),
"unexpected error: {err}"
);
assert!(
!outside.path().join("injected.txt").exists(),
"write must not land outside the workspace root"
);
}
// A *dangling* in-root symlink (target outside root, not yet created) must
// not let a write escape via `open(O_CREAT)` following the link.
#[tokio::test]
#[cfg(unix)]
async fn write_file_rejects_dangling_symlink_escape() {
use std::os::unix::fs::symlink;
let ws = crate::handle::tests::make_confining_handle();
let root = ws.root_cwd().unwrap();
let outside = tempfile::tempdir().unwrap();
let target = outside.path().join("new.txt");
symlink(&target, root.join("lnk")).unwrap();
let err = FsWriteFileReq {
path: "lnk".into(),
cwd: Some(root.clone()),
content: "x".into(),
create_dirs: true,
}
.execute(&ws, None)
.await
.expect_err("dangling symlink escape write must be rejected");
assert!(
err.to_string().contains("workspace root"),
"unexpected error: {err}"
);
assert!(
!target.exists(),
"write must not create the file outside root"
);
}
#[tokio::test]
#[cfg(unix)]
async fn list_excludes_symlink_escape() {
use std::os::unix::fs::symlink;
let ws = crate::handle::tests::make_confining_handle();
let root = ws.root_cwd().unwrap();
let outside = tempfile::tempdir().unwrap();
std::fs::write(outside.path().join("secret.txt"), b"x").unwrap();
symlink(outside.path(), root.join("escape_link")).unwrap();
std::fs::write(root.join("inside.txt"), b"y").unwrap();
let mut req = list_req(1000);
req.path = ".".into();
req.cwd = Some(root.clone());
req.depth = 2;
let data = req.execute(&ws, None).await.expect("list must succeed");
let names: Vec<&str> = data.nodes.iter().map(|n| n.name.as_str()).collect();
assert!(names.contains(&"inside.txt"), "in-root file: {names:?}");
assert!(
!data.nodes.iter().any(|n| n.path.contains("secret.txt")),
"escaping symlink target must not be enumerated: {names:?}"
);
}
}
@@ -0,0 +1,526 @@
//! Generates the file-tree based on how we are using it during training
//! This gives the model an overview of the project and helps it navigate and understand
//! the repository better, cold-starting the exploration
use std::collections::{HashMap, VecDeque};
use std::path::{Path, PathBuf};
use std::time::Instant;
use dashmap::DashMap;
use ignore::{WalkBuilder, WalkState};
use crate::file_system::FsError;
/// Number of threads for parallel directory walking
const NUM_WALK_THREADS: usize = 8;
/// Configuration for limiting file tree traversal to prevent runaway I/O
/// on very large or deeply nested directories.
#[derive(Debug, Clone, Copy)]
pub struct ListContentsLimits {
/// Maximum number of characters in the output
pub max_characters: usize,
/// Maximum depth to traverse (0 = root only)
pub max_depth: usize,
/// Maximum number of directories to visit during traversal
pub max_dirs_visited: usize,
}
impl Default for ListContentsLimits {
fn default() -> Self {
Self {
max_characters: 10_000,
max_depth: 12,
max_dirs_visited: 2000,
}
}
}
impl ListContentsLimits {
pub fn new(max_characters: usize, max_depth: usize, max_dirs_visited: usize) -> Self {
Self {
max_characters,
max_depth,
max_dirs_visited,
}
}
}
fn get_top_exts(files: &[String]) -> Vec<(String, usize)> {
let mut ext_counts: HashMap<String, usize> = HashMap::new();
for item in files {
let ext = if let Some(e) = Path::new(item).extension() {
format!(".{}", e.to_str().unwrap_or("").to_lowercase())
} else {
String::new()
};
*ext_counts.entry(ext).or_insert(0) += 1;
}
let mut vec: Vec<_> = ext_counts.into_iter().collect();
vec.sort_by_key(|&(_, count)| std::cmp::Reverse(count));
vec
}
fn get_file_ext_str(files: &[String], k: usize) -> String {
let top_exts = get_top_exts(files);
let include_dots = top_exts.len() > k
|| (top_exts.len() == k && top_exts.iter().any(|(ext, _)| ext.is_empty()));
let top_k_exts = &top_exts[0..std::cmp::min(k, top_exts.len())];
if top_k_exts.is_empty() {
return String::new();
}
if top_k_exts.len() == 1 && top_k_exts[0].0.is_empty() {
return "(...)".to_string();
}
let filtered_top_k_exts: Vec<_> = top_k_exts
.iter()
.filter(|(ext, _)| !ext.is_empty())
.collect();
let top_counts = filtered_top_k_exts
.iter()
.map(|(ext, cnt)| format!("{} *{}", cnt, ext))
.collect::<Vec<_>>()
.join(", ");
if include_dots {
format!("({top_counts}, ...)")
} else if top_counts.is_empty() {
String::new()
} else {
format!("({top_counts})")
}
}
/// Pre-collected directory contents from a single walk
struct DirContents {
files: Vec<String>,
dirs: Vec<String>,
}
/// Performs a single parallel walk and collects all directory contents into a map.
/// Returns a map from directory path -> (files, subdirs) in that directory.
fn collect_all_contents(
root: &Path,
max_depth: usize,
max_dirs: usize,
) -> Result<HashMap<PathBuf, DirContents>, FsError> {
let _timer = (); // instrumentation_timer noop (dev infra)
let contents_map: DashMap<PathBuf, DirContents> = DashMap::new();
let files_count = std::sync::atomic::AtomicUsize::new(0);
let dirs_count = std::sync::atomic::AtomicUsize::new(0);
let entries_visited = std::sync::atomic::AtomicUsize::new(0);
// Initialize root entry
contents_map.insert(
root.to_path_buf(),
DirContents {
files: Vec::new(),
dirs: Vec::new(),
},
);
let walker = WalkBuilder::new(root)
.max_depth(Some(max_depth + 1)) // +1 because depth 0 is root itself
.follow_links(false)
.same_file_system(true)
.ignore(true)
.git_ignore(true)
.git_global(true)
.git_exclude(true)
.hidden(true)
.threads(NUM_WALK_THREADS)
.build_parallel();
tracing::debug!(
root = %root.display(),
max_depth = max_depth,
max_dirs = max_dirs,
threads = NUM_WALK_THREADS,
"Starting parallel file walk"
);
{
let _timer = (); // instrumentation_timer noop (dev infra)
walker.run(|| {
let contents_map = &contents_map;
let files_count = &files_count;
let dirs_count = &dirs_count;
let entries_visited = &entries_visited;
Box::new(move |entry| {
entries_visited.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
// Stop early if we've collected enough directories
if dirs_count.load(std::sync::atomic::Ordering::Relaxed) >= max_dirs {
return WalkState::Quit;
}
let entry = match entry {
Ok(e) => e,
Err(_) => return WalkState::Continue,
};
// Skip root itself
if entry.depth() == 0 {
return WalkState::Continue;
}
let Some(file_type) = entry.file_type() else {
return WalkState::Continue;
};
let path = entry.path();
let Some(parent) = path.parent() else {
return WalkState::Continue;
};
let name = entry.file_name().to_string_lossy().to_string();
if file_type.is_dir() {
dirs_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
contents_map
.entry(parent.to_path_buf())
.or_insert_with(|| DirContents {
files: Vec::new(),
dirs: Vec::new(),
})
.dirs
.push(format!("{name}/"));
// Pre-create entry for this directory (even if empty)
contents_map
.entry(path.to_path_buf())
.or_insert_with(|| DirContents {
files: Vec::new(),
dirs: Vec::new(),
});
} else if file_type.is_file() {
files_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
contents_map
.entry(parent.to_path_buf())
.or_insert_with(|| DirContents {
files: Vec::new(),
dirs: Vec::new(),
})
.files
.push(name);
}
WalkState::Continue
})
});
}
let _entries_total = entries_visited.load(std::sync::atomic::Ordering::Relaxed);
// (instrumentation_timer.with_field calls removed — dev-only, not part of Phase 1 move)
let mut contents_map: HashMap<PathBuf, DirContents> = {
let _timer = (); // instrumentation_timer noop (dev infra)
contents_map.into_iter().collect()
};
// Sort all entries for stable output
{
let _timer = (); // instrumentation_timer noop (dev infra)
for contents in contents_map.values_mut() {
contents.files.sort_by_cached_key(|n| n.to_lowercase());
contents.dirs.sort_by_cached_key(|n| n.to_lowercase());
}
}
Ok(contents_map)
}
struct DirectoryNode {
depth: usize,
path: PathBuf,
files: Vec<String>,
dirs: Vec<String>,
summary_str: String,
children: Option<HashMap<String, DirectoryNode>>,
num_listed_files: usize,
}
impl DirectoryNode {
fn new(path: PathBuf, depth: usize, contents: &HashMap<PathBuf, DirContents>) -> Self {
let (files, dirs) = contents
.get(&path)
.map(|c| (c.files.clone(), c.dirs.clone()))
.unwrap_or_default();
let mut node = Self {
depth,
path,
files,
dirs,
summary_str: String::new(),
children: None,
num_listed_files: 0,
};
node.summary_str = node.get_remaining_str(&[], false, 3);
node
}
fn get_remaining_str(
&self,
excluded_files: &[String],
exclude_all_dirs: bool,
k: usize,
) -> String {
let remaining_files: Vec<String> = self
.files
.iter()
.filter(|f| !excluded_files.contains(*f))
.cloned()
.collect();
let mut file_ext_str = get_file_ext_str(&remaining_files, k);
if !file_ext_str.is_empty() {
file_ext_str.push(' ');
}
let file_count = remaining_files.len();
let dir_count = if exclude_all_dirs { 0 } else { self.dirs.len() };
let indent = " ".repeat(self.depth + 1);
format!("{indent}- [+{file_count} files {file_ext_str}& {dir_count} dirs]")
}
fn is_expanded(&self) -> bool {
self.children.is_some()
}
fn expand_children(&mut self, contents: &HashMap<PathBuf, DirContents>) {
if self.is_expanded() {
return;
}
let mut children: HashMap<String, DirectoryNode> = HashMap::new();
for dir in &self.dirs {
let child_path = self.path.join(dir.trim_end_matches('/'));
let dir_node = DirectoryNode::new(child_path, self.depth + 1, contents);
children.insert(dir.clone(), dir_node);
}
self.children = Some(children);
self.num_listed_files = std::cmp::min(3, self.files.len());
}
fn unexpand_children(&mut self) {
self.children = None;
self.num_listed_files = 0;
}
fn subitem_str(&self, subitem: &str) -> String {
let indent = " ".repeat(self.depth + 1);
format!("{indent}- {subitem}")
}
fn get_complete_str(&self) -> String {
assert!(self.is_expanded());
let children = self.children.as_ref().unwrap();
let mut remaining_subitems = self.dirs.clone();
remaining_subitems.extend(self.files[0..self.num_listed_files].iter().cloned());
remaining_subitems.sort_by_key(|s| s.to_lowercase());
let mut curr_str = String::new();
for subitem in remaining_subitems {
curr_str.push_str(&self.subitem_str(&subitem));
curr_str.push('\n');
if let Some(child) = children.get(&subitem) {
if child.is_expanded() {
curr_str.push_str(&child.get_complete_str());
curr_str.push('\n');
} else {
curr_str.push_str(&child.summary_str);
curr_str.push('\n');
}
}
}
if self.files.len() > self.num_listed_files {
let remaining_str =
self.get_remaining_str(&self.files[0..self.num_listed_files], true, 3);
curr_str.push_str(&remaining_str);
}
curr_str.trim_end_matches('\n').to_string()
}
}
/// Creates the project overview
pub async fn list_contents(
path: impl Into<PathBuf>,
limits: ListContentsLimits,
) -> Result<String, FsError> {
let _timer = (); // instrumentation_timer noop (dev infra)
let t_total = Instant::now();
let path: PathBuf = path.into();
// Use this only for the printed header
let path_head = {
let s = path.to_string_lossy().replace('\\', "/");
if s.ends_with('/') {
s.to_owned()
} else {
format!("{s}/")
}
};
let lim_characters = limits.max_characters + path_head.len();
// Single walk to collect all directory contents
let path_clone = path.clone();
let max_depth = limits.max_depth;
let max_dirs = limits.max_dirs_visited;
let contents = {
let _timer = (); // instrumentation_timer noop (dev infra)
tokio::task::spawn_blocking(move || collect_all_contents(&path_clone, max_depth, max_dirs))
.await
.map_err(|e| FsError::Other(format!("walk join error: {e}")))?
}?;
let t_walk = t_total.elapsed();
let (
mut root_node,
mut remaining_chars,
to_fit_files,
dirs_visited,
max_depth_reached,
depth_limit_hit,
dirs_limit_hit,
) = {
let _timer = (); // instrumentation_timer noop (dev infra)
let mut root_node = DirectoryNode::new(path, 0, &contents);
let min_chars = path_head.len() + root_node.summary_str.len();
if min_chars > lim_characters {
return Err(FsError::Other(format!(
"Minimum possible string is too long for character limit, {} > {}",
min_chars, lim_characters
)));
}
let mut remaining_chars = lim_characters - min_chars;
let mut to_fit_files = true;
let mut dirs_visited: usize = 1; // Count root as visited
let mut max_depth_reached: usize = 0;
let mut depth_limit_hit = false;
let mut dirs_limit_hit = false;
let mut q: VecDeque<&mut DirectoryNode> = VecDeque::new();
q.push_back(&mut root_node);
while let Some(node) = q.pop_front() {
// Check if we've hit the max directories limit
if dirs_visited >= limits.max_dirs_visited {
dirs_limit_hit = true;
to_fit_files = false;
break;
}
remaining_chars += node.summary_str.len();
node.expand_children(&contents);
dirs_visited += node.dirs.len();
max_depth_reached = max_depth_reached.max(node.depth);
let test_str = node.get_complete_str();
let new_additional_len = test_str.replace('\n', "").len();
if new_additional_len > remaining_chars {
node.unexpand_children();
to_fit_files = false;
break;
}
remaining_chars -= new_additional_len;
if let Some(children_map) = node.children.as_mut() {
let mut child_values: Vec<&mut DirectoryNode> = children_map.values_mut().collect();
child_values.sort_by_key(|node| node.path.to_string_lossy().to_lowercase());
// Filter out children that exceed max_depth
for child in child_values {
if child.depth > limits.max_depth {
depth_limit_hit = true;
continue;
}
q.push_back(child);
}
}
}
(
root_node,
remaining_chars,
to_fit_files,
dirs_visited,
max_depth_reached,
depth_limit_hit,
dirs_limit_hit,
)
};
{
let _timer = (); // instrumentation_timer noop (dev infra)
let mut q: VecDeque<&mut DirectoryNode> = VecDeque::new();
if to_fit_files {
q.push_back(&mut root_node);
}
let mut file_done = false;
while let Some(node) = q.pop_front() {
if !node.is_expanded() {
continue;
}
let num_file_limit = node.files.len();
for i in node.num_listed_files..num_file_limit {
let new_additional_len = node.subitem_str(&node.files[i]).len();
if new_additional_len <= remaining_chars {
node.num_listed_files = i + 1;
remaining_chars -= new_additional_len;
} else {
file_done = true;
break;
}
}
if file_done {
break;
}
if let Some(children_map) = node.children.as_mut() {
let mut child_values: Vec<&mut DirectoryNode> = children_map.values_mut().collect();
child_values.sort_by_key(|node| node.path.to_string_lossy().to_lowercase());
q.extend(child_values);
}
}
}
let output = {
let _timer = (); // instrumentation_timer noop (dev infra)
let mut output = format!("{path_head}\n");
if root_node.is_expanded() {
output.push_str(&root_node.get_complete_str());
} else {
output.push_str(&root_node.summary_str);
}
output
};
// Log warnings when limits are hit
if depth_limit_hit {
tracing::warn!(
path = %path_head,
max_depth = limits.max_depth,
"list_contents: max_depth limit hit, some directories were not traversed"
);
}
if dirs_limit_hit {
tracing::warn!(
path = %path_head,
max_dirs_visited = limits.max_dirs_visited,
dirs_visited = dirs_visited,
"list_contents: max_dirs_visited limit hit, traversal stopped early"
);
}
tracing::debug!(
path = %path_head,
max_characters = limits.max_characters,
max_depth = limits.max_depth,
max_dirs_visited = limits.max_dirs_visited,
dirs_visited = dirs_visited,
max_depth_reached = max_depth_reached,
depth_limit_hit = depth_limit_hit,
dirs_limit_hit = dirs_limit_hit,
output_len = output.len(),
walk_ms = t_walk.as_millis() as u64,
elapsed_ms = t_total.elapsed().as_millis() as u64,
"list_contents complete"
);
Ok(output)
}
@@ -0,0 +1,143 @@
use kigi_paths::ToAbsPath;
use std::io;
use std::path::Path;
use std::sync::Arc;
#[derive(Debug, thiserror::Error)]
pub enum FsError {
#[error(transparent)]
Io(#[from] io::Error),
#[error("{0}")]
Other(String),
}
// TODO: handle atomic write
#[async_trait::async_trait]
pub trait AsyncFileSystem: Send + Sync {
/// Get the root directory for this filesystem.
///
/// This is used to resolve relative paths via `ToAbsPath::to_abs_path(fs.root())`.
fn root(&self) -> &Path;
async fn exists(&self, path: &Path) -> Result<bool, FsError>;
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, FsError>;
/// Read a file if it exists, returning `Ok(None)` when the file is not found.
///
/// The default implementation calls `exists()` then `read_file()` (two operations).
/// Backends should override this to collapse both into a single operation —
/// e.g. one ACP RPC or one syscall — to avoid a redundant round trip.
async fn try_read_file(&self, path: &Path) -> Result<Option<Vec<u8>>, FsError> {
if self.exists(path).await? {
Ok(Some(self.read_file(path).await?))
} else {
Ok(None)
}
}
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), FsError>;
/// Delete a file (for rewind functionality)
async fn delete_file(&self, path: &Path) -> Result<(), FsError>;
}
pub fn bytes_to_string(file_bytes: Vec<u8>) -> Result<String, FsError> {
String::from_utf8(file_bytes).map_err(|e| FsError::Other(e.to_string()))
}
// ============================================================================
// AsyncFsWrapper - Generic wrapper that accepts any path type
// ============================================================================
/// A wrapper around `AsyncFileSystem` that accepts any path type implementing `ToAbsPath`.
///
/// This allows callers to pass `AbsPathBuf`, `RelPathBuf`, `&Path`, or `&PathBuf` directly,
/// and the wrapper automatically resolves them to absolute paths using the filesystem's root.
///
/// # Example
/// ```ignore
/// let wrapper = AsyncFsWrapper::new(fs);
///
/// // All of these work:
/// wrapper.read_to_string(&abs_path).await?;
/// wrapper.read_to_string(&rel_path).await?;
/// wrapper.read_to_string(Path::new("relative/path")).await?;
/// ```
#[derive(Clone)]
pub struct AsyncFsWrapper {
inner: Arc<dyn AsyncFileSystem>,
}
impl AsyncFsWrapper {
pub fn new(fs: Arc<dyn AsyncFileSystem>) -> Self {
Self { inner: fs }
}
/// Get a reference to the inner `Arc<dyn AsyncFileSystem>`.
///
/// This is useful when you need raw access to the underlying filesystem
/// without the path conversion layer.
pub fn inner(&self) -> &Arc<dyn AsyncFileSystem> {
&self.inner
}
/// Get the root directory for this filesystem.
pub fn root(&self) -> &Path {
self.inner.root()
}
/// Check if a file exists.
pub async fn exists<P: ToAbsPath>(&self, path: P) -> Result<bool, FsError> {
self.inner.exists(&path.to_abs_path(self.root())).await
}
/// Read a file as bytes.
pub async fn read_file<P: ToAbsPath>(&self, path: P) -> Result<Vec<u8>, FsError> {
self.inner.read_file(&path.to_abs_path(self.root())).await
}
/// Read a file as a UTF-8 string.
pub async fn read_to_string<P: ToAbsPath>(&self, path: P) -> Result<String, FsError> {
let bytes = self.inner.read_file(&path.to_abs_path(self.root())).await?;
bytes_to_string(bytes)
}
/// Read a file as bytes if it exists, returning `Ok(None)` when not found.
///
/// Uses a single backend operation instead of separate `exists()` + `read_file()`.
pub async fn try_read_file<P: ToAbsPath>(&self, path: P) -> Result<Option<Vec<u8>>, FsError> {
self.inner
.try_read_file(&path.to_abs_path(self.root()))
.await
}
/// Read a file as a UTF-8 string if it exists, returning `Ok(None)` when not found.
///
/// Uses a single backend operation instead of separate `exists()` + `read_to_string()`.
pub async fn try_read_to_string<P: ToAbsPath>(
&self,
path: P,
) -> Result<Option<String>, FsError> {
match self
.inner
.try_read_file(&path.to_abs_path(self.root()))
.await?
{
Some(bytes) => Ok(Some(bytes_to_string(bytes)?)),
None => Ok(None),
}
}
/// Write data to a file.
pub async fn write_file<P: ToAbsPath>(&self, path: P, data: &[u8]) -> Result<(), FsError> {
self.inner
.write_file(&path.to_abs_path(self.root()), data)
.await
}
/// Delete a file.
pub async fn delete_file<P: ToAbsPath>(&self, path: P) -> Result<(), FsError> {
self.inner.delete_file(&path.to_abs_path(self.root())).await
}
}
@@ -0,0 +1,479 @@
use std::{
path::{Path, PathBuf},
sync::{
Arc, Mutex,
atomic::{AtomicBool, Ordering},
mpsc::{RecvError, RecvTimeoutError, SyncSender, sync_channel},
},
thread::{self, JoinHandle},
time::Duration,
};
use ignore::{DirEntry, WalkBuilder, WalkState, overrides::OverrideBuilder};
use nucleo::{
Match, Matcher, Nucleo, Snapshot, Utf32String,
pattern::{CaseMatching, MultiPattern, Normalization, Pattern},
};
const NUM_NUCLEO_THREADS: usize = 2;
const NUM_IGNORE_THREADS: usize = 8;
#[derive(Debug, Clone, Default)]
pub struct FuzzyMatchResult {
// Path of the matched entry.
pub path: Utf32String,
/// Matcher score, higher is better.
pub score: u32,
/// Matched indices of characters.
pub indices: Vec<u32>,
/// Is it a directory.
pub is_dir: bool,
}
#[derive(Debug, Clone, Copy, Default)]
pub struct FuzzyMatcherStatus {
pub changed: bool,
pub done: bool,
}
#[derive(Debug, Clone)]
struct MatchEntry {
pub is_dir: bool,
}
/// A very fast fuzzy matcher that does ignore-walking. Both happen in background threads.
pub struct FuzzyFileMatcher {
root: PathBuf,
query: String,
nucleo: Nucleo<MatchEntry>,
matcher: Matcher,
walk_handle: Option<JoinHandle<()>>,
cancel: Arc<AtomicBool>,
top_entries: Vec<FuzzyMatchResult>,
dirs: bool,
}
impl FuzzyFileMatcher {
/// Create a new matcher with default config focused on matching paths.
pub fn new(root: &Path) -> Self {
let matcher_config = nucleo::Config::DEFAULT.match_paths();
// matcher_config.prefer_prefix = true; // yes or no? nucleo docs lean towards no
let mut nucleo = Nucleo::new(
matcher_config.clone(),
Arc::new(move || ()),
Some(NUM_NUCLEO_THREADS),
1,
);
nucleo.pattern = MultiPattern::new(1);
Self {
root: root.to_owned(),
nucleo,
matcher: Matcher::new(matcher_config),
walk_handle: None,
cancel: Arc::new(AtomicBool::new(false)),
query: String::new(),
top_entries: Vec::new(),
dirs: false,
}
}
pub fn query(&self) -> &str {
&self.query
}
/// Start a new walk and restart nucleo matcher.
pub fn restart_walk_custom(
&mut self,
make_walker: impl FnOnce(&mut WalkBuilder) -> &mut WalkBuilder,
) {
// first, wait for previous walker to finish if it's up
self.cancel.store(true, Ordering::Relaxed);
if let Some(walk_handle) = self.walk_handle.take() {
walk_handle.join().unwrap();
}
// disconnect all injectors and clear snapshots and streams
self.nucleo.restart(true);
// we're back in business
self.cancel.store(false, Ordering::Relaxed);
// build the walker(s)
let walker_builder = make_walker(
WalkBuilder::new(&self.root)
.threads(NUM_IGNORE_THREADS)
.follow_links(false)
.git_ignore(true)
.git_global(true)
.git_exclude(true)
.ignore(true)
.hidden(true)
.require_git(false)
.overrides(
OverrideBuilder::new(&self.root)
.add("!.git")
.unwrap()
.build()
.unwrap(),
),
)
.clone();
fn check_entry<'a>(entry: &'a DirEntry, root: &Path) -> Option<(&'a str, bool)> {
let path = entry.path();
if path != root
&& let Some(file_type) = entry.file_type()
&& (file_type.is_file() || file_type.is_dir())
&& let Ok(path) = path.strip_prefix(root)
&& let Some(path) = path.as_os_str().to_str()
&& !path.is_empty()
{
Some((path, file_type.is_dir()))
} else {
None
}
}
// we'll just do it in a blocking way here assuming it's super fast anyway
let top_walker = walker_builder
.clone()
.max_depth(Some(1))
.sort_by_file_name(|a, b| a.cmp(b))
.build();
let top_entries = top_walker
.into_iter()
.filter_map(|entry| {
let entry = entry.ok()?;
let (path, is_dir) = check_entry(&entry, &self.root)?;
Some(FuzzyMatchResult {
path: path.into(),
score: 0,
indices: Vec::new(),
is_dir,
})
})
.collect::<Vec<_>>();
let injector = self.nucleo.injector();
let root = self.root.clone();
let cancel = self.cancel.clone();
// link walker threads with injectors and start it up
let walker = walker_builder.build_parallel();
let walk_handle = thread::spawn(move || {
walker.run(|| {
let injector = injector.clone();
let root = root.clone();
let cancel = cancel.clone();
Box::new(move |entry| {
if cancel.load(Ordering::Relaxed) {
return WalkState::Quit;
} else if let Ok(entry) = entry
&& let Some((path, is_dir)) = check_entry(&entry, &root)
{
injector.push(MatchEntry { is_dir }, |_entry, columns| {
columns[0] = path.into();
});
}
WalkState::Continue
})
});
});
self.walk_handle = Some(walk_handle);
self.top_entries = top_entries;
self.nucleo.tick(0);
}
/// Restart the walk with default walker parameters.
pub fn restart_walk(&mut self) {
self.restart_walk_custom(|w| w);
}
/// Set the query to a given string and trigger reparse.
///
/// It will be faster if the current query is a strict prefix of the new query.
pub fn set_query(&mut self, mut query: &str, dirs: bool) {
self.dirs = dirs;
if dirs && query.ends_with('/') {
query = &query[..query.len() - 1];
}
if query == self.query {
return;
}
// see this re: backslash etc: https://github.com/helix-editor/nucleo/pull/87
let append = query.as_bytes().starts_with(self.query.as_bytes())
&& !query.ends_with('\\')
&& !query
.as_bytes()
.last()
.is_some_and(|ch| ch.is_ascii_whitespace());
self.nucleo
.pattern
.reparse(0, query, CaseMatching::Smart, Normalization::Smart, append);
self.nucleo.tick(0);
self.query = query.to_owned();
}
/// Sends a tick to nucleo matcher. Can be safely called at any frequency.
pub fn tick(&mut self, tick_timeout_ms: u64) -> FuzzyMatcherStatus {
if self.query.is_empty() {
return FuzzyMatcherStatus {
done: true,
changed: false,
};
}
let status = self.nucleo.tick(tick_timeout_ms);
let done = self.nucleo.active_injectors() == 0 && !status.running;
FuzzyMatcherStatus {
done,
changed: status.changed,
}
}
/// Total number of currently matched items in the snapshot.
pub fn num_items(&self) -> usize {
if self.query.is_empty() {
self.top_entries.len()
} else {
self.nucleo.snapshot().item_count() as _
}
}
/// Get top `k` items from the snapshot and sort them by score, path length and path.
pub fn get_top_k(&mut self, k: usize) -> Vec<FuzzyMatchResult> {
// note: &mut only because we access self.matcher which has internal allocations
// rust is a bit dumb at times, we'll need this for sorting without cloning
fn sort_by_key_hrtb<T, F, K, Q>(slice: &mut [T], f: F)
where
F: for<'a> Fn(&'a T) -> (Q, &'a K),
K: Ord,
Q: Ord,
{
slice.sort_by(|a, b| f(a).cmp(&f(b)))
}
// special case: if query is empty, return top items only
if self.query.is_empty() {
return self
.top_entries
.iter()
// dirs_only=true means only directories; dirs_only=false means both files and directories
.filter(|e| !self.dirs || e.is_dir)
.take(k)
.cloned()
.collect(); // should be already sorted
}
// https://github.com/helix-editor/helix/blob/d79cce4e4bfc24dd204f1b294c899ed73f7e9453/helix-term/src/ui/completion.rs#L369
// suggested min score = 7 * len + 14
let len = self.query.chars().count() as u32;
let min_score = 7 + len * 14;
let mut items = Vec::with_capacity(k);
let pattern = self.nucleo.pattern.column_pattern(0);
let snapshot = self.nucleo.snapshot();
let mut iter = snapshot.matches().iter().peekable();
while items.len() < k
&& let Some(m) = iter.next()
// for empty queries, return everything; otherwise, apply heuristic min-score limit
&& (self.query.is_empty() || m.score >= min_score)
{
fn extract_match(
m: &Match,
snapshot: &Snapshot<MatchEntry>,
pattern: &Pattern,
matcher: &mut Matcher,
dirs_only: bool,
) -> Option<FuzzyMatchResult> {
let item = unsafe { snapshot.get_item_unchecked(m.idx) };
// dirs_only=true means only directories; dirs_only=false means both files and directories
if dirs_only && !item.data.is_dir {
return None;
}
let path = item.matcher_columns[0].clone();
let mut indices = Vec::new();
if !pattern.atoms.is_empty() {
pattern.indices(path.slice(..), matcher, &mut indices);
}
Some(FuzzyMatchResult {
path,
score: m.score,
indices,
is_dir: item.data.is_dir,
})
}
if !pattern.atoms.is_empty() {
let start = items.len();
items.extend(extract_match(
m,
snapshot,
pattern,
&mut self.matcher,
self.dirs,
));
while iter.peek().is_some_and(|p| p.score == m.score) {
let m = iter.next().unwrap();
items.extend(extract_match(
m,
snapshot,
pattern,
&mut self.matcher,
self.dirs,
));
}
sort_by_key_hrtb(&mut items[start..], |m| (m.path.len(), &m.path));
} else {
items.extend(extract_match(
m,
snapshot,
pattern,
&mut self.matcher,
self.dirs,
));
}
}
if items.len() > k {
items.truncate(k);
}
if pattern.atoms.is_empty() {
sort_by_key_hrtb(&mut items, |m| (true, &m.path));
}
items
}
}
impl Drop for FuzzyFileMatcher {
fn drop(&mut self) {
// note: walker threads *may* get detached for a little while but hopefully not for too long
self.cancel.store(true, Ordering::Relaxed);
}
}
#[derive(Debug, Clone, Default)]
pub struct FuzzyMatcherDaemonResults {
pub topk: Arc<[FuzzyMatchResult]>,
pub num_items: usize,
pub status: FuzzyMatcherStatus,
pub generation: usize,
}
impl AsRef<[FuzzyMatchResult]> for FuzzyMatcherDaemonResults {
fn as_ref(&self) -> &[FuzzyMatchResult] {
self.topk.as_ref()
}
}
#[derive(Debug, Clone)]
enum FuzzyMatcherDaemonMessage {
RestartWalk { hidden: bool },
SetQuery { query: String, dirs: bool },
Stop,
}
pub struct FuzzyFileMatcherDaemon {
results: Arc<Mutex<FuzzyMatcherDaemonResults>>,
tx: SyncSender<FuzzyMatcherDaemonMessage>,
_handle: JoinHandle<()>,
}
impl FuzzyFileMatcherDaemon {
pub fn new(mut matcher: FuzzyFileMatcher, topk: usize) -> Self {
let results = Arc::new(Mutex::new(FuzzyMatcherDaemonResults::default()));
let (tx, rx) = sync_channel(1024);
let res = results.clone();
let handle = thread::spawn(move || {
let results = res;
let mut done = false;
let mut generation = 0;
loop {
let msg = if !done {
rx.recv_timeout(Duration::from_micros(250))
} else {
rx.recv().map_err(|e| match e {
RecvError => RecvTimeoutError::Disconnected,
})
};
match msg {
Ok(FuzzyMatcherDaemonMessage::RestartWalk { hidden }) => {
if !hidden {
tracing::trace!("restarting normal walk");
matcher.restart_walk();
} else {
tracing::trace!("restarting hidden walk");
matcher.restart_walk_custom(|w| {
w.hidden(false).ignore(false).git_ignore(false)
});
}
generation += 1;
*results.lock().unwrap() = FuzzyMatcherDaemonResults::default();
done = false;
}
Ok(FuzzyMatcherDaemonMessage::SetQuery { query, dirs }) => {
matcher.set_query(&query, dirs);
generation += 1;
done = false;
}
Ok(FuzzyMatcherDaemonMessage::Stop) | Err(RecvTimeoutError::Disconnected) => {
break;
}
Err(RecvTimeoutError::Timeout) => {
if !done {
let status = matcher.tick(10);
done = status.done;
let num_items = matcher.num_items();
let topk: Arc<[_]> = matcher.get_top_k(topk).into();
*results.lock().unwrap() = FuzzyMatcherDaemonResults {
topk,
num_items,
status,
generation,
};
generation += 1;
}
}
}
}
});
Self {
results,
tx,
_handle: handle,
}
}
pub fn get(&self) -> FuzzyMatcherDaemonResults {
self.results.lock().unwrap().clone()
}
pub fn set_query(&self, query: impl AsRef<str>, dirs: bool) {
let query = query.as_ref().to_owned();
_ = self
.tx
.send(FuzzyMatcherDaemonMessage::SetQuery { query, dirs })
.ok();
}
pub fn restart_walk(&self, hidden: bool) {
_ = self
.tx
.send(FuzzyMatcherDaemonMessage::RestartWalk { hidden })
.ok();
}
}
impl Drop for FuzzyFileMatcherDaemon {
fn drop(&mut self) {
_ = self.tx.send(FuzzyMatcherDaemonMessage::Stop).ok();
}
}
@@ -0,0 +1,290 @@
//! Generates a compact git status for the system prompt.
//!
//! Uses the git CLI for performance — libgit2's status is 5-10x slower than
//! the native git binary on large repos due to inefficient index refresh.
//! Output is prioritized by change type and limited to ~1k characters.
use crate::file_system::FsError;
use std::fmt::Write;
use std::path::{Path, PathBuf};
/// Gets a compact git status for the system prompt using the git CLI.
///
/// Output includes:
/// 1. Branch name
/// 2. Upstream ahead/behind status
/// 3. Staged files (if any)
///
/// Total output is capped at ~1k characters.
pub async fn git_status(working_directory: impl Into<PathBuf>) -> Result<String, FsError> {
let working_directory = working_directory.into();
tokio::task::spawn_blocking(move || git_status_impl(&working_directory))
.await
.map_err(|e| FsError::Other(format!("git status task failed: {}", e)))?
}
/// Matches Node's default `execFile` `maxBuffer` (1 MiB). This cap is
/// load-bearing: `git status` output at or above it makes the spawn throw, so
/// the repo is dropped from `<git_status>` entirely (never truncated).
/// Oversized output is treated as an error -- the caller maps `Err` to a
/// dropped section.
const GIT_STATUS_BUFFER_LIMIT: usize = 1024 * 1024;
/// Whether `git status` stdout is large enough that the repo is dropped
/// (`>= 1 MiB`). Extracted as a pure predicate so it is unit-testable
/// without spawning git.
fn git_status_exceeds_buffer(stdout_len: usize) -> bool {
stdout_len >= GIT_STATUS_BUFFER_LIMIT
}
/// Collapse runs of 2+ spaces to a single space.
///
/// The `<git_status>` body collapses consecutive spaces, so the
/// porcelain two-column status renders with a single separator: `A staged.txt`
/// (index-added, clean worktree) becomes `A staged.txt`, `M mod.txt` becomes
/// `M mod.txt`, `R old -> new` becomes `R old -> new`. A single leading space
/// (e.g. ` M file`, worktree-modified) and the rename ` -> ` separator are
/// preserved because they are runs of length one.
/// Newlines are never touched.
fn collapse_status_spaces(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut prev_space = false;
for ch in s.chars() {
if ch == ' ' {
if prev_space {
continue;
}
prev_space = true;
} else {
prev_space = false;
}
out.push(ch);
}
out
}
/// Short git status for the templated user message.
///
/// Runs `git status --short --branch` and returns its output with consecutive
/// spaces collapsed via [`collapse_status_spaces`]: a leading `## <branch>`
/// line followed by the file change list (or just `## <branch>` on a clean
/// tree). This matches the body embedded in the `<git_status>` block
/// byte-for-byte.
pub async fn git_status_short(working_directory: impl Into<PathBuf>) -> Result<String, FsError> {
let working_directory = working_directory.into();
tokio::task::spawn_blocking(move || {
let output = kigi_tty_utils::git_command()
.args(["status", "--short", "--branch"])
.current_dir(&working_directory)
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::null())
.output()
.map_err(|e| FsError::Other(format!("git status --short --branch failed: {}", e)))?;
if !output.status.success() {
return Err(FsError::Other(format!(
"git status --short --branch exited with code {:?}",
output.status.code()
)));
}
// Output >= 1 MiB is dropped entirely, not truncated. Render-time
// truncation handles the < 1 MiB case.
if git_status_exceeds_buffer(output.stdout.len()) {
return Err(FsError::Other(
"git status --short --branch output exceeded 1 MiB buffer".to_string(),
));
}
// Consecutive spaces in the status body are collapsed so staged
// entries (`A file` -> `A file`) match the wire format.
Ok(collapse_status_spaces(&String::from_utf8_lossy(
&output.stdout,
)))
})
.await
.map_err(|e| FsError::Other(format!("git status --short --branch task failed: {}", e)))?
}
fn git_status_impl(working_directory: &Path) -> Result<String, FsError> {
let _timer = /* instrumentation_timer */ () ; // dev macro; noop stub ("git_status.impl")
let max_status_chars = 1000;
let mut output = String::with_capacity(max_status_chars);
// Get branch name
let branch_name = {
let _timer = /* instrumentation_timer */ () ; // dev macro; noop stub ("git_status.branch_info")
run_git(working_directory, &["rev-parse", "--abbrev-ref", "HEAD"])
};
match &branch_name {
Some(branch) if branch == "HEAD" => {
// Detached HEAD — get short commit hash
if let Some(hash) = run_git(working_directory, &["rev-parse", "--short", "HEAD"]) {
let _ = writeln!(output, "HEAD detached at {}", hash);
}
}
Some(branch) => {
let _ = writeln!(output, "On branch {}", branch);
}
None => {
return Err(FsError::Other("not a git repository".to_string()));
}
}
// Get upstream ahead/behind
{
let _timer = (); // instrumentation_timer noop stub
if let Some(upstream_name) = run_git(
working_directory,
&["rev-parse", "--abbrev-ref", "@{upstream}"],
) && let Some(counts) = run_git(
working_directory,
&["rev-list", "--count", "--left-right", "@{upstream}...HEAD"],
) {
let parts: Vec<&str> = counts.split_whitespace().collect();
if let (Some(behind_str), Some(ahead_str)) = (parts.first(), parts.get(1)) {
let behind: usize = behind_str.parse().unwrap_or(0);
let ahead: usize = ahead_str.parse().unwrap_or(0);
let status_msg = match (ahead, behind) {
(0, 0) => {
format!("Your branch is up to date with '{}'.", upstream_name)
}
(a, 0) => format!(
"Your branch is ahead of '{}' by {} commit{}.",
upstream_name,
a,
if a == 1 { "" } else { "s" }
),
(0, b) => format!(
"Your branch is behind '{}' by {} commit{}.",
upstream_name,
b,
if b == 1 { "" } else { "s" }
),
(a, b) => format!(
"Your branch and '{}' have diverged ({} ahead, {} behind).",
upstream_name, a, b
),
};
let _ = writeln!(output, "{}", status_msg);
}
}
}
// Get staged changes (index vs HEAD) — fast, no workdir scan
let staged_output = {
let _timer = /* instrumentation_timer */ () ; // dev macro; noop stub ("git_status.staged")
run_git(
working_directory,
&["diff", "--cached", "--name-status", "HEAD"],
)
};
let mut staged: Vec<String> = Vec::new();
if let Some(ref diff_output) = staged_output {
for line in diff_output.lines() {
let mut parts = line.splitn(2, '\t');
let status_char = parts.next().unwrap_or("");
let path = parts.next().unwrap_or("");
if path.is_empty() {
continue;
}
let formatted = match status_char.chars().next() {
Some('A') => format!("\tnew file: {}", path),
Some('M') => format!("\tmodified: {}", path),
Some('D') => format!("\tdeleted: {}", path),
Some('R') => format!("\trenamed: {}", path),
_ => format!("\t{}: {}", status_char, path),
};
staged.push(formatted);
}
}
// Check if clean
if staged.is_empty() {
let _ = writeln!(output, "\nnothing to commit, working tree clean");
return Ok(output);
}
// Reserve space for truncation message
let reserve_for_truncation = 50;
let char_budget = max_status_chars - reserve_for_truncation;
// Write staged files
if !staged.is_empty() && output.len() < char_budget {
let _ = writeln!(output, "\nChanges to be committed:");
for (shown, line) in staged.iter().enumerate() {
if output.len() + line.len() + 1 > char_budget {
let remaining = staged.len() - shown;
if remaining > 0 {
let _ = writeln!(output, "\t... and {} more staged", remaining);
}
break;
}
let _ = writeln!(output, "{}", line);
}
}
Ok(output)
}
/// Run a read-only git command and return its stdout, trimmed.
/// Returns None on failure.
///
/// Uses `--no-optional-locks` to avoid creating `index.lock` for stat-cache
/// refreshes. This function is called from background tasks (system prompt
/// generation) and must never contend with foreground git operations.
fn run_git(cwd: &Path, args: &[&str]) -> Option<String> {
let output = kigi_tty_utils::git_command()
.args(args)
.current_dir(cwd)
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::null())
.output()
.ok()?;
if !output.status.success() {
return None;
}
let stdout = String::from_utf8_lossy(&output.stdout).trim().to_string();
if stdout.is_empty() {
None
} else {
Some(stdout)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn git_status_buffer_cap_matches_spec() {
assert!(!git_status_exceeds_buffer(0));
assert!(!git_status_exceeds_buffer(GIT_STATUS_BUFFER_LIMIT - 1));
// At or above 1 MiB -> dropped.
assert!(git_status_exceeds_buffer(GIT_STATUS_BUFFER_LIMIT));
assert!(git_status_exceeds_buffer(GIT_STATUS_BUFFER_LIMIT + 1));
}
/// Staged entries collapse the porcelain double space, while leading
/// single spaces and ` -> ` are preserved.
#[test]
fn collapse_status_spaces_matches_spec() {
let raw = "## main...origin/main\n M committed.txt\nA staged.txt\nM mod.txt\nR old.txt -> new.txt\n?? untracked.txt\n";
let want = "## main...origin/main\n M committed.txt\nA staged.txt\nM mod.txt\nR old.txt -> new.txt\n?? untracked.txt\n";
assert_eq!(collapse_status_spaces(raw), want);
}
/// Newlines are never collapsed (blank lines preserved).
#[test]
fn collapse_status_spaces_preserves_newlines() {
assert_eq!(collapse_status_spaces("a\n\n\nb"), "a\n\n\nb");
assert_eq!(collapse_status_spaces(""), "");
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,85 @@
//! Compact jj status for the system prompt.
use std::fmt::Write;
use std::path::{Path, PathBuf};
use std::process::Command;
use crate::file_system::FsError;
/// jj template: change ID, commit ID, description, bookmarks.
const JJ_LOG_TEMPLATE: &str = r#"separate("\n",
"Change: " ++ change_id.shortest(8),
"Commit: " ++ commit_id.shortest(8),
if(description,
"Description: " ++ description.first_line(),
"Description: (no description set)"),
if(bookmarks,
"Bookmarks: " ++ bookmarks.join(", "),
"")
)"#;
/// Compact jj status for the system prompt (~1k chars max).
pub async fn jj_status(working_directory: impl Into<PathBuf>) -> Result<String, FsError> {
let working_directory = working_directory.into();
tokio::task::spawn_blocking(move || jj_status_impl(&working_directory))
.await
.map_err(|e| FsError::Other(format!("jj status task failed: {e}")))?
}
fn jj_status_impl(cwd: &Path) -> Result<String, FsError> {
let max_chars = 1000;
let mut out = String::with_capacity(max_chars);
let log = run_jj(
cwd,
&["log", "--no-graph", "-r", "@", "-T", JJ_LOG_TEMPLATE],
)
.ok_or_else(|| FsError::Other("not a jujutsu repository".into()))?;
for line in log.lines().filter(|l| !l.is_empty()) {
let _ = writeln!(out, "{line}");
}
match run_jj(cwd, &["st"]) {
Some(st) if st.contains("The working copy is clean") || st.is_empty() => {
let _ = writeln!(out, "\nWorking copy is clean");
}
Some(st) => {
let _ = writeln!(out);
let budget = max_chars - 50;
for (i, line) in st.lines().enumerate() {
if out.len() + line.len() + 1 > budget {
let remaining = st.lines().count() - i;
if remaining > 0 {
let _ = writeln!(out, "... and {remaining} more lines");
}
break;
}
let _ = writeln!(out, "{line}");
}
}
None => {}
}
Ok(out)
}
/// Run a jj command synchronously, returning trimmed stdout or `None` on failure.
fn run_jj(cwd: &Path, args: &[&str]) -> Option<String> {
let mut cmd = Command::new("jj");
cmd.arg("--ignore-working-copy")
.args(args)
.current_dir(cwd)
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::null())
.stdin(std::process::Stdio::null());
kigi_tools::util::detach_std_command(&mut cmd);
let output = cmd.output().ok()?;
if !output.status.success() {
return None;
}
let stdout = String::from_utf8_lossy(&output.stdout).trim().to_string();
(!stdout.is_empty()).then_some(stdout)
}
@@ -0,0 +1,50 @@
use std::path::{Path, PathBuf};
use tokio::fs;
use crate::file_system::{AsyncFileSystem, FsError};
pub struct LocalFs {
root: PathBuf,
}
impl LocalFs {
pub fn new(root: PathBuf) -> Self {
Self { root }
}
}
#[async_trait::async_trait]
impl AsyncFileSystem for LocalFs {
fn root(&self) -> &Path {
&self.root
}
async fn exists(&self, path: &Path) -> Result<bool, FsError> {
Ok(fs::try_exists(path).await?)
}
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, FsError> {
Ok(fs::read(path).await?)
}
async fn try_read_file(&self, path: &Path) -> Result<Option<Vec<u8>>, FsError> {
match fs::read(path).await {
Ok(bytes) => Ok(Some(bytes)),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(e) => Err(e.into()),
}
}
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), FsError> {
if let Some(dir) = path.parent() {
fs::create_dir_all(dir).await?;
}
fs::write(path, data).await?;
Ok(())
}
async fn delete_file(&self, path: &Path) -> Result<(), FsError> {
fs::remove_file(path).await?;
Ok(())
}
}
@@ -0,0 +1,58 @@
use std::collections::HashMap;
use std::io;
use std::path::{Path, PathBuf};
use tokio::sync::RwLock;
use crate::file_system::{AsyncFileSystem, FsError};
pub struct MockFs {
root: PathBuf,
files: RwLock<HashMap<PathBuf, Vec<u8>>>,
}
#[async_trait::async_trait]
impl AsyncFileSystem for MockFs {
fn root(&self) -> &Path {
&self.root
}
async fn exists(&self, path: &Path) -> Result<bool, FsError> {
let map = self.files.read().await;
Ok(map.contains_key(path))
}
async fn read_file(&self, path: &Path) -> Result<Vec<u8>, FsError> {
let map = self.files.read().await;
if let Some(bytes) = map.get(path) {
Ok(bytes.clone())
} else {
Err(io::Error::new(io::ErrorKind::NotFound, "File not found").into())
}
}
async fn try_read_file(&self, path: &Path) -> Result<Option<Vec<u8>>, FsError> {
let map = self.files.read().await;
Ok(map.get(path).cloned())
}
async fn write_file(&self, path: &Path, data: &[u8]) -> Result<(), FsError> {
let mut map = self.files.write().await;
map.insert(path.to_path_buf(), data.to_vec());
Ok(())
}
async fn delete_file(&self, path: &Path) -> Result<(), FsError> {
let mut map = self.files.write().await;
map.remove(path);
Ok(())
}
}
impl MockFs {
pub fn new(root: PathBuf) -> Self {
Self {
root,
files: RwLock::new(HashMap::new()),
}
}
}
@@ -0,0 +1,360 @@
mod acp_fs;
pub use acp_fs::AcpSessionFs;
mod ext_fs;
pub use ext_fs::{
FsDeleteFileReq, FsExistsData, FsExistsReq, FsListData, FsListNode, FsListReq, FsReadFileData,
FsReadFileReq, FsWriteFileReq,
};
// Client-facing read-only fs ops (`workspace.client_fs_*`). Not re-exported:
// its wire types live in `kigi_workspace_types::rpc::fs` (the `ClientFs*`
// types) and would collide with the shell-facing `ext_fs` names above.
pub(crate) mod client_fs;
// Shared filesystem core: paginated listing + binary-safe ranged reads,
// used by `client_fs`, `ext_fs`, and the shell-local `session::file_system`.
mod walk;
pub use walk::{
ChunkPayload, ListOptions, ListPage, ListedEntry, MAX_LIST_COLLECT, MAX_READ_BYTES,
clamp_read_length, encode_chunk, list_directory_paged, read_range,
};
// Re-exported so shell-side fs ops can name the shared read encoding.
pub use kigi_workspace_types::rpc::fs::FsReadEncoding;
pub mod adapter;
pub use adapter::AcpFsAdapter;
mod codebase_index;
pub use codebase_index::CodebaseIndexManager;
mod fs;
pub use fs::{AsyncFileSystem, AsyncFsWrapper, FsError, bytes_to_string};
mod local_fs;
pub use local_fs::LocalFs;
mod mock_fs;
pub use mock_fs::MockFs;
mod file_tree;
pub use file_tree::{ListContentsLimits, list_contents};
mod git_status;
pub use git_status::{git_status, git_status_short};
mod jj_status;
pub use jj_status::jj_status;
mod attach_file;
pub use attach_file::{FileReference, render_embedded_resource, render_file_reference};
mod fuzzy;
pub use fuzzy::{
FuzzyFileMatcher, FuzzyFileMatcherDaemon, FuzzyMatchResult, FuzzyMatcherDaemonResults,
FuzzyMatcherStatus,
};
mod index;
pub use index::{FileEntry, FileIndex, FileIndexDelta, SegmentId, StringInterner, WalkOptions};
mod content;
pub use content::{
ContentMatch, ContentMatchFile, ContentSearchBatch, ContentSearchData, ContentSearchParams,
content_search_streaming,
};
use serde::Serialize;
use std::{
collections::HashMap,
path::{Path, PathBuf},
time::{Duration, Instant},
};
use uuid::Uuid;
// Canonical in kigi-workspace-types; re-exported for existing paths.
pub use kigi_workspace_types::rpc::search::{ClientId, ContentSearchRequest, TargetClientId};
impl From<ContentSearchRequest> for ContentSearchParams {
fn from(req: ContentSearchRequest) -> Self {
let pattern = if req.is_regex {
req.pattern
} else if req.whole_word {
format!("\\b{}\\b", regex::escape(&req.pattern))
} else {
req.pattern
};
let literal = !req.is_regex && !req.whole_word;
let globs: Vec<String> = req
.include_globs
.into_iter()
.chain(req.exclude_globs.into_iter().map(|g| format!("!{g}")))
.collect();
Self {
pattern,
case_insensitive: req.case_insensitive,
literal,
globs,
max_files: req.max_files,
max_matches: req.max_matches,
respect_gitignore: req.respect_gitignore,
}
}
}
const DEFAULT_SEARCH_TIMEOUT_SECS: u64 = 30;
const DEFAULT_TOP_K: usize = 1000;
pub type FuzzySearchId = String;
impl Serialize for FuzzyMatchResult {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer,
{
use serde::ser::SerializeStruct;
use std::borrow::Cow;
let path_str = self.path.to_string();
let node_type = if self.is_dir { "directory" } else { "file" };
let name: Cow<str> = std::path::Path::new(&path_str)
.file_name()
.map(|s| s.to_string_lossy())
.unwrap_or(Cow::Borrowed(&path_str));
let mut state = serializer.serialize_struct("FuzzyMatchResult", 5)?;
state.serialize_field("name", &name)?;
state.serialize_field("type", node_type)?;
state.serialize_field("path", &path_str)?;
state.serialize_field("score", &self.score)?;
state.serialize_field("indices", &self.indices)?;
state.end()
}
}
#[derive(Debug, Clone, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct FuzzySearchData {
pub matches: Vec<FuzzyMatchResult>,
pub total: usize,
pub done: bool,
pub generation: usize,
}
/// Result of one fuzzy-search poll tick (see [`WorkspaceHandle::fuzzy_poll`]).
///
/// Consumed in-process by the shell's notification driver, so it carries the
/// (non-`Deserialize`) match results directly rather than going over RPC.
///
/// [`WorkspaceHandle::fuzzy_poll`]: crate::handle::WorkspaceHandle::fuzzy_poll
#[derive(Debug, Clone)]
pub enum FuzzyPollOutcome {
/// The query was superseded by a newer change — stop polling.
Stale,
/// The search no longer exists — stop polling.
Closed,
/// The search exists but produced no new results this tick — keep polling.
Pending,
/// New results, with paths already absolutized against the search root.
Update(FuzzySearchData),
}
pub struct FuzzySearchContext {
pub daemon: FuzzyFileMatcherDaemon,
pub created_at: Instant,
pub last_activity: Instant,
pub hidden: bool,
pub min_generation: usize,
pub has_query: bool,
pub query_version: usize,
/// The root path for this search (used to convert relative paths to absolute).
pub root: PathBuf,
/// Session ID for routing notifications.
/// Used by the relay to route notifications to session subscribers.
pub session_id: Option<String>,
/// Target client ID for routing notifications.
/// Extracted from `_meta.clientId` in the open request.
pub target_client_id: TargetClientId,
}
impl FuzzySearchContext {
pub fn new(
root: &Path,
hidden: bool,
session_id: Option<String>,
target_client_id: TargetClientId,
) -> Self {
let matcher = FuzzyFileMatcher::new(root);
let daemon = FuzzyFileMatcherDaemon::new(matcher, DEFAULT_TOP_K);
daemon.restart_walk(hidden);
Self {
daemon,
created_at: Instant::now(),
last_activity: Instant::now(),
hidden,
min_generation: 0,
has_query: false,
query_version: 0,
root: root.to_path_buf(),
session_id,
target_client_id,
}
}
pub fn is_stale(&self, timeout: Duration) -> bool {
self.last_activity.elapsed() > timeout
}
}
pub struct FuzzySearchManager {
searches: HashMap<FuzzySearchId, FuzzySearchContext>,
timeout: Duration,
}
impl FuzzySearchManager {
pub fn new(timeout: Duration) -> Self {
Self {
searches: HashMap::new(),
timeout,
}
}
pub fn open(
&mut self,
root: &Path,
request_id: Option<String>,
hidden: bool,
session_id: Option<String>,
target_client_id: TargetClientId,
) -> FuzzySearchId {
self.cleanup_stale();
let search_id = request_id.unwrap_or_else(|| Uuid::now_v7().to_string());
let context = FuzzySearchContext::new(root, hidden, session_id, target_client_id);
self.searches.insert(search_id.clone(), context);
search_id
}
/// Get the session ID for a search, if one was set.
/// Used for routing notifications to session subscribers.
pub fn get_session_id(&self, search_id: &str) -> Option<String> {
self.searches
.get(search_id)
.and_then(|ctx| ctx.session_id.clone())
}
/// Get the target client ID for a search, if one was set.
/// Used for routing notifications to the correct client via relay.
pub fn get_target_client_id(&self, search_id: &str) -> TargetClientId {
self.searches
.get(search_id)
.map(|ctx| ctx.target_client_id.clone())
.unwrap_or_default()
}
/// Get the root path for a search.
/// Used to convert relative paths to absolute paths in results.
pub fn get_root(&self, search_id: &str) -> Option<PathBuf> {
self.searches.get(search_id).map(|ctx| ctx.root.clone())
}
pub fn change(
&mut self,
search_id: &str,
query: &str,
dirs_only: bool,
) -> Option<(usize, bool, usize)> {
let ctx = self.searches.get_mut(search_id)?;
ctx.last_activity = Instant::now();
// Rewalk on empty query to refresh index when picker opens.
if query.is_empty() {
ctx.daemon.restart_walk(ctx.hidden);
}
ctx.daemon.set_query(query, dirs_only);
ctx.min_generation += 1;
ctx.has_query = !query.is_empty();
ctx.query_version += 1;
Some((ctx.min_generation, ctx.has_query, ctx.query_version))
}
pub fn is_current_query(&self, search_id: &str, query_version: usize) -> bool {
self.searches
.get(search_id)
.is_some_and(|ctx| ctx.query_version == query_version)
}
pub fn get_results(&mut self, search_id: &str) -> Option<FuzzySearchData> {
let ctx = self.searches.get_mut(search_id)?;
ctx.last_activity = Instant::now();
let results = ctx.daemon.get();
Some(FuzzySearchData {
matches: results.topk.to_vec(),
total: results.num_items,
done: results.status.done,
generation: results.generation,
})
}
pub fn get_results_filtered(
&mut self,
search_id: &str,
min_gen: usize,
has_query: bool,
) -> Option<FuzzySearchData> {
let ctx = self.searches.get_mut(search_id)?;
ctx.last_activity = Instant::now();
let results = ctx.daemon.get();
if results.generation < min_gen {
return None;
}
// Skip intermediate states: empty results or unscored defaults while scanning
if has_query && !results.status.done {
if results.topk.is_empty() {
return None;
}
let all_unscored = results
.topk
.iter()
.all(|m| m.score == 0 && m.indices.is_empty());
if all_unscored {
return None;
}
}
Some(FuzzySearchData {
matches: results.topk.to_vec(),
total: results.num_items,
done: results.status.done,
generation: results.generation,
})
}
pub fn close(&mut self, search_id: &str) -> bool {
self.searches.remove(search_id).is_some()
}
pub fn cleanup_stale(&mut self) {
let timeout = self.timeout;
self.searches.retain(|_, ctx| !ctx.is_stale(timeout));
}
pub fn active_count(&self) -> usize {
self.searches.len()
}
}
impl Default for FuzzySearchManager {
fn default() -> Self {
Self::new(Duration::from_secs(DEFAULT_SEARCH_TIMEOUT_SECS))
}
}
@@ -0,0 +1,344 @@
//! Shared filesystem core for the fs list/read ops.
//!
//! Owns the `ignore::WalkBuilder` configuration, glob overrides, the
//! dirs-first paginated listing ([`list_directory_paged`]), and the
//! binary-safe ranged-read primitives ([`read_range`], [`encode_chunk`])
//! used by all three fs surfaces — the shell-local
//! `session::file_system`, the shell-facing
//! [`ext_fs`](super::ext_fs) `workspace.fs_*`, and the client-facing
//! [`client_fs`](super::client_fs) `workspace.client_fs_*` — so walk and
//! read fixes apply to every consumer. Each consumer maps the neutral
//! [`ListedEntry`] / [`ChunkPayload`] to its own wire shape (absolute vs
//! root-relative paths, RFC 3339 vs epoch-ms timestamps, MIME vs
//! text/binary type tags).
use std::path::{Path, PathBuf};
use std::time::SystemTime;
use base64::Engine;
use ignore::{WalkBuilder, overrides::OverrideBuilder};
use kigi_workspace_types::rpc::fs::FsReadEncoding;
/// Hard cap on entries collected per list call before sorting. A
/// pathological directory truncates (`truncated = true`) instead of
/// ballooning memory. Shared by every fs surface.
pub const MAX_LIST_COLLECT: usize = 50_000;
/// Server-side cap on a single ranged read's effective byte budget
/// (`min(length, max_bytes)`). 4 MiB raw (≈ 5.3 MiB base64) stays under
/// the server's 8 MiB frame cap. Shared by every fs read surface.
pub const MAX_READ_BYTES: u64 = 4 * 1024 * 1024;
/// Resolve a ranged read's effective byte budget, shared by every fs read
/// surface so the clamp policy can't drift between them. An absent `length`
/// means "to EOF", but the result is always capped at the caller's
/// `max_bytes` and the hard [`MAX_READ_BYTES`] server limit — so a short
/// read is expected, and callers detect "more data" by comparing the
/// returned bytes (at `offset`) against the file `size`.
pub fn clamp_read_length(length: Option<u64>, max_bytes: u64) -> u64 {
length
.unwrap_or(u64::MAX)
.min(max_bytes)
.min(MAX_READ_BYTES)
}
/// Walk configuration. Field semantics mirror the `x.ai/fs/list` request.
pub(super) struct FsWalk<'a> {
pub depth: usize,
pub follow_symlinks: bool,
pub respect_git_ignore: bool,
pub include_hidden: bool,
pub include_globs: &'a [String],
pub exclude_globs: &'a [String],
/// When set, symlink entries whose canonical target leaves this
/// canonical root are excluded — and not descended into — so a walk
/// of a confined tree cannot enumerate paths outside it. `None`
/// preserves the shell's unconfined semantics.
pub confine_to_canonical_root: Option<PathBuf>,
}
/// One raw walk entry; `metadata` follows symlinks (like `fs::metadata`).
pub(super) struct RawFsEntry {
pub path: PathBuf,
pub name: String,
pub is_symlink: bool,
pub metadata: std::fs::Metadata,
}
/// Walk `abs_dir` per `opts`, collecting up to `max_entries` entries (the
/// root itself is skipped; unreadable entries are skipped without counting).
/// Returns `(entries, hit_cap)` where `hit_cap` means the walk stopped at
/// the cap with entries left over.
pub(super) fn walk_fs_entries(
abs_dir: &Path,
opts: FsWalk<'_>,
max_entries: usize,
) -> (Vec<RawFsEntry>, bool) {
let overrides = build_glob_overrides(abs_dir, opts.include_globs, opts.exclude_globs);
let mut builder = WalkBuilder::new(abs_dir);
builder
.max_depth(Some(opts.depth))
.follow_links(opts.follow_symlinks)
.same_file_system(true)
.standard_filters(true)
.git_ignore(opts.respect_git_ignore)
.git_global(opts.respect_git_ignore)
.git_exclude(opts.respect_git_ignore)
.hidden(!opts.include_hidden)
.overrides(overrides);
if let Some(canonical_root) = opts.confine_to_canonical_root {
builder.filter_entry(move |dent| symlink_stays_in_root(dent.path(), &canonical_root));
}
let mut entries: Vec<RawFsEntry> = Vec::new();
let mut hit_cap = false;
for dent in builder.build() {
let Ok(entry) = dent else { continue };
if entry.depth() == 0 {
continue;
}
if entries.len() >= max_entries {
hit_cap = true;
break;
}
let path = entry.path().to_path_buf();
let is_symlink = std::fs::symlink_metadata(&path)
.map(|m| m.file_type().is_symlink())
.unwrap_or(false);
let metadata = match std::fs::metadata(&path) {
Ok(m) => m,
Err(_) => continue,
};
entries.push(RawFsEntry {
name: entry.file_name().to_string_lossy().into_owned(),
path,
is_symlink,
metadata,
});
}
(entries, hit_cap)
}
/// `true` when `path` is not a symlink, or is a symlink whose canonical
/// target stays under `canonical_root`. Unverifiable symlinks (e.g.
/// dangling) are excluded — confinement fails closed.
fn symlink_stays_in_root(path: &Path, canonical_root: &Path) -> bool {
let is_symlink = std::fs::symlink_metadata(path)
.map(|m| m.file_type().is_symlink())
.unwrap_or(false);
if !is_symlink {
return true;
}
dunce::canonicalize(path)
.map(|c| c.starts_with(canonical_root))
.unwrap_or(false)
}
pub(super) fn build_glob_overrides(
base: &Path,
include: &[String],
exclude: &[String],
) -> ignore::overrides::Override {
let mut ob = OverrideBuilder::new(base);
for pat in include {
let patt = if pat.starts_with('!') {
pat.clone()
} else {
format!("!{}", pat)
};
let _ = ob.add(&patt);
}
for pat in exclude {
let _ = ob.add(pat);
}
ob.build().unwrap_or_else(|_| {
OverrideBuilder::new(base)
.build()
.expect("override build fallback")
})
}
// =========================================================================
// Paginated listing
// =========================================================================
/// One listed node in neutral form (no wire serialization). Consumers map
/// this to their own node shape.
pub struct ListedEntry {
/// File name (final path component).
pub name: String,
/// Absolute path on the workspace/host filesystem.
pub abs_path: PathBuf,
/// Whether the entry is a directory.
pub is_dir: bool,
/// Whether the entry itself is a symlink.
pub is_symlink: bool,
/// Size in bytes (files only).
pub size: Option<u64>,
/// Modification time, when readable.
pub modified: Option<SystemTime>,
}
/// One page of a directory listing.
pub struct ListPage {
/// The page slice `[offset, offset + limit)` after the dirs-first sort.
pub entries: Vec<ListedEntry>,
/// `true` when more entries exist beyond this page, or the collection
/// cap was hit before the walk finished.
pub truncated: bool,
}
/// Options for [`list_directory_paged`].
pub struct ListOptions<'a> {
pub depth: usize,
pub follow_symlinks: bool,
pub respect_git_ignore: bool,
pub include_hidden: bool,
pub include_globs: &'a [String],
pub exclude_globs: &'a [String],
/// Pagination offset applied after the sort.
pub offset: u64,
/// Page size (already clamped by the caller as appropriate).
pub limit: usize,
/// When set, mid-walk symlink escapes outside this canonical root are
/// excluded; `None` keeps the shell's unconfined semantics.
pub confine_to_canonical_root: Option<PathBuf>,
}
/// Whether more entries exist than this page returned. True when entries
/// remain beyond the page (`end < total`) OR the walk hit the collection cap
/// AND the caller has not yet paged past the collected window
/// (`hit_cap && start < total`). Gating the cap term on `start < total` is
/// what makes a `while truncated { offset += limit }` loop terminate: once a
/// client has consumed every collected entry the flag drops to false instead
/// of reporting the (unreachable) over-cap remainder forever.
fn page_truncated(start: usize, end: usize, total: usize, hit_cap: bool) -> bool {
end < total || (hit_cap && start < total)
}
/// Walk `abs_dir`, sort directories-first / case-insensitive (with exact
/// name as a deterministic tiebreak), then return the stable slice
/// `[offset, offset + limit)`. See [`page_truncated`] for the `truncated`
/// semantics (incomplete listing OR more pages, but terminating).
pub fn list_directory_paged(abs_dir: &Path, opts: ListOptions<'_>, max_collect: usize) -> ListPage {
let (raw, hit_cap) = walk_fs_entries(
abs_dir,
FsWalk {
depth: opts.depth,
follow_symlinks: opts.follow_symlinks,
respect_git_ignore: opts.respect_git_ignore,
include_hidden: opts.include_hidden,
include_globs: opts.include_globs,
exclude_globs: opts.exclude_globs,
confine_to_canonical_root: opts.confine_to_canonical_root,
},
max_collect,
);
let mut entries: Vec<ListedEntry> = raw
.into_iter()
.map(|e| ListedEntry {
is_dir: e.metadata.is_dir(),
size: e.metadata.is_file().then_some(e.metadata.len()),
modified: e.metadata.modified().ok(),
is_symlink: e.is_symlink,
abs_path: e.path,
name: e.name,
})
.collect();
// Directories first, then case-insensitive by name; exact name as a
// tiebreak so page boundaries are deterministic.
entries.sort_by_cached_key(|n| (!n.is_dir, n.name.to_lowercase(), n.name.clone()));
let total = entries.len();
let start = usize::try_from(opts.offset)
.unwrap_or(usize::MAX)
.min(total);
let end = start.saturating_add(opts.limit).min(total);
let truncated = page_truncated(start, end, total, hit_cap);
entries.truncate(end);
let page = entries.split_off(start);
ListPage {
entries: page,
truncated,
}
}
// =========================================================================
// Ranged, binary-safe reads
// =========================================================================
/// A read chunk in the requested transfer encoding.
pub enum ChunkPayload {
/// Valid UTF-8 text (caller places it in a `content` field).
Text(String),
/// Base64 of the raw bytes (caller places it in a `contentBase64`
/// field). Used when `base64` is requested or the bytes are not UTF-8.
Base64(String),
}
/// Encode `bytes` per `encoding`, returning the payload and whether the
/// bytes were valid UTF-8 (`is_text`). One UTF-8 validation pass; on the
/// `Utf8` request the error hands the bytes back for base64 fallback.
pub fn encode_chunk(bytes: Vec<u8>, encoding: FsReadEncoding) -> (ChunkPayload, bool) {
let b64 = |b: &[u8]| base64::engine::general_purpose::STANDARD.encode(b);
match (encoding, String::from_utf8(bytes)) {
(FsReadEncoding::Utf8, Ok(text)) => (ChunkPayload::Text(text), true),
(FsReadEncoding::Utf8, Err(e)) => (ChunkPayload::Base64(b64(e.as_bytes())), false),
(FsReadEncoding::Base64, Ok(text)) => (ChunkPayload::Base64(b64(text.as_bytes())), true),
(FsReadEncoding::Base64, Err(e)) => (ChunkPayload::Base64(b64(e.as_bytes())), false),
}
}
/// Read only `[offset, offset + length)` of `abs` (no hashing).
pub async fn read_range(abs: &Path, offset: u64, length: u64) -> std::io::Result<Vec<u8>> {
use tokio::io::{AsyncReadExt, AsyncSeekExt};
let mut f = tokio::fs::File::open(abs).await?;
if offset > 0 {
f.seek(std::io::SeekFrom::Start(offset)).await?;
}
let mut chunk = Vec::new();
f.take(length).read_to_end(&mut chunk).await?;
Ok(chunk)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn clamp_read_length_caps_at_max_bytes_and_hard_limit() {
// Absent length -> capped at max_bytes.
assert_eq!(clamp_read_length(None, 1024), 1024);
// Explicit length above max_bytes -> max_bytes wins.
assert_eq!(clamp_read_length(Some(8192), 1024), 1024);
// Explicit length below max_bytes -> length wins.
assert_eq!(clamp_read_length(Some(512), 1024), 512);
// max_bytes above the hard server limit -> hard limit wins.
assert_eq!(clamp_read_length(None, u64::MAX), MAX_READ_BYTES);
assert_eq!(clamp_read_length(Some(u64::MAX), u64::MAX), MAX_READ_BYTES);
}
#[test]
fn page_truncated_signals_more_pages_within_collected_set() {
// 100 collected, no cap, page [0,10) -> more remain.
assert!(page_truncated(0, 10, 100, false));
// Last page [90,100) -> nothing remains.
assert!(!page_truncated(90, 100, 100, false));
}
#[test]
fn page_truncated_terminates_when_paging_past_collection_cap() {
// Cap hit, total clamped to 50 collected, limit 10.
// Populated pages stay truncated (listing is incomplete)...
assert!(page_truncated(0, 10, 50, true));
assert!(page_truncated(40, 50, 50, true));
// ...but once the client pages past the collected window the flag
// drops, so `while truncated { offset += limit }` terminates instead
// of fetching empty pages forever.
assert!(!page_truncated(50, 50, 50, true));
assert!(!page_truncated(60, 50, 50, true));
}
}