Hard fork of xai-org/grok-build (Apache-2.0) re-targeted as Kigi, an
unofficial Kimi Code CLI community build.
Rename & identity
- 72 xai-*/xai-grok-* crates -> kigi-* (explicit: xai-grok-pager-bin ->
kigi-bin [binary `kigi`], xai-grok-pager -> kigi-tui; rest mechanical);
ptyctl, ptyctl-cli, third_party/ unchanged; proto package
xai.grok.tools.v1 -> kigi.tools.v1
- Config home ~/.kigi (KIGI_SHARE_DIR override), env prefix GROK_* ->
KIGI_*, `kigi --version` carries the unofficial-community-build notice
- clap identity, help text, startup banner, prompt templates rebranded
(templates re-encrypted)
Deletions (PRD removal list #5/#6/#7/#9/#10)
- voice input (xai-grok-voice) and all TUI wiring
- telemetry: Mixpanel client, external OTel stream, Sentry, OTLP layers,
trace/GCS/S3 upload queues (kigi-file-utils halved), workspace upload
module & dc_log, heap-profile uploader, auth-diagnostics uploader,
session-analytics halves of feedback; local zero-egress observability
preserved in new kigi-log crate (unified log, --debug firehose,
subsystem file logs, opt-in instrumentation)
- announcements (crate, remote-settings fields, TUI surfaces)
- plugin marketplace (crate, sources/browse/CTA/extensions-modal tab);
direct plugin install/uninstall/update via kigi-agent git_install kept
- relay/gateway/assets endpoints and features (agent relay, headless
relay transport, gateway bridge, LeaderEnvUrls); leader IPC socket now
~/.kigi/leader.sock + KIGI_LEADER_SOCKET, no ws-url derivation
- functional types rehomed instead of deleted: PermissionMode ->
kigi-config-types, McpInitStrategy -> kigi-mcp, PrCreationSource ->
session signals, TerminalDiagnostics -> kigi-pager-render, agent_id ->
shell util
Endpoints
- kigi-env rewritten: single production KigiEndpoints {coding_api_base_url
https://api.kimi.com/coding/v1 (KIGI_CODE_BASE_URL), oauth_host
https://auth.kimi.com (KIGI_OAUTH_HOST), update_base_url (GitHub
Releases API), upgrade_page_url}; GrokBuildEnvironment enum deleted
Toolchain & workspace hygiene
- Rust 1.97.0 pinned; edition 2024; full cargo update; git2 hoisted to
workspace at 0.21 (Option->Result API migration), quick-xml 0.41
- Root Cargo.toml hand-maintained (PRD §8.1): version 0.1.0 inherited by
all members, members sorted, unused deps pruned
- cargo-deny advisories gate (deny.toml with documented transitive
exceptions); CI workflow (check/clippy/fmt/deny/test, macOS+Linux)
- cross-crate test seams re-gated behind `test-support` cargo feature;
insta snapshot baselines renamed to the kigi_tui prefix
- clippy --workspace --all-targets: zero warnings; fmt clean
Fixes surfaced by the port
- updater probe/installer divergence (bin/kigi vs bin/grok symlink set)
- idle model-metadata refresh dead under KIGI_CODE_BASE_URL override
(new is_effective_coding_endpoint_url, loopback+override aware)
- macOS symlinked-TMPDIR fixture canonicalization (foreign_sessions,
fast-worktree); RSS measurement tests serialized via serial_test
Docs & legal (Apache §4)
- NOTICE added (upstream attribution + change statement); THIRD-PARTY
notices sustained; kigi-tools ported-code notices extended; README,
CONTRIBUTING, SECURITY, AGENTS.md rewritten
Out of scope for M0 (tracked): Kimi auth/inference (M1), search/fetch,
command parity, config import (M2), Computer Hub excision & final
brand-token sweep (M2), distribution & self-update rewrite (M3).
6158 lines
280 KiB
Rust
6158 lines
280 KiB
Rust
use std::collections::{HashMap, HashSet};
|
|
use std::path::{Path, PathBuf};
|
|
use std::sync::Arc;
|
|
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
|
|
use std::time::{Duration, Instant};
|
|
/// The binary version of the currently running leader process.
|
|
///
|
|
/// Compared against each registering client's `ClientCapabilities::client_version`
|
|
/// to detect mismatches early and surface a structured ACP notification.
|
|
/// In development builds where `VERSION_WITH_COMMIT` is not set, this is
|
|
/// `"unknown"` and version-mismatch detection is disabled (no notification sent).
|
|
const LEADER_VERSION: &str = match option_env!("VERSION_WITH_COMMIT") {
|
|
Some(v) => v,
|
|
None => "unknown",
|
|
};
|
|
use super::protocol::{
|
|
ClientCapabilities, ClientId, ClientMessage, ClientMode, ControlCommand, ControlPayload,
|
|
LEADER_PROTOCOL_VERSION, LeaderCapabilities, ProtocolError, ServerMessage, read_message,
|
|
write_message,
|
|
};
|
|
use super::transport::{LeaderListener, LeaderStream};
|
|
use crate::agent::activity::AgentActivity;
|
|
use crate::auth::AuthManager;
|
|
use crate::cpu_profile::{
|
|
ControlError, ControlErrorCode, CpuProfileManager, CpuProfileStartOptions, CpuProfileStatus,
|
|
ShutdownStopDisposition,
|
|
};
|
|
use agent_client_protocol::AGENT_METHOD_NAMES;
|
|
use kanal::{AsyncReceiver, AsyncSender};
|
|
use kigi_computer_hub_sdk::{AuthCredential, AuthIdentity, AuthProvider};
|
|
use kigi_workspace::WorkspaceHandle;
|
|
use parking_lot::Mutex;
|
|
use tokio::sync::{mpsc, watch};
|
|
use tokio_util::sync::CancellationToken;
|
|
use tracing::{debug, error, info, trace, warn};
|
|
const REGISTRATION_TIMEOUT: Duration = Duration::from_secs(30);
|
|
/// Separator for namespacing request IDs. Using pipe character which is:
|
|
/// - Valid in JSON strings (no escaping needed)
|
|
/// - Unlikely to appear in typical JSON-RPC IDs (usually numbers or UUIDs)
|
|
const ID_NAMESPACE_SEP: char = '|';
|
|
/// Cap on live notifications buffered per in-flight `session/load` (see
|
|
/// `load_live_buffer`). A normal load resolves in well under a second, so the
|
|
/// buffer is tiny; this bound just prevents unbounded growth if a load stalls.
|
|
/// On overflow we stop buffering and forward live normally (correctness of the
|
|
/// transcript is preserved by the client's eventId dedup; only the ordering
|
|
/// nicety is lost in this degenerate case).
|
|
const MAX_BUFFERED_LIVE_PER_LOAD: usize = 4096;
|
|
enum ServerEvent {
|
|
Disconnected(ClientId),
|
|
Registered(ClientId, ClientMode, ClientCapabilities, String),
|
|
Message(ClientId, ClientMessage),
|
|
}
|
|
/// A live notification buffered during an in-flight `session/load`: the
|
|
/// shared payload plus its `event_seq` (computed at buffer time, when the
|
|
/// message is already parsed, so the post-load flush never re-parses).
|
|
type BufferedLive = (Arc<str>, Option<u64>);
|
|
/// Message queued to a client handler task.
|
|
///
|
|
/// ACP payloads are by far the hot path (every chunk of every session fans out
|
|
/// to every subscriber), so they ride as a shared `Arc<str>`: the routing loop
|
|
/// pays one refcount bump per recipient instead of a full `String` clone, and
|
|
/// the live-load buffer / interaction cache share the same allocation. The
|
|
/// handler serializes the wire envelope via [`ServerMessageRef`] without ever
|
|
/// materializing an owned `ServerMessage::Acp`.
|
|
#[derive(Debug, Clone)]
|
|
enum ClientOutbound {
|
|
/// An ACP payload, shared (refcounted) across fan-out targets.
|
|
Acp(Arc<str>),
|
|
/// Everything else (registration, control results, ping, shutdown, errors).
|
|
Message(ServerMessage),
|
|
}
|
|
impl From<ServerMessage> for ClientOutbound {
|
|
fn from(msg: ServerMessage) -> Self {
|
|
Self::Message(msg)
|
|
}
|
|
}
|
|
/// Serialize-only mirror of [`ServerMessage`]'s `Acp` variant that borrows the
|
|
/// payload, so the per-client writer can frame a shared `Arc<str>` without
|
|
/// copying it into an owned `ServerMessage`. Must stay wire-identical to
|
|
/// `ServerMessage::Acp` — asserted by the
|
|
/// `server_message_ref_is_wire_identical` test.
|
|
#[derive(serde::Serialize)]
|
|
#[serde(tag = "type", rename_all = "snake_case")]
|
|
enum ServerMessageRef<'a> {
|
|
Acp { payload: &'a str },
|
|
}
|
|
/// Write one [`ClientOutbound`] to a client connection.
|
|
async fn write_outbound<W>(writer: &mut W, msg: &ClientOutbound) -> Result<(), ProtocolError>
|
|
where
|
|
W: tokio::io::AsyncWrite + Unpin,
|
|
{
|
|
match msg {
|
|
ClientOutbound::Acp(payload) => {
|
|
write_message(writer, &ServerMessageRef::Acp { payload }).await
|
|
}
|
|
ClientOutbound::Message(m) => write_message(writer, m).await,
|
|
}
|
|
}
|
|
struct ClientState {
|
|
tx: AsyncSender<ClientOutbound>,
|
|
mode: ClientMode,
|
|
capabilities: ClientCapabilities,
|
|
/// The client type string from IPC registration (e.g., "grok-tui", "grok-code-extension").
|
|
/// Injected into `initialize` requests as `clientIdentifier` so the agent knows the real
|
|
/// client type even when multiple clients share one leader process.
|
|
client_type: String,
|
|
/// Set to `true` once the client's `initialize` request has been seen and had
|
|
/// `clientIdentifier` injected. Until `initialize` is observed (regardless of how many
|
|
/// earlier messages arrived), each ACP message is checked so we never miss a late
|
|
/// `initialize`. After it is seen once, we skip the per-message parse as an optimisation.
|
|
initialize_seen: bool,
|
|
/// Patch the next response's `modelState.currentModelId` to match `default_model`.
|
|
/// Set on outbound `initialize`, cleared after patching the response.
|
|
patch_initialize_model: bool,
|
|
/// Whether this client has completed IPC registration. Used to keep `client_count`
|
|
/// accurate — only registered clients are counted, so pre-registration connections
|
|
/// (which may time out) don't inflate the count and block auto-updates.
|
|
registered: bool,
|
|
}
|
|
#[derive(Debug, Clone)]
|
|
pub struct LeaderServerMetadata {
|
|
pub pid: u32,
|
|
pub socket_path: PathBuf,
|
|
pub lock_path: PathBuf,
|
|
pub socket_suffix: String,
|
|
pub leader_binary_version: String,
|
|
}
|
|
#[derive(Debug, Clone)]
|
|
pub struct LeaderServerControlState {
|
|
pub metadata: LeaderServerMetadata,
|
|
pub cpu_profile: Arc<Mutex<CpuProfileManager>>,
|
|
pub workspace: Arc<WorkspaceControl>,
|
|
}
|
|
impl LeaderServerControlState {
|
|
pub fn new(metadata: LeaderServerMetadata) -> Self {
|
|
Self {
|
|
metadata,
|
|
cpu_profile: Arc::new(Mutex::new(CpuProfileManager::new())),
|
|
workspace: Arc::new(WorkspaceControl::new(None)),
|
|
}
|
|
}
|
|
pub fn with_default_hub_url(mut self, default_hub_url: Option<String>) -> Self {
|
|
self.workspace = Arc::new(WorkspaceControl::new(default_hub_url));
|
|
self
|
|
}
|
|
fn leader_capabilities(&self) -> LeaderCapabilities {
|
|
let manager = self.cpu_profile.lock();
|
|
LeaderCapabilities {
|
|
control_v1: true,
|
|
runtime_cpu_profile: manager.runtime_cpu_profile(),
|
|
profile_formats: manager.profile_formats().to_vec(),
|
|
workspace_exposure: true,
|
|
relaunch_v1: true,
|
|
}
|
|
}
|
|
}
|
|
pub struct WorkspaceControl {
|
|
default_hub_url: Option<String>,
|
|
/// Hub credential, wired to the leader's `AuthManager` once auth is ready.
|
|
/// A `watch` so a starting leader (socket up, auth pending) can be awaited
|
|
/// instead of failing the command.
|
|
auth: tokio::sync::watch::Sender<Option<Arc<dyn AuthProvider>>>,
|
|
/// Serializes mutating commands (start/pause/resume/stop) so their long
|
|
/// awaits (drain, reconnect) never interleave.
|
|
lock: tokio::sync::Mutex<()>,
|
|
/// Current exposure, published for lock-free reads so `status` never
|
|
/// blocks behind an in-flight drain/reconnect.
|
|
exposure: arc_swap::ArcSwapOption<WorkspaceExposure>,
|
|
}
|
|
impl WorkspaceControl {
|
|
fn new(default_hub_url: Option<String>) -> Self {
|
|
Self {
|
|
default_hub_url,
|
|
auth: tokio::sync::watch::channel(None).0,
|
|
lock: tokio::sync::Mutex::new(()),
|
|
exposure: arc_swap::ArcSwapOption::empty(),
|
|
}
|
|
}
|
|
/// Wire the hub credential to the leader's shared `AuthManager` (sole
|
|
/// owner of refresh + persistence).
|
|
pub fn set_auth_manager(&self, auth_manager: Arc<AuthManager>) {
|
|
self.auth
|
|
.send_replace(Some(Arc::new(LeaderAuthProvider { auth_manager })));
|
|
}
|
|
}
|
|
impl std::fmt::Debug for WorkspaceControl {
|
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
|
f.debug_struct("WorkspaceControl")
|
|
.field("default_hub_url", &self.default_hub_url)
|
|
.finish_non_exhaustive()
|
|
}
|
|
}
|
|
/// Hub [`AuthProvider`] backed by the leader's `AuthManager`: returns the
|
|
/// current token at each connect/reconnect; never writes auth.json.
|
|
struct LeaderAuthProvider {
|
|
auth_manager: Arc<AuthManager>,
|
|
}
|
|
impl std::fmt::Debug for LeaderAuthProvider {
|
|
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
|
f.debug_struct("LeaderAuthProvider").finish_non_exhaustive()
|
|
}
|
|
}
|
|
impl AuthProvider for LeaderAuthProvider {
|
|
fn current(&self) -> AuthCredential {
|
|
let token = self
|
|
.auth_manager
|
|
.current_or_expired()
|
|
.map(|a| a.key)
|
|
.unwrap_or_default();
|
|
AuthCredential::bearer(token)
|
|
}
|
|
/// Owner identity from the leader's `AuthManager`, surfaced on the auth
|
|
/// provider instead of a separate auth.json
|
|
/// read. Mirrors the in-process path (`mvp_agent`): prefer `GrokAuth.team_id`
|
|
/// (what shell telemetry/snapshot use) mapped onto a `"Team"` principal so
|
|
/// team attribution is derived; otherwise pass principal fields through.
|
|
/// `None` when no credential is available (identity resolution never blocks).
|
|
fn identity(&self) -> Option<AuthIdentity> {
|
|
let a = self.auth_manager.current_or_expired()?;
|
|
Some(match a.team_id.filter(|t| !t.is_empty()) {
|
|
Some(team) => AuthIdentity {
|
|
user_id: a.user_id,
|
|
principal_type: Some("Team".to_string()),
|
|
principal_id: Some(team),
|
|
},
|
|
None => AuthIdentity {
|
|
user_id: a.user_id,
|
|
principal_type: a.principal_type,
|
|
principal_id: a.principal_id,
|
|
},
|
|
})
|
|
}
|
|
}
|
|
struct WorkspaceExposure {
|
|
handle: WorkspaceHandle,
|
|
hub_url: String,
|
|
cwd: PathBuf,
|
|
started_at: Instant,
|
|
paused: std::sync::atomic::AtomicBool,
|
|
}
|
|
/// Rewrite JSON-RPC request ID **in place** by prefixing with client ID to
|
|
/// avoid collisions.
|
|
///
|
|
/// Uses a pipe separator which is valid in JSON but unlikely to appear in
|
|
/// typical JSON-RPC IDs (which are usually numbers or UUIDs).
|
|
///
|
|
/// Only rewrites IDs for **requests** (messages with a "method" field).
|
|
/// Responses (messages with "result" or "error" but no "method") are left
|
|
/// untouched so the agent can match them to its pending requests.
|
|
///
|
|
/// Returns `Some((namespaced_id, original_id))` when the message is a request
|
|
/// carrying an ID (and `json` was mutated); `None` otherwise (no mutation).
|
|
/// The returned `namespaced_id` lets the caller key per-request state (e.g.
|
|
/// `pending_load_by_req`) without re-parsing the rewritten payload.
|
|
fn rewrite_request_id(
|
|
json: &mut serde_json::Value,
|
|
client_id: ClientId,
|
|
) -> Option<(String, serde_json::Value)> {
|
|
json.get("method")?;
|
|
let original_id = json.get("id").cloned()?;
|
|
let original_json = serde_json::to_string(&original_id).unwrap_or_default();
|
|
let namespaced_id = format!("{}{}{}", client_id.0, ID_NAMESPACE_SEP, original_json);
|
|
json["id"] = serde_json::json!(namespaced_id);
|
|
Some((namespaced_id, original_id))
|
|
}
|
|
/// Parse a namespaced response ID to find the target client, restoring the
|
|
/// original ID **in place**.
|
|
///
|
|
/// Expects format: "client_id|original_id_json" where original_id_json is the
|
|
/// JSON-serialized form of the original ID (preserving type information).
|
|
///
|
|
/// Returns `Some((client_id, namespaced_id))` if successful (`json` now
|
|
/// carries the restored original ID). The returned `namespaced_id` is the raw
|
|
/// pre-restore ID, so the caller can match per-request state (e.g.
|
|
/// `pending_load_by_req`) without re-parsing the original payload. On `None`,
|
|
/// `json` is untouched.
|
|
fn parse_response_id(json: &mut serde_json::Value) -> Option<(ClientId, String)> {
|
|
let id = json.get("id")?;
|
|
let id_str = id.as_str()?;
|
|
let (client_part, original_json) = id_str.split_once(ID_NAMESPACE_SEP)?;
|
|
let client_id: u64 = client_part.parse().ok()?;
|
|
let original_id: serde_json::Value = serde_json::from_str(original_json).ok()?;
|
|
let namespaced_id = id_str.to_string();
|
|
json["id"] = original_id;
|
|
Some((ClientId(client_id), namespaced_id))
|
|
}
|
|
/// Extract session_id from a message's params (for session-based routing).
|
|
fn extract_session_id(json: &serde_json::Value) -> Option<String> {
|
|
let params = json.get("params")?;
|
|
params
|
|
.get("sessionId")
|
|
.or_else(|| params.get("session_id"))
|
|
.and_then(|v| v.as_str())
|
|
.map(|s| s.to_string())
|
|
.or_else(|| {
|
|
params
|
|
.get("params")
|
|
.and_then(|inner| inner.get("sessionId").or_else(|| inner.get("session_id")))
|
|
.and_then(|v| v.as_str())
|
|
.map(|s| s.to_string())
|
|
})
|
|
}
|
|
/// Whether a payload is a `session/load` request. Used to start buffering live
|
|
/// broadcasts to the loading client until its replay completes (the
|
|
/// live-before-replay race, see `load_live_buffer`). Only `session/load`
|
|
/// triggers this — `session/new` creators receive everything live correctly.
|
|
fn is_session_load_request(json: &serde_json::Value) -> bool {
|
|
json.get("method")
|
|
.and_then(|m| m.as_str())
|
|
.is_some_and(|m| m == "session/load")
|
|
}
|
|
/// Extract the leader unicast target `ClientId` from a notification's
|
|
/// `params._meta["x.ai/leaderClientId"]`.
|
|
///
|
|
/// The agent stamps this onto every `session/load` replay notification (echoing
|
|
/// the id the leader injected into the load request) so the replay can be routed
|
|
/// back to ONLY the loading client instead of broadcasting to all subscribers.
|
|
/// Live (non-replay) turn deltas are never tagged, so they keep broadcasting.
|
|
fn extract_target_client_id(json: &serde_json::Value) -> Option<ClientId> {
|
|
let params = json.get("params")?;
|
|
params
|
|
.get("_meta")
|
|
.and_then(|m| m.get("x.ai/leaderClientId"))
|
|
.or_else(|| {
|
|
params
|
|
.get("params")
|
|
.and_then(|inner| inner.get("_meta"))
|
|
.and_then(|m| m.get("x.ai/leaderClientId"))
|
|
})
|
|
.and_then(|v| v.as_u64())
|
|
.map(ClientId)
|
|
}
|
|
/// Extract the monotonic `event_seq` counter from a notification's
|
|
/// `_meta.eventId` (format `"{sessionId}-{counter}"`). Mirrors the `_meta`
|
|
/// lookup in [`extract_target_client_id`] (also checks the ExtNotification
|
|
/// `params.params` nesting) and the suffix parse used by
|
|
/// `session::storage` and the client's `acp::meta`. Returns `None` for
|
|
/// notifications without an `eventId` (xAI one-shots / older shell).
|
|
fn event_seq_of(json: &serde_json::Value) -> Option<u64> {
|
|
let params = json.get("params")?;
|
|
let event_id = params
|
|
.get("_meta")
|
|
.and_then(|m| m.get("eventId"))
|
|
.or_else(|| {
|
|
params
|
|
.get("params")
|
|
.and_then(|inner| inner.get("_meta"))
|
|
.and_then(|m| m.get("eventId"))
|
|
})
|
|
.and_then(|v| v.as_str())?;
|
|
event_id.rsplit_once('-')?.1.parse::<u64>().ok()
|
|
}
|
|
/// Whether a payload is a machine-wide notification (no `sessionId`) that
|
|
/// must be **broadcast to every client** instead of falling through to the
|
|
/// last-active-client fallback:
|
|
///
|
|
/// - `x.ai/sessions/changed` — roster delta; every open dashboard must stay
|
|
/// in sync.
|
|
/// - `x.ai/models/update` — the model catalog changed (config.toml
|
|
/// `[model.*]`/`[models]` hot-reload, `models_cache.json` external write,
|
|
/// auth change, response-header etag refresh). Every connected client's
|
|
/// model picker must refresh, not just the most recently active one.
|
|
/// - `x.ai/mcp/servers_updated` — the MCP catalog resolved/changed (managed
|
|
/// connectors fetched in the background after `initialize`). Deliberately
|
|
/// session-agnostic on the wire (no `sessionId`, see
|
|
/// `extensions::mcp::notify_servers_updated`); the push fires seconds after
|
|
/// `initialize` returns, so last-active-client fallback routinely delivered
|
|
/// it to the wrong client (or dropped it) in multi-client leaders — managed
|
|
/// connectors then "disappeared" from every other client's `/mcp` view.
|
|
/// Broadcast is safe: the pager handler only debounce-refetches `mcp/list`
|
|
/// for agents with an open extensions modal.
|
|
///
|
|
/// (`x.ai/settings/update` stays non-broadcast — it carries auth/gate state.)
|
|
///
|
|
/// Matched via [`method_of`], NOT the raw top-level `method`: agent ext
|
|
/// notifications arrive `_`-prefixed on the wire (`_x.ai/sessions/changed`),
|
|
/// so a raw compare would miss the production form.
|
|
fn is_machine_wide_broadcast_notification(json: &serde_json::Value) -> bool {
|
|
matches!(
|
|
method_of(json),
|
|
Some("x.ai/sessions/changed" | "x.ai/models/update" | "x.ai/mcp/servers_updated")
|
|
)
|
|
}
|
|
/// Whether a payload is the `x.ai/scheduled_task_inject_prompt` notification.
|
|
///
|
|
/// This notification tells the receiving client to enqueue AND drive a
|
|
/// scheduled (`/loop`) cron prompt. Unlike ordinary `sessionId`-bearing
|
|
/// notifications (which fan out to every subscriber so each renders an
|
|
/// identical stream), it must be routed to the SINGLE session driver: if every
|
|
/// attached client received it, each would enqueue + try to drive the same cron
|
|
/// turn, duplicating it (phantom `#N` queue entries, competing drivers, stuck
|
|
/// turns). The other clients render the resulting turn from the broadcast
|
|
/// `session/update` deltas, exactly like any other turn the driver runs.
|
|
/// The namespaced method a leader payload carries, normalizing the two ext wire
|
|
/// forms the gateway produces:
|
|
/// - direct: `{"method":"x.ai/foo", ...}` -> `x.ai/foo`
|
|
/// - wrapped: `{"method":"_x.ai/foo","params":{"method":"x.ai/foo",...}}` -> `x.ai/foo`
|
|
///
|
|
/// Gateway-forwarded ext methods/notifications (`ext_method` / `ext_notification`
|
|
/// — e.g. `ask_user_question`, `exit_plan_mode`, `scheduled_task_inject_prompt`,
|
|
/// `session_notification`) arrive WRAPPED: a top-level `_`-prefixed method with
|
|
/// the real method + params nested one level under `params`. Plain methods
|
|
/// (`session/request_permission`, `session/update`, …) arrive direct. Anything
|
|
/// that classifies a payload by method name MUST use this — matching the raw
|
|
/// top-level `method` misses the wrapped form. See `interaction_inner_params`
|
|
/// for the matching params accessor.
|
|
fn method_of(json: &serde_json::Value) -> Option<&str> {
|
|
let top = json.get("method")?.as_str()?;
|
|
if let Some(stripped) = top.strip_prefix('_') {
|
|
return Some(
|
|
json.get("params")
|
|
.and_then(|p| p.get("method"))
|
|
.and_then(|m| m.as_str())
|
|
.unwrap_or(stripped),
|
|
);
|
|
}
|
|
Some(top)
|
|
}
|
|
/// The real params object for a payload, unwrapping the gateway ext wrapper:
|
|
/// for a wrapped ext (its `params` carries its own `method` + nested `params`)
|
|
/// the real params live at `params.params`; otherwise `params` is already real.
|
|
fn interaction_inner_params(json: &serde_json::Value) -> Option<&serde_json::Value> {
|
|
let params = json.get("params")?;
|
|
if params.get("method").is_some()
|
|
&& let Some(inner) = params.get("params")
|
|
{
|
|
Some(inner)
|
|
} else {
|
|
Some(params)
|
|
}
|
|
}
|
|
/// Whether a payload is the `x.ai/scheduled_task_inject_prompt` notification.
|
|
///
|
|
/// This notification tells the receiving client to enqueue AND drive a
|
|
/// scheduled (`/loop`) cron prompt. Unlike ordinary `sessionId`-bearing
|
|
/// notifications (which fan out to every subscriber so each renders an
|
|
/// identical stream), it must be routed to the SINGLE session driver: if every
|
|
/// attached client received it, each would enqueue + try to drive the same cron
|
|
/// turn, duplicating it (phantom `#N` queue entries, competing drivers, stuck
|
|
/// turns). The other clients render the resulting turn from the broadcast
|
|
/// `session/update` deltas, exactly like any other turn the driver runs.
|
|
fn is_scheduled_task_inject_prompt(json: &serde_json::Value) -> bool {
|
|
method_of(json) == Some("x.ai/scheduled_task_inject_prompt")
|
|
}
|
|
/// Whether a payload is a blocking *interaction* reverse-request — a tool
|
|
/// permission, `ask_user_question`, or plan-approval. Unlike other
|
|
/// reverse-requests (driver-only), these are **shared**: broadcast to every
|
|
/// subscriber so any client can render + answer the modal, first-answer-wins.
|
|
/// See `SHARED_INTERACTIVE_MODALS.md`.
|
|
fn is_interaction_request(json: &serde_json::Value) -> bool {
|
|
matches!(
|
|
method_of(json),
|
|
Some("session/request_permission" | "x.ai/ask_user_question" | "x.ai/exit_plan_mode")
|
|
)
|
|
}
|
|
/// Extract the `tool_call_id` an interaction reverse-request carries, so the
|
|
/// leader can cache it (keyed by id) for replay-on-attach and evict it on
|
|
/// `InteractionResolved`. The ext-methods (`ask_user_question` /
|
|
/// `exit_plan_mode`) carry it directly under (inner) `params`;
|
|
/// `request_permission` nests it under `toolCall`. Tolerant of the gateway
|
|
/// wrapper (via `interaction_inner_params`) and camel/snake spelling.
|
|
fn extract_interaction_tool_call_id(json: &serde_json::Value) -> Option<String> {
|
|
let params = interaction_inner_params(json)?;
|
|
if let Some(id) = params
|
|
.get("toolCallId")
|
|
.or_else(|| params.get("tool_call_id"))
|
|
.and_then(|v| v.as_str())
|
|
{
|
|
return Some(id.to_string());
|
|
}
|
|
let tc = params.get("toolCall").or_else(|| params.get("tool_call"))?;
|
|
tc.get("toolCallId")
|
|
.or_else(|| tc.get("tool_call_id"))
|
|
.or_else(|| tc.get("id"))
|
|
.and_then(|v| v.as_str())
|
|
.map(String::from)
|
|
}
|
|
/// If a payload is the `InteractionResolved` broadcast (an
|
|
/// `x.ai/session_notification` whose `update.sessionUpdate ==
|
|
/// "interaction_resolved"`), return its `tool_call_id` so the leader can evict
|
|
/// the cached interaction request (first-answer-wins). Tolerant of the gateway
|
|
/// wrapper and camel/snake spelling for the inner field.
|
|
fn extract_interaction_resolved_tool_call_id(json: &serde_json::Value) -> Option<String> {
|
|
if method_of(json) != Some("x.ai/session_notification") {
|
|
return None;
|
|
}
|
|
let update = interaction_inner_params(json)?.get("update")?;
|
|
if update.get("sessionUpdate").and_then(|v| v.as_str()) != Some("interaction_resolved") {
|
|
return None;
|
|
}
|
|
update
|
|
.get("tool_call_id")
|
|
.or_else(|| update.get("toolCallId"))
|
|
.and_then(|v| v.as_str())
|
|
.map(String::from)
|
|
}
|
|
/// Extract session_id from a prompt-complete notification.
|
|
fn extract_session_id_from_prompt_complete(json: &serde_json::Value) -> Option<String> {
|
|
let method = json.get("method")?.as_str()?;
|
|
if method != "x.ai/session/prompt_complete" {
|
|
return None;
|
|
}
|
|
json.get("params")?
|
|
.get("sessionId")
|
|
.or_else(|| json.get("params")?.get("session_id"))
|
|
.and_then(|v| v.as_str())
|
|
.map(|s| s.to_string())
|
|
}
|
|
/// Extract session_id from a response's result (for session/new and session/load responses).
|
|
/// This is used to track session ownership when the session is first created.
|
|
fn extract_session_id_from_result(json: &serde_json::Value) -> Option<String> {
|
|
let result = json.get("result")?;
|
|
result
|
|
.get("session_id")
|
|
.or_else(|| result.get("sessionId"))
|
|
.and_then(|v| v.as_str())
|
|
.map(|s| s.to_string())
|
|
}
|
|
#[derive(Debug)]
|
|
enum ChildSessionEvent {
|
|
Spawned(String),
|
|
Finished(String),
|
|
}
|
|
/// Extract child session lifecycle events from subagent notifications.
|
|
fn extract_child_session_event(json: &serde_json::Value) -> Option<ChildSessionEvent> {
|
|
let params = json.get("params")?;
|
|
let update = params
|
|
.get("update")
|
|
.or_else(|| params.get("params")?.get("update"))?;
|
|
let child_sid = update.get("child_session_id").and_then(|v| v.as_str())?;
|
|
match update.get("sessionUpdate")?.as_str()? {
|
|
"subagent_spawned" => Some(ChildSessionEvent::Spawned(child_sid.to_string())),
|
|
"subagent_finished" => Some(ChildSessionEvent::Finished(child_sid.to_string())),
|
|
_ => None,
|
|
}
|
|
}
|
|
/// Drop a finished child's route + driver and detach it from the child forest,
|
|
/// RE-PARENTING any still-live grandchildren onto the finished child's own
|
|
/// parent. Re-parenting (not cascade-prune) keeps a still-running grandchild
|
|
/// of a finished intermediate reachable from the root: `backfill_child_routes`
|
|
/// only follows forward edges, so a subtree left dangling under the removed
|
|
/// child would never be reached on a root `session/load`. A genuinely-dead leaf
|
|
/// (no surviving children) is simply removed.
|
|
///
|
|
/// The current parent is found by searching the forest — not taken from the
|
|
/// finish notification's sessionId — so a grandchild already re-parented by an
|
|
/// earlier intermediate finish is still detached from its correct edge.
|
|
fn prune_child_route(
|
|
child_sid: &str,
|
|
session_subscribers: &mut HashMap<String, HashSet<ClientId>>,
|
|
session_driver: &mut HashMap<String, ClientId>,
|
|
child_sessions: &mut HashMap<String, HashSet<String>>,
|
|
) {
|
|
session_subscribers.remove(child_sid);
|
|
session_driver.remove(child_sid);
|
|
let parent = child_sessions
|
|
.iter()
|
|
.find_map(|(p, kids)| kids.contains(child_sid).then(|| p.clone()));
|
|
if let Some(ref parent) = parent
|
|
&& let Some(kids) = child_sessions.get_mut(parent)
|
|
{
|
|
kids.remove(child_sid);
|
|
}
|
|
if let Some(grandchildren) = child_sessions.remove(child_sid)
|
|
&& let Some(ref parent) = parent
|
|
{
|
|
child_sessions
|
|
.entry(parent.clone())
|
|
.or_default()
|
|
.extend(grandchildren);
|
|
}
|
|
if let Some(parent) = parent
|
|
&& child_sessions.get(&parent).is_some_and(HashSet::is_empty)
|
|
{
|
|
child_sessions.remove(&parent);
|
|
}
|
|
}
|
|
/// Subscribe `client` to every live descendant of `parent` (walking the
|
|
/// parent→children index, depth-safe via a visited set) and give driverless
|
|
/// descendants the parent's driver. Child routes are otherwise spawn-time
|
|
/// snapshots, so without this a client that attaches to the parent AFTER a
|
|
/// subagent spawned (late attach, reconnect) never receives the child's live
|
|
/// updates.
|
|
fn backfill_child_routes(
|
|
parent: &str,
|
|
client: ClientId,
|
|
child_sessions: &HashMap<String, HashSet<String>>,
|
|
session_subscribers: &mut HashMap<String, HashSet<ClientId>>,
|
|
session_driver: &mut HashMap<String, ClientId>,
|
|
) {
|
|
let parent_driver = session_driver.get(parent).copied();
|
|
let mut stack: Vec<&str> = vec![parent];
|
|
let mut visited: HashSet<&str> = HashSet::new();
|
|
while let Some(sid) = stack.pop() {
|
|
let Some(children) = child_sessions.get(sid) else {
|
|
continue;
|
|
};
|
|
for child in children {
|
|
if !visited.insert(child.as_str()) {
|
|
continue;
|
|
}
|
|
session_subscribers
|
|
.entry(child.clone())
|
|
.or_default()
|
|
.insert(client);
|
|
if let Some(driver) = parent_driver {
|
|
session_driver.entry(child.clone()).or_insert(driver);
|
|
}
|
|
stack.push(child);
|
|
}
|
|
}
|
|
}
|
|
/// Inject client capabilities into a session/new request, **in place**.
|
|
///
|
|
/// If the payload is a session/new request:
|
|
/// - If the client has yolo_mode enabled, injects `yoloMode: true` into the request's `_meta` object.
|
|
/// - If the client has default_model set and the request doesn't already have a modelId,
|
|
/// injects `modelId` into the request's `_meta` object.
|
|
/// - Injects `clientIdentifier` so the agent can track which client owns each session
|
|
/// (used for scoping `yolo_mode_changed` broadcasts in leader mode).
|
|
///
|
|
/// Returns `true` when `json` was mutated.
|
|
fn inject_capabilities_into_session_new(
|
|
json: &mut serde_json::Value,
|
|
capabilities: &ClientCapabilities,
|
|
client_type: &str,
|
|
client_id: ClientId,
|
|
) -> bool {
|
|
let has_model = capabilities
|
|
.default_model
|
|
.as_ref()
|
|
.is_some_and(|m| !m.is_empty());
|
|
if !capabilities.yolo_mode
|
|
&& !capabilities.auto_mode
|
|
&& !has_model
|
|
&& client_type.is_empty()
|
|
&& !capabilities.code_nav_enabled
|
|
{
|
|
return false;
|
|
}
|
|
let method = json.get("method").and_then(|m| m.as_str()).unwrap_or("");
|
|
let is_session_new = method == AGENT_METHOD_NAMES.session_new;
|
|
let is_session_load = method == AGENT_METHOD_NAMES.session_load;
|
|
if !is_session_new && !is_session_load {
|
|
return false;
|
|
}
|
|
let mut mutated = false;
|
|
if let Some(params) = json.get_mut("params").and_then(|p| p.as_object_mut()) {
|
|
let meta = params
|
|
.entry("_meta")
|
|
.or_insert_with(|| serde_json::json!({}));
|
|
if let Some(meta_obj) = meta.as_object_mut() {
|
|
mutated = true;
|
|
if is_session_new && capabilities.yolo_mode && !meta_obj.contains_key("yoloMode") {
|
|
meta_obj.insert("yoloMode".to_string(), serde_json::json!(true));
|
|
debug!("Injected yoloMode=true into session/new request");
|
|
}
|
|
if capabilities.auto_mode
|
|
&& !capabilities.yolo_mode
|
|
&& !meta_obj.contains_key("autoMode")
|
|
{
|
|
meta_obj.insert("autoMode".to_string(), serde_json::json!(true));
|
|
debug!("Injected autoMode=true into session request");
|
|
}
|
|
if is_session_new
|
|
&& let Some(ref model_id) = capabilities.default_model
|
|
&& !model_id.is_empty()
|
|
&& !meta_obj.contains_key("modelId")
|
|
{
|
|
meta_obj.insert("modelId".to_string(), serde_json::json!(model_id));
|
|
debug!(model_id, "Injected modelId into session/new request");
|
|
}
|
|
if !client_type.is_empty() && !meta_obj.contains_key("clientIdentifier") {
|
|
meta_obj.insert(
|
|
"clientIdentifier".to_string(),
|
|
serde_json::json!(client_type),
|
|
);
|
|
}
|
|
if !meta_obj.contains_key("x.ai/leaderClientId") {
|
|
meta_obj.insert(
|
|
"x.ai/leaderClientId".to_string(),
|
|
serde_json::json!(client_id.0),
|
|
);
|
|
}
|
|
meta_obj.insert(
|
|
"codeNavEnabled".to_string(),
|
|
serde_json::json!(capabilities.code_nav_enabled),
|
|
);
|
|
debug!(
|
|
code_nav_enabled = capabilities.code_nav_enabled,
|
|
"Injected codeNavEnabled into session request"
|
|
);
|
|
meta_obj.insert(
|
|
"clientTerminal".to_string(),
|
|
serde_json::json!(capabilities.terminal),
|
|
);
|
|
meta_obj.insert(
|
|
"clientFsRead".to_string(),
|
|
serde_json::json!(capabilities.fs_read),
|
|
);
|
|
meta_obj.insert(
|
|
"clientFsWrite".to_string(),
|
|
serde_json::json!(capabilities.fs_write),
|
|
);
|
|
}
|
|
}
|
|
mutated
|
|
}
|
|
/// Inject client identity into an `initialize` request.
|
|
///
|
|
/// In leader mode, multiple clients (TUI, IDE extension, web) share one agent process.
|
|
/// The agent's `client_type` is set during `initialize` from `_meta.clientIdentifier`,
|
|
/// so the leader injects the IPC registration `client_type` to ensure the agent knows
|
|
/// the real client identity.
|
|
///
|
|
/// Only injects if `clientIdentifier` is not already present in `_meta` — respects
|
|
/// explicit client-provided values.
|
|
///
|
|
/// Mutates `json` in place. Returns `(mutated, was_initialize)`. The second
|
|
/// boolean is `true` only when the message was an `initialize` request,
|
|
/// allowing the caller to record that `initialize` has been seen.
|
|
fn inject_client_identity_into_initialize(
|
|
json: &mut serde_json::Value,
|
|
client_type: &str,
|
|
) -> (bool, bool) {
|
|
if client_type.is_empty() {
|
|
return (false, false);
|
|
}
|
|
let is_initialize = json
|
|
.get("method")
|
|
.and_then(|m| m.as_str())
|
|
.is_some_and(|m| m == AGENT_METHOD_NAMES.initialize);
|
|
if !is_initialize {
|
|
return (false, false);
|
|
}
|
|
let mut mutated = false;
|
|
if let Some(params) = json.get_mut("params").and_then(|p| p.as_object_mut()) {
|
|
let meta = params
|
|
.entry("_meta")
|
|
.or_insert_with(|| serde_json::json!({}));
|
|
if let Some(meta_obj) = meta.as_object_mut()
|
|
&& !meta_obj.contains_key("clientIdentifier")
|
|
{
|
|
meta_obj.insert(
|
|
"clientIdentifier".to_string(),
|
|
serde_json::json!(client_type),
|
|
);
|
|
mutated = true;
|
|
debug!(
|
|
client_type,
|
|
"Injected clientIdentifier into initialize request"
|
|
);
|
|
}
|
|
}
|
|
(mutated, true)
|
|
}
|
|
/// Extract yolo_mode change from x.ai/yolo_mode_changed notification.
|
|
///
|
|
/// Returns Some(yolo_mode) if this is a yolo mode change notification.
|
|
fn extract_yolo_mode_change(json: &serde_json::Value) -> Option<bool> {
|
|
let method = json.get("method")?.as_str()?;
|
|
if method != "x.ai/yolo_mode_changed" {
|
|
return None;
|
|
}
|
|
let params = json.get("params")?;
|
|
params.get("yolo_mode").and_then(|v| v.as_bool())
|
|
}
|
|
/// Extract the auto-mode intent from an `x.ai/yolo_mode_changed` notification, so
|
|
/// the leader can keep `ClientCapabilities.auto_mode` fresh the same way it tracks
|
|
/// `yolo_mode`. Without this, a stale connect-time `auto_mode` capability would be
|
|
/// injected into later `session/new` requests, re-enabling Auto after the user opted
|
|
/// out. Returns `None` when the notification doesn't change auto state.
|
|
fn extract_auto_mode_change(json: &serde_json::Value) -> Option<bool> {
|
|
let method = json.get("method")?.as_str()?;
|
|
if method != "x.ai/yolo_mode_changed" {
|
|
return None;
|
|
}
|
|
let params = json.get("params")?;
|
|
if let Some(b) = params.get("auto_mode").and_then(|v| v.as_bool()) {
|
|
return Some(b);
|
|
}
|
|
match params.get("permission_mode").and_then(|v| v.as_str()) {
|
|
Some("auto") => Some(true),
|
|
Some("always-approve" | "ask" | "default") => Some(false),
|
|
_ => None,
|
|
}
|
|
}
|
|
/// Inject `clientIdentifier` into a `yolo_mode_changed` notification's params.
|
|
///
|
|
/// In leader mode, multiple clients share one agent. Without this injection, the agent
|
|
/// can't tell which client sent the yolo toggle and updates ALL sessions. With the
|
|
/// `clientIdentifier` in params, the agent scopes the update to only sessions owned
|
|
/// by the sending client.
|
|
///
|
|
/// Mutates `json` in place; returns `true` when mutated.
|
|
fn inject_client_identity_into_yolo_notification(
|
|
json: &mut serde_json::Value,
|
|
client_type: &str,
|
|
) -> bool {
|
|
if client_type.is_empty() {
|
|
return false;
|
|
}
|
|
let is_yolo = json
|
|
.get("method")
|
|
.and_then(|m| m.as_str())
|
|
.is_some_and(|m| m == "x.ai/yolo_mode_changed");
|
|
if !is_yolo {
|
|
return false;
|
|
}
|
|
let mut mutated = false;
|
|
if let Some(params) = json.get_mut("params").and_then(|p| p.as_object_mut()) {
|
|
params.insert(
|
|
"clientIdentifier".to_string(),
|
|
serde_json::json!(client_type),
|
|
);
|
|
mutated = true;
|
|
debug!(
|
|
client_type,
|
|
"Injected clientIdentifier into yolo_mode_changed notification"
|
|
);
|
|
}
|
|
mutated
|
|
}
|
|
/// Build a JSON-RPC error response for requests that arrive before the leader is ready.
|
|
///
|
|
/// Returns `Some(payload)` when the message has an `id` field (i.e. is a request),
|
|
/// so the client gets a structured response it can act on instead of hanging.
|
|
/// Returns `None` for notifications (no `id`) — those are silently dropped.
|
|
fn make_leader_starting_error(json: &serde_json::Value) -> Option<String> {
|
|
let id = json.get("id").filter(|v| !v.is_null()).cloned()?;
|
|
let response = serde_json::json!(
|
|
{ "jsonrpc" : "2.0", "id" : id, "error" : { "code" : - 32002, "message" :
|
|
"leader_starting", "data" :
|
|
"Leader is still initializing (auth/prefetch in progress). Retry shortly." } }
|
|
);
|
|
Some(response.to_string())
|
|
}
|
|
/// Choose the bytes forwarded to the agent: the re-serialized `json` when an
|
|
/// injection/rewrite mutated it, the original `payload` verbatim otherwise
|
|
/// (including non-JSON payloads, which are never parsed or re-serialized).
|
|
fn select_outbound_payload(
|
|
json: Option<&serde_json::Value>,
|
|
payload_mutated: bool,
|
|
payload: String,
|
|
) -> String {
|
|
match json {
|
|
Some(j) if payload_mutated => j.to_string(),
|
|
_ => payload,
|
|
}
|
|
}
|
|
/// Patch the `initialize` response so `meta.modelState.currentModelId` reflects the
|
|
/// client's `default_model` instead of the agent's global `current_model_id`.
|
|
///
|
|
/// Without this the TUI briefly shows the agent's startup default then jumps to the
|
|
/// client's preferred model once the first `session/new` response arrives.
|
|
///
|
|
/// Mutates `json` in place; returns `true` when patched.
|
|
fn patch_initialize_response_model(
|
|
json: &mut serde_json::Value,
|
|
default_model: &Option<String>,
|
|
) -> bool {
|
|
let Some(model) = default_model.as_ref().filter(|m| !m.is_empty()) else {
|
|
return false;
|
|
};
|
|
let needs_patch = json
|
|
.pointer("/result/meta/modelState/currentModelId")
|
|
.and_then(|v| v.as_str())
|
|
.is_some_and(|current| current != model.as_str());
|
|
if needs_patch {
|
|
json["result"]["meta"]["modelState"]["currentModelId"] =
|
|
serde_json::Value::String(model.clone());
|
|
debug!(patched_model = % model, "Patched initialize response currentModelId");
|
|
return true;
|
|
}
|
|
false
|
|
}
|
|
/// Extract model ID from a `session/setModel` request (for keeping `default_model` in sync).
|
|
fn extract_model_id_from_set_model(json: &serde_json::Value) -> Option<String> {
|
|
let method = json.get("method")?.as_str()?;
|
|
if method != AGENT_METHOD_NAMES.session_set_model {
|
|
return None;
|
|
}
|
|
let params = json.get("params")?;
|
|
params
|
|
.get("modelId")
|
|
.or_else(|| params.get("model_id"))
|
|
.and_then(|v| v.as_str())
|
|
.filter(|s| !s.is_empty())
|
|
.map(|s| s.to_string())
|
|
}
|
|
fn cpu_profile_status_payload(status: CpuProfileStatus) -> ControlPayload {
|
|
match status {
|
|
CpuProfileStatus::Inactive => ControlPayload::CpuProfileStatus {
|
|
active: false,
|
|
stopping: false,
|
|
started_at: None,
|
|
svg_path: None,
|
|
frequency_hz: None,
|
|
},
|
|
CpuProfileStatus::Active {
|
|
started_at,
|
|
svg_path,
|
|
frequency_hz,
|
|
} => ControlPayload::CpuProfileStatus {
|
|
active: true,
|
|
stopping: false,
|
|
started_at: Some(started_at),
|
|
svg_path: Some(svg_path),
|
|
frequency_hz: Some(frequency_hz),
|
|
},
|
|
CpuProfileStatus::Stopping {
|
|
started_at,
|
|
svg_path,
|
|
frequency_hz,
|
|
} => ControlPayload::CpuProfileStatus {
|
|
active: false,
|
|
stopping: true,
|
|
started_at: Some(started_at),
|
|
svg_path: Some(svg_path),
|
|
frequency_hz: Some(frequency_hz),
|
|
},
|
|
}
|
|
}
|
|
fn leader_info_payload(control_state: &LeaderServerControlState) -> ControlPayload {
|
|
let manager = control_state.cpu_profile.lock();
|
|
let status = manager.status();
|
|
let (cpu_profile_active, cpu_profile_stopping, profile_started_at) = match status {
|
|
CpuProfileStatus::Inactive => (false, false, None),
|
|
CpuProfileStatus::Active { started_at, .. } => (true, false, Some(started_at)),
|
|
CpuProfileStatus::Stopping { started_at, .. } => (false, true, Some(started_at)),
|
|
};
|
|
ControlPayload::LeaderInfo {
|
|
pid: control_state.metadata.pid,
|
|
socket_path: control_state.metadata.socket_path.clone(),
|
|
lock_path: control_state.metadata.lock_path.clone(),
|
|
socket_suffix: control_state.metadata.socket_suffix.clone(),
|
|
leader_protocol_version: LEADER_PROTOCOL_VERSION,
|
|
leader_binary_version: control_state.metadata.leader_binary_version.clone(),
|
|
profiling_supported: manager.runtime_cpu_profile(),
|
|
profiling_compiled_in: manager.profiling_compiled_in(),
|
|
cpu_profile_active,
|
|
cpu_profile_stopping,
|
|
profile_started_at,
|
|
profile_formats: manager.profile_formats().to_vec(),
|
|
}
|
|
}
|
|
const PROD_COMPUTER_HUB_URL: &str = "wss://computer-hub.kigi.com/v1/tools";
|
|
const WORKSPACE_DRAIN_TIMEOUT: Duration = Duration::from_secs(10);
|
|
fn workspace_err(message: impl Into<String>) -> ControlError {
|
|
ControlError {
|
|
code: ControlErrorCode::InternalError,
|
|
message: message.into(),
|
|
details: None,
|
|
}
|
|
}
|
|
/// Resolve the hub credential, waiting if the leader is still wiring auth
|
|
/// (the IPC socket comes up first). Resolves the instant auth is wired or the
|
|
/// leader cancels — event-driven, no timeout.
|
|
async fn wait_for_leader_auth(
|
|
ws: &WorkspaceControl,
|
|
cancel: &CancellationToken,
|
|
) -> Result<Arc<dyn AuthProvider>, ControlError> {
|
|
let mut rx = ws.auth.subscribe();
|
|
tokio::select! {
|
|
result = rx.wait_for(| v | v.is_some()) => match result { Ok(guard) => Ok(guard
|
|
.clone().expect("waited for Some")), Err(_) =>
|
|
Err(workspace_err("leader is shutting down; cannot expose workspace to the hub",)),
|
|
}, _ = cancel.cancelled() =>
|
|
Err(workspace_err("leader is shutting down; cannot expose workspace to the hub",)),
|
|
}
|
|
}
|
|
fn workspace_server_id() -> String {
|
|
let raw = gethostname::gethostname()
|
|
.to_string_lossy()
|
|
.to_ascii_lowercase();
|
|
let sanitized: String = raw
|
|
.chars()
|
|
.map(|c| {
|
|
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
|
|
c
|
|
} else {
|
|
'-'
|
|
}
|
|
})
|
|
.collect();
|
|
let name = sanitized.trim_matches('-');
|
|
if name.is_empty() {
|
|
"grok-workspace".to_string()
|
|
} else {
|
|
name.to_string()
|
|
}
|
|
}
|
|
async fn drain_and_disconnect(handle: &WorkspaceHandle) {
|
|
let tracker = handle.activity_tracker().clone();
|
|
tracker.set_draining();
|
|
if tokio::time::timeout(WORKSPACE_DRAIN_TIMEOUT, tracker.wait_until_drained())
|
|
.await
|
|
.is_err()
|
|
{
|
|
warn!(
|
|
active = tracker.total_active(),
|
|
"workspace drain timed out; disconnecting hub anyway"
|
|
);
|
|
}
|
|
handle.shutdown_hub().await;
|
|
}
|
|
fn build_workspace_status(
|
|
metadata: &LeaderServerMetadata,
|
|
exposure: Option<&WorkspaceExposure>,
|
|
) -> ControlPayload {
|
|
match exposure {
|
|
None => ControlPayload::WorkspaceStatus {
|
|
state: "none".to_string(),
|
|
hub_url: None,
|
|
cwd: None,
|
|
uptime_ms: 0,
|
|
active_tool_calls: 0,
|
|
sessions: Vec::new(),
|
|
pid: metadata.pid,
|
|
},
|
|
Some(exp) => {
|
|
let snapshot = exp.handle.activity_tracker().snapshot();
|
|
let mut sessions = exp.handle.session_ids();
|
|
sessions.sort();
|
|
ControlPayload::WorkspaceStatus {
|
|
state: if exp.paused.load(std::sync::atomic::Ordering::Relaxed) {
|
|
"paused"
|
|
} else {
|
|
"running"
|
|
}
|
|
.to_string(),
|
|
hub_url: Some(exp.hub_url.clone()),
|
|
cwd: Some(exp.cwd.display().to_string()),
|
|
uptime_ms: exp.started_at.elapsed().as_millis() as u64,
|
|
active_tool_calls: snapshot.active_tool_calls,
|
|
sessions,
|
|
pid: metadata.pid,
|
|
}
|
|
}
|
|
}
|
|
}
|
|
async fn handle_workspace_start(
|
|
control_state: LeaderServerControlState,
|
|
hub_url: Option<String>,
|
|
cwd: String,
|
|
cancel: CancellationToken,
|
|
) -> Result<ControlPayload, ControlError> {
|
|
let ws = &control_state.workspace;
|
|
let url_str = hub_url
|
|
.filter(|u| !u.trim().is_empty())
|
|
.or_else(|| ws.default_hub_url.clone())
|
|
.unwrap_or_else(|| PROD_COMPUTER_HUB_URL.to_string());
|
|
let url = url::Url::parse(&url_str)
|
|
.map_err(|e| workspace_err(format!("invalid hub url {url_str}: {e}")))?;
|
|
let cwd_path = PathBuf::from(&cwd);
|
|
let _serialize = ws.lock.lock().await;
|
|
if let Some(existing) = ws.exposure.load_full()
|
|
&& !existing.paused.load(Ordering::Relaxed)
|
|
&& existing.cwd == cwd_path
|
|
&& existing.hub_url == url_str
|
|
{
|
|
return Ok(build_workspace_status(
|
|
&control_state.metadata,
|
|
Some(existing.as_ref()),
|
|
));
|
|
}
|
|
let allow_insecure_ws =
|
|
url.scheme() == "ws" && matches!(url.host_str(), Some("localhost" | "127.0.0.1" | "::1"));
|
|
let status_config = kigi_workspace::StatusConfig::from_env();
|
|
let alpha_test_key = None;
|
|
let auth = wait_for_leader_auth(ws, &cancel).await?;
|
|
let server_id = workspace_server_id();
|
|
let metadata = serde_json::json!(
|
|
{ "source" : "grok-workspace", "hostname" : gethostname::gethostname()
|
|
.to_string_lossy(), "cwd" : cwd_path.display().to_string(), }
|
|
);
|
|
crate::agent::folder_trust::resolve_and_record(&cwd_path, None, false);
|
|
let project_lsp_trusted = crate::agent::folder_trust::project_scope_allowed(&cwd_path);
|
|
let handle = kigi_workspace::connect_local_workspace(
|
|
cwd_path.clone(),
|
|
url,
|
|
auth,
|
|
Some(metadata),
|
|
Some(server_id),
|
|
alpha_test_key,
|
|
allow_insecure_ws,
|
|
status_config,
|
|
project_lsp_trusted,
|
|
None,
|
|
None,
|
|
false,
|
|
false,
|
|
)
|
|
.await
|
|
.map_err(|e| workspace_err(format!("failed to connect workspace to hub: {e}")))?;
|
|
let exposure = Arc::new(WorkspaceExposure {
|
|
handle,
|
|
hub_url: url_str,
|
|
cwd: cwd_path,
|
|
started_at: Instant::now(),
|
|
paused: AtomicBool::new(false),
|
|
});
|
|
let payload = build_workspace_status(&control_state.metadata, Some(exposure.as_ref()));
|
|
if let Some(old) = ws.exposure.swap(Some(exposure)) {
|
|
drain_and_disconnect(&old.handle).await;
|
|
}
|
|
Ok(payload)
|
|
}
|
|
async fn handle_workspace_pause(
|
|
control_state: LeaderServerControlState,
|
|
) -> Result<ControlPayload, ControlError> {
|
|
let ws = &control_state.workspace;
|
|
let _serialize = ws.lock.lock().await;
|
|
let Some(exp) = ws.exposure.load_full() else {
|
|
return Err(workspace_err("no workspace exposure is running"));
|
|
};
|
|
if !exp.paused.load(Ordering::Relaxed) {
|
|
drain_and_disconnect(&exp.handle).await;
|
|
exp.paused.store(true, Ordering::Relaxed);
|
|
}
|
|
Ok(build_workspace_status(
|
|
&control_state.metadata,
|
|
Some(exp.as_ref()),
|
|
))
|
|
}
|
|
async fn handle_workspace_resume(
|
|
control_state: LeaderServerControlState,
|
|
) -> Result<ControlPayload, ControlError> {
|
|
let ws = &control_state.workspace;
|
|
let _serialize = ws.lock.lock().await;
|
|
let Some(exp) = ws.exposure.load_full() else {
|
|
return Err(workspace_err("no workspace exposure is running"));
|
|
};
|
|
if exp.paused.load(Ordering::Relaxed) {
|
|
exp.handle.activity_tracker().set_active();
|
|
if let Err(e) = exp.handle.connect_hub().await {
|
|
exp.handle.activity_tracker().set_draining();
|
|
return Err(workspace_err(format!("failed to reconnect to hub: {e}")));
|
|
}
|
|
exp.paused.store(false, Ordering::Relaxed);
|
|
}
|
|
Ok(build_workspace_status(
|
|
&control_state.metadata,
|
|
Some(exp.as_ref()),
|
|
))
|
|
}
|
|
async fn handle_workspace_stop(
|
|
control_state: LeaderServerControlState,
|
|
) -> Result<ControlPayload, ControlError> {
|
|
let ws = &control_state.workspace;
|
|
let _serialize = ws.lock.lock().await;
|
|
if let Some(exp) = ws.exposure.swap(None) {
|
|
drain_and_disconnect(&exp.handle).await;
|
|
}
|
|
Ok(build_workspace_status(&control_state.metadata, None))
|
|
}
|
|
async fn handle_workspace_status(
|
|
control_state: LeaderServerControlState,
|
|
) -> Result<ControlPayload, ControlError> {
|
|
let exposure = control_state.workspace.exposure.load_full();
|
|
Ok(build_workspace_status(
|
|
&control_state.metadata,
|
|
exposure.as_deref(),
|
|
))
|
|
}
|
|
async fn finalize_workspace_on_shutdown(control_state: LeaderServerControlState) {
|
|
let ws = &control_state.workspace;
|
|
let _serialize = ws.lock.lock().await;
|
|
if let Some(exp) = ws.exposure.swap(None) {
|
|
info!("Draining workspace exposure on leader shutdown");
|
|
drain_and_disconnect(&exp.handle).await;
|
|
}
|
|
}
|
|
fn handle_control_command(
|
|
control_state: &LeaderServerControlState,
|
|
command: ControlCommand,
|
|
) -> Result<ControlPayload, ControlError> {
|
|
match command {
|
|
ControlCommand::GetLeaderInfo => Ok(leader_info_payload(control_state)),
|
|
ControlCommand::CpuProfileStatus => {
|
|
let manager = control_state.cpu_profile.lock();
|
|
Ok(cpu_profile_status_payload(manager.status()))
|
|
}
|
|
ControlCommand::StartCpuProfile {
|
|
output,
|
|
frequency_hz,
|
|
} => {
|
|
let mut manager = control_state.cpu_profile.lock();
|
|
let status = manager.start(CpuProfileStartOptions {
|
|
output: output.map(PathBuf::from),
|
|
frequency_hz,
|
|
})?;
|
|
match status {
|
|
CpuProfileStatus::Active {
|
|
started_at,
|
|
svg_path,
|
|
frequency_hz,
|
|
} => Ok(ControlPayload::CpuProfileStarted {
|
|
pid: control_state.metadata.pid,
|
|
svg_path,
|
|
frequency_hz,
|
|
started_at,
|
|
}),
|
|
CpuProfileStatus::Inactive => {
|
|
Ok(cpu_profile_status_payload(CpuProfileStatus::Inactive))
|
|
}
|
|
CpuProfileStatus::Stopping {
|
|
started_at,
|
|
svg_path,
|
|
frequency_hz,
|
|
} => Ok(cpu_profile_status_payload(CpuProfileStatus::Stopping {
|
|
started_at,
|
|
svg_path,
|
|
frequency_hz,
|
|
})),
|
|
}
|
|
}
|
|
ControlCommand::StopCpuProfile => {
|
|
unreachable!("StopCpuProfile must be handled asynchronously")
|
|
}
|
|
ControlCommand::WorkspaceStart { .. }
|
|
| ControlCommand::WorkspacePause
|
|
| ControlCommand::WorkspaceResume
|
|
| ControlCommand::WorkspaceStop
|
|
| ControlCommand::WorkspaceStatus => {
|
|
unreachable!("workspace control commands are handled asynchronously")
|
|
}
|
|
ControlCommand::RelaunchForUpdate { .. } => {
|
|
unreachable!("RelaunchForUpdate must be handled asynchronously")
|
|
}
|
|
}
|
|
}
|
|
async fn handle_stop_cpu_profile(
|
|
control_state: LeaderServerControlState,
|
|
) -> Result<ControlPayload, ControlError> {
|
|
let stop_handle = {
|
|
let mut manager = control_state.cpu_profile.lock();
|
|
manager.take_stop_handle()?
|
|
};
|
|
let pid = control_state.metadata.pid;
|
|
let result = tokio::task::spawn_blocking(move || stop_handle.finish()).await;
|
|
control_state.cpu_profile.lock().complete_stop();
|
|
let result = result.map_err(|join_error| ControlError {
|
|
code: ControlErrorCode::InternalError,
|
|
message: "CPU profile stop task failed".to_string(),
|
|
details: Some(serde_json::json!({ "error" : join_error.to_string() })),
|
|
})??;
|
|
Ok(ControlPayload::CpuProfileStopped {
|
|
pid,
|
|
svg_path: result.svg_path,
|
|
started_at: result.started_at,
|
|
stopped_at: result.stopped_at,
|
|
})
|
|
}
|
|
async fn finalize_cpu_profile_on_shutdown(control_state: LeaderServerControlState) {
|
|
let (disposition, stop_handle, mut stop_completion_rx) = {
|
|
let mut manager = control_state.cpu_profile.lock();
|
|
let disposition = manager.shutdown_stop_disposition();
|
|
let stop_completion_rx = manager.subscribe_stop_completion();
|
|
let stop_handle = match manager.take_shutdown_stop_handle() {
|
|
Ok(stop_handle) => stop_handle,
|
|
Err(error) => {
|
|
warn!(
|
|
error = % error,
|
|
"Failed to prepare active CPU profile for leader shutdown"
|
|
);
|
|
return;
|
|
}
|
|
};
|
|
(disposition, stop_handle, stop_completion_rx)
|
|
};
|
|
let Some(disposition) = disposition else {
|
|
return;
|
|
};
|
|
let Some(stop_handle) = stop_handle else {
|
|
match disposition {
|
|
ShutdownStopDisposition::AlreadyStopping => {
|
|
debug!(
|
|
"CPU profile stop already in progress during leader shutdown; waiting for in-flight finalization task"
|
|
);
|
|
if !*stop_completion_rx.borrow() && stop_completion_rx.changed().await.is_err() {
|
|
warn!("CPU profile stop completion channel closed during leader shutdown wait");
|
|
}
|
|
}
|
|
ShutdownStopDisposition::StartedShutdownStop => {
|
|
warn!("Expected shutdown CPU profile stop handle but none was available");
|
|
}
|
|
}
|
|
return;
|
|
};
|
|
let result = tokio::task::spawn_blocking(move || stop_handle.finish()).await;
|
|
control_state.cpu_profile.lock().complete_stop();
|
|
match result {
|
|
Ok(Ok(result)) => {
|
|
info!(
|
|
path = % result.svg_path.display(), started_at = % result.started_at,
|
|
stopped_at = % result.stopped_at,
|
|
"Finalized active CPU profile during leader shutdown"
|
|
);
|
|
}
|
|
Ok(Err(error)) => {
|
|
warn!(
|
|
error = % error,
|
|
"Failed to finalize active CPU profile during leader shutdown"
|
|
);
|
|
}
|
|
Err(join_error) => {
|
|
warn!(error = % join_error, "CPU profile shutdown finalization task failed");
|
|
}
|
|
}
|
|
}
|
|
/// Bounded grace the leader waits for in-flight turns to finish before a
|
|
/// `RelaunchForUpdate` relaunch. If the agent is still busy when this elapses,
|
|
/// the leader exits anyway — the in-flight turn ends and the session reloads
|
|
/// cleanly (truncated at the last persisted boundary).
|
|
const RELAUNCH_GRACE: Duration = Duration::from_secs(5);
|
|
/// Bound on the post-drain session flush ([`AgentActivity::flush_all_sessions`]).
|
|
const RELAUNCH_FLUSH_GRACE: Duration = Duration::from_secs(5);
|
|
/// Total shutdown budget advertised to clients in the `Relaunching` ack:
|
|
/// idle-drain plus session flush.
|
|
const RELAUNCH_TOTAL_GRACE: Duration =
|
|
Duration::from_millis((RELAUNCH_GRACE.as_millis() + RELAUNCH_FLUSH_GRACE.as_millis()) as u64);
|
|
/// Poll cadence while waiting for the agent to go idle during the grace period.
|
|
const RELAUNCH_GRACE_POLL: Duration = Duration::from_millis(100);
|
|
/// Decide whether a [`ControlCommand::RelaunchForUpdate`] is accepted (the
|
|
/// synchronous half — kept separate from arming the drain so the caller can send
|
|
/// the `Relaunching` ack BEFORE the leader begins shutting down; otherwise an
|
|
/// idle leader can race the ack and the client sees a dropped control response).
|
|
///
|
|
/// Declines unless the target is strictly newer (directional guard) and no
|
|
/// relaunch is already in progress (idempotent across multiple clients). On
|
|
/// accept it sets `relaunching` so duplicate requests are declined.
|
|
fn decide_relaunch_for_update(
|
|
control_state: &LeaderServerControlState,
|
|
to_version: String,
|
|
relaunching: &AtomicBool,
|
|
) -> Result<ControlPayload, ControlError> {
|
|
let leader_version = control_state.metadata.leader_binary_version.clone();
|
|
if !super::leader_is_older_than(&leader_version, &to_version) {
|
|
debug!(
|
|
from_version = % leader_version, to_version = % to_version,
|
|
"RelaunchForUpdate declined: target is not strictly newer (or unparseable)"
|
|
);
|
|
return Ok(ControlPayload::RelaunchDeclined {
|
|
reason: format!("leader version {leader_version} is not older than {to_version}"),
|
|
});
|
|
}
|
|
if relaunching.swap(true, Ordering::SeqCst) {
|
|
return Ok(ControlPayload::RelaunchDeclined {
|
|
reason: "a relaunch is already in progress".to_string(),
|
|
});
|
|
}
|
|
info!(
|
|
from_version = % leader_version, to_version = % to_version, grace_ms =
|
|
RELAUNCH_TOTAL_GRACE.as_millis() as u64,
|
|
"RelaunchForUpdate accepted; draining before relaunch onto new binary"
|
|
);
|
|
Ok(ControlPayload::Relaunching {
|
|
from_version: leader_version,
|
|
to_version,
|
|
grace_ms: RELAUNCH_TOTAL_GRACE.as_millis() as u64,
|
|
})
|
|
}
|
|
/// Arm the bounded-grace drain for an accepted relaunch: wait up to
|
|
/// [`RELAUNCH_GRACE`] for the agent to go idle (`agent_busy` for IPC traffic
|
|
/// AND [`AgentActivity::is_busy`] for relay-driven turns / subagents), flush
|
|
/// every session actor, then set [`ShutdownReason::AutoUpdate`] and cancel —
|
|
/// the same exit path the auto-update checker uses. Must be called *after*
|
|
/// the `Relaunching` ack has been sent so the ack is delivered before
|
|
/// `ShuttingDown`.
|
|
fn spawn_relaunch_drain(
|
|
shutdown_tx: watch::Sender<super::protocol::ShutdownReason>,
|
|
cancel: CancellationToken,
|
|
agent_busy: Arc<AtomicBool>,
|
|
agent_activity: AgentActivity,
|
|
) {
|
|
tokio::spawn(async move {
|
|
let deadline = tokio::time::Instant::now() + RELAUNCH_GRACE;
|
|
while agent_busy.load(Ordering::Relaxed) || agent_activity.is_busy() {
|
|
if tokio::time::Instant::now() >= deadline {
|
|
warn!(
|
|
"RelaunchForUpdate grace elapsed while agent busy; relaunching anyway (in-flight turn ends)"
|
|
);
|
|
break;
|
|
}
|
|
tokio::select! {
|
|
_ = cancel.cancelled() => return, _ =
|
|
tokio::time::sleep(RELAUNCH_GRACE_POLL) => {}
|
|
}
|
|
}
|
|
agent_activity
|
|
.flush_all_sessions(RELAUNCH_FLUSH_GRACE)
|
|
.await;
|
|
let _ = shutdown_tx.send(super::protocol::ShutdownReason::AutoUpdate);
|
|
cancel.cancel();
|
|
});
|
|
}
|
|
#[derive(Debug, thiserror::Error)]
|
|
pub enum ServerError {
|
|
#[error("Failed to acquire leader lock: {0}")]
|
|
LockFailed(#[from] super::lock::LockError),
|
|
#[error("IO error: {0}")]
|
|
Io(#[from] std::io::Error),
|
|
}
|
|
/// Build the ACP notification payload for a leader/client version mismatch, or
|
|
/// return `None` when versions match or detection is disabled.
|
|
///
|
|
/// Extracted as a standalone function so the notification shape can be unit-tested
|
|
/// without running a full server.
|
|
fn make_version_mismatch_notification(
|
|
client_version: &str,
|
|
leader_version: &str,
|
|
) -> Option<String> {
|
|
if client_version == leader_version || leader_version == "unknown" {
|
|
return None;
|
|
}
|
|
Some(
|
|
serde_json::json!(
|
|
{ "jsonrpc" : "2.0", "method" : "x.ai/leader/version_mismatch", "params" : {
|
|
"clientVersion" : client_version, "leaderVersion" : leader_version, "message"
|
|
:
|
|
format!("Client version {client_version} differs from leader version \
|
|
{leader_version}. Restart the grok binary to use the same version.")
|
|
} }
|
|
)
|
|
.to_string(),
|
|
)
|
|
}
|
|
/// Run the leader IPC server.
|
|
///
|
|
/// The socket_path is where the Unix socket will be created.
|
|
/// Caller is responsible for:
|
|
/// 1. Cleaning up any stale socket file before calling this
|
|
/// 2. Acquiring the leader lock AFTER this function creates the socket
|
|
///
|
|
/// This ordering ensures that:
|
|
/// - Clients waiting for socket can connect as soon as we're ready
|
|
/// - The lock acquisition happens after we're actually listening
|
|
///
|
|
/// # Readiness gating
|
|
///
|
|
/// The `ready_rx` watch channel controls whether ACP messages are forwarded to the
|
|
/// agent. While `*ready_rx.borrow() == false` (leader still initializing):
|
|
/// - Client connections and IPC registrations are accepted normally.
|
|
/// - ACP requests (messages with an `id`) receive a structured `leader_starting`
|
|
/// JSON-RPC error so the client can retry rather than hang.
|
|
/// - ACP notifications (no `id`) are dropped with a trace log.
|
|
///
|
|
/// Once `ready_rx` is signaled `true` (auth + prefetch complete), all subsequent
|
|
/// ACP traffic is forwarded to the agent as normal.
|
|
///
|
|
/// # Arguments
|
|
///
|
|
/// * `socket_path` - Path for the Unix domain socket
|
|
/// * `acp_tx` - Channel to send ACP messages from clients to the agent
|
|
/// * `response_rx` - Channel to receive responses from the agent to route to clients
|
|
/// * `cancel` - Cancellation token for graceful shutdown
|
|
/// * `no_exit_on_disconnect` - If true, don't exit when all clients disconnect
|
|
/// * `client_count` - Atomic counter tracking the number of connected clients
|
|
/// * `agent_busy` - Atomic flag set while the agent has in-flight **IPC**
|
|
/// requests; relay-driven traffic never sets it
|
|
/// * `agent_activity` - Agent-derived activity view (running turns, parked
|
|
/// interactions, live subagents) consulted by the `RelaunchForUpdate` drain
|
|
/// alongside `agent_busy`, plus the pre-shutdown session flush
|
|
/// * `ready_rx` - Watch receiver; ACP forwarding is gated until this is `true`
|
|
/// * `shutdown_tx` - Watch sender for the shutdown reason. The server subscribes
|
|
/// its own receiver and reads it once when `cancel` fires (defaults to
|
|
/// [`ShutdownReason::Manual`]). The auto-update checker and the
|
|
/// [`ControlCommand::RelaunchForUpdate`] handler send [`ShutdownReason::AutoUpdate`]
|
|
/// before cancelling so clients see the real reason; senders must write before
|
|
/// cancelling.
|
|
/// * `leader_version_override` - If `Some`, overrides [`LEADER_VERSION`] for version
|
|
/// mismatch detection. Pass `None` in production; pass a test version string in
|
|
/// integration tests to bypass the `"unknown"` constant that appears in dev builds
|
|
/// where `VERSION_WITH_COMMIT` is not set.
|
|
/// * `control_state` - Leader-local control metadata and CPU profiling state
|
|
pub async fn run_leader_server(
|
|
socket_path: std::path::PathBuf,
|
|
acp_tx: mpsc::UnboundedSender<String>,
|
|
mut response_rx: mpsc::UnboundedReceiver<String>,
|
|
cancel: CancellationToken,
|
|
no_exit_on_disconnect: bool,
|
|
client_count: Arc<AtomicUsize>,
|
|
agent_busy: Arc<AtomicBool>,
|
|
agent_activity: AgentActivity,
|
|
ready_rx: watch::Receiver<bool>,
|
|
shutdown_tx: watch::Sender<super::protocol::ShutdownReason>,
|
|
leader_version_override: Option<&'static str>,
|
|
control_state: LeaderServerControlState,
|
|
) -> Result<(), ServerError> {
|
|
let _ = std::fs::remove_file(&socket_path);
|
|
let shutdown_reason_rx = shutdown_tx.subscribe();
|
|
let listener = LeaderListener::bind(&socket_path)?;
|
|
info!("Leader server listening");
|
|
let (event_tx, event_rx) = kanal::unbounded_async::<ServerEvent>();
|
|
let mut clients: HashMap<ClientId, ClientState> = HashMap::new();
|
|
let mut session_driver: HashMap<String, ClientId> = HashMap::new();
|
|
let mut session_subscribers: HashMap<String, std::collections::HashSet<ClientId>> =
|
|
HashMap::new();
|
|
let mut child_sessions: HashMap<String, HashSet<String>> = HashMap::new();
|
|
let mut pending_load_by_req: HashMap<String, (ClientId, String)> = HashMap::new();
|
|
let mut load_live_buffer: HashMap<(ClientId, String), Vec<BufferedLive>> = HashMap::new();
|
|
let mut orphan_replay_warned: HashSet<ClientId> = HashSet::new();
|
|
let mut load_replay_max_seq: HashMap<(ClientId, String), u64> = HashMap::new();
|
|
let mut interaction_requests: HashMap<String, HashMap<String, Arc<str>>> = HashMap::new();
|
|
let mut last_active_client: Option<ClientId> = None;
|
|
let mut had_clients = false;
|
|
let mut pending_requests: usize = 0;
|
|
let relaunching = Arc::new(AtomicBool::new(false));
|
|
loop {
|
|
tokio::select! {
|
|
biased; _ = cancel.cancelled() => { let reason = shutdown_reason_rx.borrow()
|
|
.clone(); info!(? reason, "Leader server shutting down (cancelled)"); if
|
|
pending_requests > 0 { debug!(pending_requests,
|
|
"Resetting agent_busy on shutdown"); agent_busy.store(false,
|
|
Ordering::Relaxed); } broadcast_shutdown(& clients, reason). await; break; }
|
|
accept_result = listener.accept() => { match accept_result { Ok((stream, _))
|
|
=> { had_clients = true; let client_id = ClientId::new(); let (tx, rx) =
|
|
kanal::unbounded_async(); clients.insert(client_id, ClientState { tx, mode :
|
|
ClientMode::Stdio, capabilities : ClientCapabilities::default(), client_type
|
|
: String::new(), initialize_seen : false, patch_initialize_model : false,
|
|
registered : false, }); spawn_client_handler(client_id, stream, rx, event_tx
|
|
.clone(), cancel.child_token(), ready_rx.clone(), control_state.clone(),); }
|
|
Err(e) => error!(error = % e, "Accept failed"), } } Ok(event) = event_rx
|
|
.recv() => { match event { ServerEvent::Registered(id, mode, capabilities,
|
|
client_type) => { if let Some(client) = clients.get_mut(& id) { client.mode =
|
|
mode; client.capabilities = capabilities; client.client_type = client_type;
|
|
client.registered = true; client_count.fetch_add(1, Ordering::Relaxed);
|
|
debug!(client_id = id.0, ? mode, yolo_mode = client.capabilities.yolo_mode,
|
|
client_type = % client.client_type, "Client registered");
|
|
kigi_log::unified_log::info("leader.client.registered", None,
|
|
Some(serde_json::json!({ "client_id" : id.0, "client_type" : client
|
|
.client_type, })),); let
|
|
effective_leader_version = leader_version_override.unwrap_or(LEADER_VERSION);
|
|
if let Some(ref cv) = client.capabilities.client_version && let Some(payload)
|
|
= make_version_mismatch_notification(cv.as_str(), effective_leader_version,)
|
|
{ warn!(client_id = id.0, client_version = cv.as_str(), leader_version =
|
|
effective_leader_version,
|
|
"Version mismatch: client binary differs from leader binary"); let _ = client
|
|
.tx.try_send(ClientOutbound::Acp(payload.into())); } } }
|
|
ServerEvent::Disconnected(id) => { let was_registered = clients.get(& id)
|
|
.is_some_and(| c | c.registered); clients.remove(& id); if was_registered {
|
|
client_count.fetch_sub(1, Ordering::Relaxed);
|
|
kigi_log::unified_log::info("leader.client.disconnected", None,
|
|
Some(serde_json::json!({ "client_id" : id.0 })),); } pending_load_by_req
|
|
.retain(| _, (c, _) | * c != id); load_live_buffer.retain(| (c, _), _ | * c
|
|
!= id); load_replay_max_seq.retain(| (c, _), _ | * c != id); let mut
|
|
detached_sessions : Vec < String > = Vec::new(); let viewed : Vec < String >
|
|
= session_subscribers.iter().filter(| (_, subs) | subs.contains(& id)).map(|
|
|
(sid, _) | sid.clone()).collect(); for sid in viewed { let now_empty = if let
|
|
Some(subs) = session_subscribers.get_mut(& sid) { subs.remove(& id); subs
|
|
.is_empty() } else { true }; if now_empty { session_subscribers.remove(&
|
|
sid); session_driver.remove(& sid); detached_sessions.push(sid); } else if
|
|
session_driver.get(& sid) == Some(& id) { if let Some(& next) =
|
|
session_subscribers.get(& sid).and_then(| s | s.iter().next()) {
|
|
session_driver.insert(sid.clone(), next); debug!(session_id = % sid,
|
|
old_driver = id.0, new_driver = next.0,
|
|
"Transferred session driver after disconnect"); } else { session_driver
|
|
.remove(& sid); } } } if last_active_client == Some(id) { last_active_client
|
|
= None; }
|
|
if ! detached_sessions.is_empty() { let evict_notification =
|
|
serde_json::json!({ "jsonrpc" : "2.0", "method" :
|
|
"x.ai/internal/evict_sessions", "params" : { "sessionIds" : detached_sessions
|
|
} }); let _ = acp_tx.send(evict_notification.to_string()); info!(client_id =
|
|
id.0, session_count = detached_sessions.len(),
|
|
"Sent client-disconnect detach notification for disconnected client"); }
|
|
debug!(client_id = id.0, "Client removed"); if clients.is_empty() &&
|
|
had_clients && ! no_exit_on_disconnect {
|
|
info!("Leader server shutting down (all clients disconnected)"); break; } }
|
|
ServerEvent::Message(id, ClientMessage::Control { request_id, command }) => {
|
|
if let Some(client) = clients.get(& id) { let client_tx = client.tx.clone();
|
|
let control_state = control_state.clone(); let cancel = cancel.clone(); let
|
|
shutdown_tx = shutdown_tx.clone(); let agent_busy = agent_busy.clone(); let
|
|
agent_activity = agent_activity.clone(); let relaunching = relaunching
|
|
.clone(); tokio::spawn(async move { let result = match command {
|
|
ControlCommand::StopCpuProfile => { handle_stop_cpu_profile(control_state).
|
|
await } ControlCommand::WorkspaceStart { hub_url, cwd } => {
|
|
handle_workspace_start(control_state, hub_url, cwd, cancel.clone(),). await }
|
|
ControlCommand::WorkspacePause => { handle_workspace_pause(control_state).
|
|
await } ControlCommand::WorkspaceResume => {
|
|
handle_workspace_resume(control_state). await } ControlCommand::WorkspaceStop
|
|
=> { handle_workspace_stop(control_state). await }
|
|
ControlCommand::WorkspaceStatus => { handle_workspace_status(control_state).
|
|
await } ControlCommand::RelaunchForUpdate { to_version } => {
|
|
decide_relaunch_for_update(& control_state, to_version, & relaunching,) }
|
|
other => handle_control_command(& control_state, other), }; let arm_relaunch
|
|
= matches!(result, Ok(ControlPayload::Relaunching { .. })); if let Err(e) =
|
|
client_tx.send(ServerMessage::ControlResult { request_id, result } .into()).
|
|
await { warn!(client_id = id.0, error = % e,
|
|
"Failed to send control response to client"); }
|
|
if arm_relaunch {
|
|
spawn_relaunch_drain(shutdown_tx, cancel, agent_busy, agent_activity,); } });
|
|
} } ServerEvent::Message(id, ClientMessage::Acp { payload }) => { let mut
|
|
json : Option < serde_json::Value > = serde_json::from_str(& payload).ok();
|
|
let mut payload_mutated = false; if !* ready_rx.borrow() { if let
|
|
Some(error_payload) = json.as_ref().and_then(make_leader_starting_error) { if
|
|
let Some(client) = clients.get(& id) { let _ = client.tx
|
|
.try_send(ClientOutbound::Acp(error_payload.into())); } trace!(client_id = id
|
|
.0, "Returned leader_starting error (not yet ready)"); } else {
|
|
trace!(client_id = id.0,
|
|
"Dropped pre-ready notification (leader not yet ready)"); } continue; }
|
|
if
|
|
let Some(client) = clients.get(& id) && client.mode == ClientMode::Stdio {
|
|
last_active_client = Some(id); }
|
|
if let Some(session_id) = json.as_ref()
|
|
.and_then(extract_session_id) { session_subscribers.entry(session_id.clone())
|
|
.or_default().insert(id); session_driver.entry(session_id.clone())
|
|
.or_insert(id); backfill_child_routes(& session_id, id, & child_sessions, &
|
|
mut session_subscribers, & mut session_driver,); }
|
|
if let (Some(json),
|
|
Some(client)) = (json.as_ref(), clients.get_mut(& id)) { if let
|
|
Some(yolo_mode) = extract_yolo_mode_change(json) { client.capabilities
|
|
.yolo_mode = yolo_mode; debug!(client_id = id.0, yolo_mode,
|
|
"Updated client yolo_mode from notification"); }
|
|
if let Some(auto_mode) =
|
|
extract_auto_mode_change(json) { client.capabilities.auto_mode = auto_mode;
|
|
debug!(client_id = id.0, auto_mode,
|
|
"Updated client auto_mode from notification"); }
|
|
if let Some(new_model) =
|
|
extract_model_id_from_set_model(json) { debug!(client_id = id.0, model = %
|
|
new_model, "Updated client default_model from session/setModel"); client
|
|
.capabilities.default_model = Some(new_model); } }
|
|
if let (Some(json),
|
|
Some(client)) = (json.as_mut(), clients.get_mut(& id)) { if ! client
|
|
.initialize_seen { let (injected, was_initialize) =
|
|
inject_client_identity_into_initialize(json, & client.client_type);
|
|
payload_mutated |= injected; if was_initialize { client.initialize_seen =
|
|
true; if client.capabilities.default_model.as_ref().is_some_and(| m | ! m
|
|
.is_empty()) { client.patch_initialize_model = true; } } } payload_mutated |=
|
|
inject_capabilities_into_session_new(json, & client.capabilities, & client
|
|
.client_type, id); payload_mutated |=
|
|
inject_client_identity_into_yolo_notification(json, & client.client_type); }
|
|
let rewritten = json.as_mut().and_then(| j | rewrite_request_id(j, id));
|
|
payload_mutated |= rewritten.is_some(); if let Some(json) = json.as_ref() &&
|
|
is_session_load_request(json) && let Some(load_sid) =
|
|
extract_session_id(json) && let Some((ns_id, _)) = rewritten.as_ref() {
|
|
pending_load_by_req.insert(ns_id.clone(), (id, load_sid.clone()));
|
|
load_live_buffer.entry((id, load_sid)).or_default(); }
|
|
if rewritten.is_some()
|
|
{ pending_requests += 1; agent_busy.store(true, Ordering::Relaxed); } let
|
|
outbound = select_outbound_payload(json.as_ref(), payload_mutated, payload);
|
|
let _ = acp_tx.send(outbound); } ServerEvent::Message(_, _) => {} } }
|
|
Some(payload) = response_rx.recv() => { let mut json : Option <
|
|
serde_json::Value > = serde_json::from_str(& payload).ok(); let
|
|
parsed_response = json.as_mut().and_then(parse_response_id); if
|
|
parsed_response.is_some() { pending_requests = pending_requests
|
|
.saturating_sub(1); agent_busy.store(pending_requests > 0,
|
|
Ordering::Relaxed); }
|
|
if let Some((orphan_client, ref orphan_req_id)) =
|
|
parsed_response && ! clients.contains_key(& orphan_client) { warn!(client_id
|
|
= orphan_client.0, request_id = orphan_req_id.as_str(),
|
|
"Dropping RPC response: requesting client disconnected (response orphaned)");
|
|
kigi_log::unified_log::warn("leader.response.orphaned", None,
|
|
Some(serde_json::json!({ "client_id" : orphan_client.0, "request_id" :
|
|
orphan_req_id, })),); }
|
|
if let Some((client_id, ref raw_response_id)) =
|
|
parsed_response && let Some(client) = clients.get_mut(& client_id) && let
|
|
Some(json) = json.as_mut() { if let Some(session_id) =
|
|
extract_session_id_from_result(json) { session_subscribers.entry(session_id
|
|
.clone()).or_default().insert(client_id); session_driver.entry(session_id
|
|
.clone()).or_insert(client_id); backfill_child_routes(& session_id,
|
|
client_id, & child_sessions, & mut session_subscribers, & mut
|
|
session_driver,); trace!(client_id = client_id.0, session_id,
|
|
"Subscribed client to session from response"); }
|
|
if client
|
|
.patch_initialize_model { client.patch_initialize_model = false;
|
|
patch_initialize_response_model(json, & client.capabilities.default_model); }
|
|
let restored_payload : Arc < str > = json.to_string().into(); match client.tx
|
|
.try_send(ClientOutbound::Acp(restored_payload)) { Ok(true) => {
|
|
trace!(client_id = client_id.0, "Routed response via request ID"); }
|
|
Ok(false) => { warn!(client_id = client_id.0,
|
|
"Failed to send response to client (channel full)");
|
|
kigi_log::unified_log::warn("leader.response.send_failed", None,
|
|
Some(serde_json::json!({ "client_id" : client_id.0, "reason" :
|
|
"channel_full", })),); } Err(e) => { warn!(client_id = client_id.0, error = %
|
|
e, "Failed to send response to client (channel closed)");
|
|
kigi_log::unified_log::warn("leader.response.send_failed", None,
|
|
Some(serde_json::json!({ "client_id" : client_id.0, "reason" :
|
|
"channel_closed", })),); } } if let Some((buf_client, buf_sid)) =
|
|
pending_load_by_req.remove(raw_response_id) { let replay_cutoff : Option <
|
|
u64 > = load_replay_max_seq.remove(& (buf_client, buf_sid.clone())); if let
|
|
Some(buffered) = load_live_buffer.remove(& (buf_client, buf_sid.clone())) &&
|
|
let Some(target) = clients.get(& buf_client) { let mut count = 0usize; let
|
|
mut deduped = 0usize; for (buffered_payload, buffered_seq) in buffered { if
|
|
let Some(cutoff) = replay_cutoff && buffered_seq.is_some_and(| s | s <=
|
|
cutoff) { deduped += 1; continue; }
|
|
if let Err(e) = target.tx
|
|
.try_send(ClientOutbound::Acp(buffered_payload)) { warn!(client_id =
|
|
buf_client.0, error = % e,
|
|
"Failed to flush buffered live notification after load (channel closed)");
|
|
break; } count += 1; } if count > 0 || deduped > 0 { trace!(client_id =
|
|
buf_client.0, count, deduped,
|
|
"Flushed buffered live notifications after load (replay-overlap dropped)"); }
|
|
}
|
|
if let Some(cached) = interaction_requests.get(buf_sid.as_str()) && let
|
|
Some(target) = clients.get(& buf_client) { let count = cached.len(); for req
|
|
in cached.values() { if let Err(e) = target.tx
|
|
.try_send(ClientOutbound::Acp(req.clone())) { warn!(client_id = buf_client.0,
|
|
error = % e,
|
|
"Failed to replay interaction request after load (channel closed)"); break; }
|
|
} if count > 0 { trace!(client_id = buf_client.0, count, session_id = buf_sid
|
|
.as_str(), "Replayed pending interaction modals to newly-attached client"); }
|
|
} } continue; } let payload : Arc < str > = payload.into(); let json = json;
|
|
if json.as_ref().is_some_and(is_machine_wide_broadcast_notification) { for
|
|
client in clients.values() { let _ = client.tx
|
|
.try_send(ClientOutbound::Acp(payload.clone())); }
|
|
trace!("Broadcast machine-wide notification to all clients"); continue; }
|
|
if
|
|
let Some(target) = json.as_ref().and_then(extract_target_client_id) { if let
|
|
Some(client) = clients.get(& target) { match json.as_ref()
|
|
.and_then(extract_child_session_event) {
|
|
Some(ChildSessionEvent::Spawned(child_sid)) => { if let Some(parent) = json
|
|
.as_ref().and_then(extract_session_id) { child_sessions.entry(parent)
|
|
.or_default().insert(child_sid.clone()); } debug!(client_id = target.0,
|
|
child_session_id = % child_sid,
|
|
"Registered child route from replayed SubagentSpawned"); session_subscribers
|
|
.entry(child_sid).or_default().insert(target); }
|
|
Some(ChildSessionEvent::Finished(child_sid)) => { let emptied =
|
|
session_subscribers.get_mut(& child_sid).is_some_and(| subs | { subs.remove(&
|
|
target); subs.is_empty() }); if emptied { prune_child_route(& child_sid, &
|
|
mut session_subscribers, & mut session_driver, & mut child_sessions,); } }
|
|
None => {} } let replay_seq = json.as_ref().and_then(extract_session_id)
|
|
.zip(json.as_ref().and_then(event_seq_of)); match client.tx
|
|
.try_send(ClientOutbound::Acp(payload)) { Ok(true) => { if let Some((sid,
|
|
seq)) = replay_seq { let entry = load_replay_max_seq.entry((target, sid))
|
|
.or_insert(0); * entry = (* entry).max(seq); } trace!(client_id = target.0,
|
|
"Unicast replay notification to loading client"); } Ok(false) => {
|
|
warn!(client_id = target.0,
|
|
"Replay notification dropped: loading client channel full (not counted toward flush cutoff)");
|
|
} Err(e) => { warn!(client_id = target.0, error = % e,
|
|
"Failed to unicast replay notification to loading client (channel closed)");
|
|
} } } else { if let Some(ChildSessionEvent::Finished(child_sid)) = json
|
|
.as_ref().and_then(extract_child_session_event) && session_subscribers.get(&
|
|
child_sid).is_none_or(| subs | subs.is_empty()) { prune_child_route(&
|
|
child_sid, & mut session_subscribers, & mut session_driver, & mut
|
|
child_sessions,); }
|
|
if orphan_replay_warned.insert(target) { warn!(client_id
|
|
= target.0,
|
|
"Dropping targeted replay notification: loading client disconnected mid-replay (rest of burst logged at trace)");
|
|
} else { trace!(client_id = target.0,
|
|
"Dropping targeted replay notification: loading client disconnected mid-replay");
|
|
} } continue; } let session_id = json.as_ref().and_then(extract_session_id)
|
|
.or_else(|| json.as_ref().and_then(extract_session_id_from_prompt_complete));
|
|
if let Some(ref sid) = session_id && let Some(tcid) = json.as_ref()
|
|
.and_then(extract_interaction_resolved_tool_call_id) && let Some(map) =
|
|
interaction_requests.get_mut(sid.as_str()) { map.remove(& tcid); if map
|
|
.is_empty() { interaction_requests.remove(sid.as_str()); } } let
|
|
is_reverse_request = json.as_ref().is_some_and(| j | j.get("id").is_some() &&
|
|
j.get("method").is_some()); let is_inject_prompt = json.as_ref()
|
|
.is_some_and(is_scheduled_task_inject_prompt); let is_interaction =
|
|
is_reverse_request && json.as_ref().is_some_and(is_interaction_request); if
|
|
is_interaction && let Some(ref sid) = session_id && let Some(tcid) = json
|
|
.as_ref().and_then(extract_interaction_tool_call_id) { interaction_requests
|
|
.entry(sid.clone()).or_default().insert(tcid, payload.clone()); }
|
|
if let
|
|
Some(ref sid) = session_id && session_subscribers.contains_key(sid.as_str())
|
|
{ let child_event = json.as_ref().and_then(extract_child_session_event); let
|
|
event_seq = json.as_ref().and_then(event_seq_of); if (is_reverse_request && !
|
|
is_interaction) || is_inject_prompt { if let Some(& driver_id) =
|
|
session_driver.get(sid.as_str()) { if let Some(client) = clients.get(&
|
|
driver_id) { if let Err(e) = client.tx.try_send(ClientOutbound::Acp(payload
|
|
.clone())) { warn!(client_id = driver_id.0, session_id = sid.as_str(),
|
|
is_inject = is_inject_prompt, error = % e,
|
|
"Failed to route driver-only message (channel closed)"); } else {
|
|
trace!(client_id = driver_id.0, session_id = sid.as_str(), is_inject =
|
|
is_inject_prompt, "Routed driver-only message to driver"); } } else {
|
|
trace!(session_id = sid.as_str(), is_inject = is_inject_prompt,
|
|
"Dropping driver-only message: no live driver"); } } else { trace!(session_id
|
|
= sid.as_str(), is_inject = is_inject_prompt,
|
|
"Dropping driver-only message: session has no driver"); } } else if let
|
|
Some(subs) = session_subscribers.get(sid.as_str()) { for & cid in subs.iter()
|
|
{ if let Some(buf) = load_live_buffer.get_mut(& (cid, sid.clone())) { if buf
|
|
.len() < MAX_BUFFERED_LIVE_PER_LOAD { buf.push((payload.clone(), event_seq));
|
|
trace!(client_id = cid.0, session_id = sid.as_str(),
|
|
"Buffered live notification during in-flight load"); continue; }
|
|
warn!(client_id = cid.0, session_id = sid.as_str(),
|
|
"Live buffer for in-flight load exceeded cap; forwarding live (ordering not guaranteed)");
|
|
}
|
|
if let Some(client) = clients.get(& cid) { if let Err(e) = client.tx
|
|
.try_send(ClientOutbound::Acp(payload.clone())) { warn!(client_id = cid.0,
|
|
session_id = sid.as_str(), error = % e,
|
|
"Failed to broadcast notification to subscriber (channel closed)"); } else {
|
|
trace!(client_id = cid.0, session_id = sid.as_str(),
|
|
"Broadcast notification to subscriber"); } } } } match child_event {
|
|
Some(ChildSessionEvent::Spawned(child_sid)) => { let parent_subs =
|
|
session_subscribers.get(sid.as_str()).cloned().unwrap_or_default();
|
|
info!(child_session_id = % child_sid, subscriber_count = parent_subs.len(),
|
|
"Registered child session from SubagentSpawned"); session_subscribers
|
|
.insert(child_sid.clone(), parent_subs); if let Some(& driver_id) =
|
|
session_driver.get(sid.as_str()) { session_driver.insert(child_sid.clone(),
|
|
driver_id); } child_sessions.entry(sid.clone()).or_default()
|
|
.insert(child_sid); } Some(ChildSessionEvent::Finished(child_sid)) => {
|
|
debug!(child_session_id = % child_sid,
|
|
"Deregistered child session from SubagentFinished"); prune_child_route(&
|
|
child_sid, & mut session_subscribers, & mut session_driver, & mut
|
|
child_sessions,); } None => {} } continue; } let is_notification = json
|
|
.as_ref().is_some_and(| j | j.get("id").is_none()); let
|
|
is_relay_session_notification = is_notification && session_id.as_ref()
|
|
.is_some_and(| s | ! session_subscribers.contains_key(s.as_str())); if !
|
|
is_notification {
|
|
trace!("Dropping non-routable response (likely relay-originated)"); } else if
|
|
is_relay_session_notification { if let
|
|
Some(ChildSessionEvent::Finished(child_sid)) = json.as_ref()
|
|
.and_then(extract_child_session_event) && session_subscribers.get(&
|
|
child_sid).is_none_or(| subs | subs.is_empty()) { prune_child_route(&
|
|
child_sid, & mut session_subscribers, & mut session_driver, & mut
|
|
child_sessions,); }
|
|
trace!("Dropping notification for relay-owned session (already delivered via WS)");
|
|
} else if let Some(client_id) = last_active_client && let Some(client) =
|
|
clients.get(& client_id) { debug!(client_id = client_id.0,
|
|
"Using fallback routing to last active client"); if let Err(e) = client.tx
|
|
.try_send(ClientOutbound::Acp(payload)) { warn!(client_id = client_id.0,
|
|
error = % e,
|
|
"Failed to send notification via fallback routing (channel closed)"); } }
|
|
else {
|
|
debug!("No client available for notification routing, message dropped"); } }
|
|
}
|
|
}
|
|
finalize_workspace_on_shutdown(control_state.clone()).await;
|
|
finalize_cpu_profile_on_shutdown(control_state).await;
|
|
let _ = std::fs::remove_file(&socket_path);
|
|
Ok(())
|
|
}
|
|
fn spawn_client_handler(
|
|
client_id: ClientId,
|
|
stream: LeaderStream,
|
|
server_rx: AsyncReceiver<ClientOutbound>,
|
|
event_tx: AsyncSender<ServerEvent>,
|
|
cancel: CancellationToken,
|
|
ready_rx: watch::Receiver<bool>,
|
|
control_state: LeaderServerControlState,
|
|
) {
|
|
tokio::spawn(async move {
|
|
let result = run_client_session(
|
|
client_id,
|
|
stream,
|
|
server_rx,
|
|
event_tx.clone(),
|
|
cancel,
|
|
ready_rx,
|
|
control_state,
|
|
)
|
|
.await;
|
|
if let Err(e) = &result {
|
|
debug!(client_id = client_id.0, error = % e, "Client session ended");
|
|
}
|
|
let _ = event_tx.send(ServerEvent::Disconnected(client_id)).await;
|
|
});
|
|
}
|
|
async fn run_client_session(
|
|
client_id: ClientId,
|
|
stream: LeaderStream,
|
|
server_rx: AsyncReceiver<ClientOutbound>,
|
|
event_tx: AsyncSender<ServerEvent>,
|
|
cancel: CancellationToken,
|
|
mut ready_rx: watch::Receiver<bool>,
|
|
control_state: LeaderServerControlState,
|
|
) -> Result<(), ProtocolError> {
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
let msg: ClientMessage =
|
|
match tokio::time::timeout(REGISTRATION_TIMEOUT, read_message(&mut reader)).await {
|
|
Ok(Ok(msg)) => msg,
|
|
Ok(Err(e)) => {
|
|
warn!(client_id = client_id.0, error = % e, "Registration failed");
|
|
return Err(e);
|
|
}
|
|
Err(_) => {
|
|
warn!(
|
|
client_id = client_id.0,
|
|
"Registration timeout - client did not register within {:?}",
|
|
REGISTRATION_TIMEOUT
|
|
);
|
|
let _ = write_message(
|
|
&mut writer,
|
|
&ServerMessage::Error {
|
|
code: 3,
|
|
message: "Registration timeout".into(),
|
|
},
|
|
)
|
|
.await;
|
|
return Ok(());
|
|
}
|
|
};
|
|
let (client_type, mode, capabilities, was_ready_at_registration) = match msg {
|
|
ClientMessage::Register {
|
|
client_type,
|
|
mode,
|
|
capabilities,
|
|
} => {
|
|
let ready = *ready_rx.borrow();
|
|
write_message(
|
|
&mut writer,
|
|
&ServerMessage::Registered {
|
|
client_id: client_id.0,
|
|
ready,
|
|
leader_protocol_version: Some(LEADER_PROTOCOL_VERSION),
|
|
leader_binary_version: Some(
|
|
control_state.metadata.leader_binary_version.clone(),
|
|
),
|
|
leader_capabilities: Some(control_state.leader_capabilities()),
|
|
},
|
|
)
|
|
.await?;
|
|
(client_type, mode, capabilities, ready)
|
|
}
|
|
_ => {
|
|
write_message(
|
|
&mut writer,
|
|
&ServerMessage::Error {
|
|
code: 1,
|
|
message: "Expected Register message".into(),
|
|
},
|
|
)
|
|
.await?;
|
|
return Ok(());
|
|
}
|
|
};
|
|
if !was_ready_at_registration {
|
|
debug!(
|
|
client_id = client_id.0,
|
|
"Client registered before leader ready; waiting for readiness"
|
|
);
|
|
while !*ready_rx.borrow() {
|
|
tokio::select! {
|
|
biased; _ = cancel.cancelled() => { for _ in 0..10 { if ! server_rx
|
|
.is_empty() { break; } tokio::task::yield_now(). await; } while let
|
|
Ok(Some(msg)) = server_rx.try_recv() { if write_outbound(& mut writer, &
|
|
msg). await .is_err() { break; } } return Ok(()); } result = ready_rx
|
|
.changed() => { if result.is_err() { return Ok(()); } }
|
|
}
|
|
}
|
|
write_message(&mut writer, &ServerMessage::LeaderReady).await?;
|
|
debug!(
|
|
client_id = client_id.0,
|
|
"Leader ready; sent LeaderReady to client"
|
|
);
|
|
}
|
|
let _ = event_tx
|
|
.send(ServerEvent::Registered(
|
|
client_id,
|
|
mode,
|
|
capabilities.clone(),
|
|
client_type.clone(),
|
|
))
|
|
.await;
|
|
info!(
|
|
client_id = client_id.0, client_type = % client_type, ? mode, yolo_mode =
|
|
capabilities.yolo_mode, client_version = ? capabilities.client_version,
|
|
"Client registered"
|
|
);
|
|
loop {
|
|
tokio::select! {
|
|
biased; _ = cancel.cancelled() => { for _ in 0..10 { if ! server_rx
|
|
.is_empty() { break; } tokio::task::yield_now(). await; } while let
|
|
Ok(Some(msg)) = server_rx.try_recv() { if write_outbound(& mut writer, & msg)
|
|
. await .is_err() { break; } } break; } Ok(msg) = server_rx.recv() => { if
|
|
write_outbound(& mut writer, & msg). await .is_err() { break; } } msg_result
|
|
= read_message::< _, ClientMessage > (& mut reader) => { match msg_result {
|
|
Ok(msg @ (ClientMessage::Acp { .. } | ClientMessage::Control { .. })) => {
|
|
let _ = event_tx.send(ServerEvent::Message(client_id, msg)). await; }
|
|
Ok(ClientMessage::Ping) => { write_message(& mut writer, &
|
|
ServerMessage::Pong). await ?; } Ok(ClientMessage::Disconnect) |
|
|
Err(ProtocolError::ConnectionClosed) => { info!(client_id = client_id.0,
|
|
"Client disconnected"); break; } Ok(ClientMessage::Register { .. }) => {
|
|
write_message(& mut writer, & ServerMessage::Error { code : 2, message :
|
|
"Already registered".into(), }). await ?; } Err(e) => { warn!(client_id =
|
|
client_id.0, error = % e, "Protocol error"); break; } } }
|
|
}
|
|
}
|
|
Ok(())
|
|
}
|
|
/// Broadcast a planned shutdown to all connected clients.
|
|
///
|
|
/// Sends `ShuttingDown` (advance notice with reason and `delay_ms: 0`)
|
|
/// followed immediately by `Shutdown`. Both messages are sent before the
|
|
/// server exits, so clients that process the channel quickly will see both.
|
|
///
|
|
/// `delay_ms` is set to 0 because the server sends `Shutdown` immediately
|
|
/// after `ShuttingDown` — there is no actual grace period. The cancel token
|
|
/// propagates to client session handlers simultaneously, so a sleep between
|
|
/// the two messages would allow session writers to exit before `Shutdown`
|
|
/// is delivered. Clients should treat `ShuttingDown` as a signal that
|
|
/// `Shutdown` is imminent and pre-arm their reconnection handlers.
|
|
async fn broadcast_shutdown(
|
|
clients: &HashMap<ClientId, ClientState>,
|
|
reason: super::protocol::ShutdownReason,
|
|
) {
|
|
for client in clients.values() {
|
|
let _ = client
|
|
.tx
|
|
.send(
|
|
ServerMessage::ShuttingDown {
|
|
reason: reason.clone(),
|
|
delay_ms: 0,
|
|
}
|
|
.into(),
|
|
)
|
|
.await;
|
|
let _ = client.tx.send(ServerMessage::Shutdown.into()).await;
|
|
}
|
|
}
|
|
pub struct ServerHandle {
|
|
pub cancel: CancellationToken,
|
|
/// Receive ACP messages from clients (server routes them here)
|
|
pub acp_rx: mpsc::UnboundedReceiver<String>,
|
|
/// Send ACP responses back (server routes to correct client based on request ID)
|
|
pub response_tx: mpsc::UnboundedSender<String>,
|
|
/// Atomic counter tracking the number of connected clients
|
|
pub client_count: Arc<AtomicUsize>,
|
|
/// Atomic flag: `true` while the agent has pending (in-flight) requests
|
|
pub agent_busy: Arc<AtomicBool>,
|
|
/// Signal the IPC server that the leader is fully ready (auth + prefetch complete).
|
|
///
|
|
/// Send `true` once the leader has finished initializing. Until then, ACP requests
|
|
/// receive a `leader_starting` error and ACP notifications are dropped.
|
|
///
|
|
/// `spawn_leader_server` sends `true` immediately so that callers that do not need
|
|
/// staged startup (e.g. tests, in-process use) get a fully-ready server out of the box.
|
|
/// Production leader startup (`run_leader`) holds this back until auth + prefetch succeed.
|
|
pub ready_tx: watch::Sender<bool>,
|
|
/// Set the shutdown reason before cancelling so clients receive the correct `ShuttingDown`
|
|
/// reason. The default value is [`ShutdownReason::Manual`]; send
|
|
/// [`ShutdownReason::AutoUpdate`] before cancelling for auto-update shutdowns.
|
|
pub shutdown_tx: watch::Sender<super::protocol::ShutdownReason>,
|
|
/// Leader-local control metadata and CPU profiling state, exposed for tests.
|
|
pub control_state: LeaderServerControlState,
|
|
}
|
|
fn default_test_control_state(socket_path: &Path) -> LeaderServerControlState {
|
|
LeaderServerControlState::new(LeaderServerMetadata {
|
|
pid: std::process::id(),
|
|
socket_path: socket_path.to_path_buf(),
|
|
lock_path: socket_path.with_extension("lock"),
|
|
socket_suffix: String::new(),
|
|
leader_binary_version: env!("CARGO_PKG_VERSION").to_string(),
|
|
})
|
|
}
|
|
pub async fn spawn_leader_server(socket_path: PathBuf) -> Result<ServerHandle, ServerError> {
|
|
let (acp_tx, acp_rx) = mpsc::unbounded_channel();
|
|
let (response_tx, response_rx) = mpsc::unbounded_channel();
|
|
let cancel = CancellationToken::new();
|
|
let client_count = Arc::new(AtomicUsize::new(0));
|
|
let agent_busy = Arc::new(AtomicBool::new(false));
|
|
let (ready_tx, ready_rx) = watch::channel(true);
|
|
let (shutdown_tx, _shutdown_reason_rx) =
|
|
watch::channel(super::protocol::ShutdownReason::Manual);
|
|
let control_state = default_test_control_state(&socket_path);
|
|
let cancel_clone = cancel.clone();
|
|
let socket_path_clone = socket_path.clone();
|
|
let client_count_clone = client_count.clone();
|
|
let agent_busy_clone = agent_busy.clone();
|
|
let control_state_for_server = control_state.clone();
|
|
let shutdown_tx_for_server = shutdown_tx.clone();
|
|
tokio::spawn(async move {
|
|
if let Err(e) = run_leader_server(
|
|
socket_path_clone,
|
|
acp_tx,
|
|
response_rx,
|
|
cancel_clone,
|
|
false,
|
|
client_count_clone,
|
|
agent_busy_clone,
|
|
AgentActivity::default(),
|
|
ready_rx,
|
|
shutdown_tx_for_server,
|
|
None,
|
|
control_state_for_server,
|
|
)
|
|
.await
|
|
{
|
|
error!(error = % e, "Leader server error");
|
|
}
|
|
});
|
|
Ok(ServerHandle {
|
|
cancel,
|
|
acp_rx,
|
|
response_tx,
|
|
client_count,
|
|
agent_busy,
|
|
ready_tx,
|
|
shutdown_tx,
|
|
control_state,
|
|
})
|
|
}
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
use std::time::Duration;
|
|
use tempfile::TempDir;
|
|
/// Parse a raw payload for the parse-once helper APIs. Panics on invalid
|
|
/// JSON — the routing loop parses once up front, and non-JSON payloads
|
|
/// never reach the helpers (they forward/drop verbatim).
|
|
fn pv(payload: &str) -> serde_json::Value {
|
|
serde_json::from_str(payload).expect("test payload must be valid JSON")
|
|
}
|
|
/// The relaunch drain must wait on the agent-derived activity signal —
|
|
/// not just the IPC `agent_busy` flag, which relay-driven turns never set
|
|
/// — and must flush registered session actors before cancelling.
|
|
#[tokio::test]
|
|
async fn relaunch_drain_waits_for_agent_activity_and_flushes_sessions() {
|
|
let (shutdown_tx, _shutdown_rx) =
|
|
watch::channel(super::super::protocol::ShutdownReason::Manual);
|
|
let cancel = CancellationToken::new();
|
|
let agent_busy = Arc::new(AtomicBool::new(false));
|
|
let activity = AgentActivity::default();
|
|
let (mut cmd_rx, prompt_id, _pending) = activity.register_for_test("s1");
|
|
*prompt_id.lock().unwrap() = Some("prompt-1".to_string());
|
|
let cancel_for_actor = cancel.clone();
|
|
let actor = tokio::spawn(async move {
|
|
while let Some(cmd) = cmd_rx.recv().await {
|
|
if matches!(cmd, crate::session::SessionCommand::Shutdown) {
|
|
assert!(
|
|
!cancel_for_actor.is_cancelled(),
|
|
"flush must run before the leader cancels"
|
|
);
|
|
return;
|
|
}
|
|
}
|
|
});
|
|
spawn_relaunch_drain(shutdown_tx, cancel.clone(), agent_busy, activity);
|
|
tokio::time::sleep(Duration::from_millis(300)).await;
|
|
assert!(
|
|
!cancel.is_cancelled(),
|
|
"drain must not cancel while a relay-driven turn is running"
|
|
);
|
|
*prompt_id.lock().unwrap() = None;
|
|
tokio::time::timeout(Duration::from_secs(5), cancel.cancelled())
|
|
.await
|
|
.expect("drain should cancel once the agent goes idle");
|
|
actor.await.expect("session actor should get Shutdown");
|
|
}
|
|
/// `ServerMessageRef::Acp` (the borrowed serialize-only mirror the client
|
|
/// writer uses for shared payloads) must stay byte-identical on the wire
|
|
/// to `ServerMessage::Acp`, or clients would fail to decode ACP frames.
|
|
#[test]
|
|
fn server_message_ref_is_wire_identical() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"session/update","params":{"x":1}}"#;
|
|
let owned = serde_json::to_vec(&ServerMessage::Acp {
|
|
payload: payload.to_string(),
|
|
})
|
|
.unwrap();
|
|
let borrowed = serde_json::to_vec(&ServerMessageRef::Acp { payload }).unwrap();
|
|
assert_eq!(owned, borrowed);
|
|
let decoded: ServerMessage = serde_json::from_slice(&borrowed).unwrap();
|
|
match decoded {
|
|
ServerMessage::Acp { payload: p } => assert_eq!(p, payload),
|
|
other => panic!("expected Acp, got {other:?}"),
|
|
}
|
|
}
|
|
/// An UNMUTATED payload forwards to the agent byte-for-byte: parsing for
|
|
/// classification must never normalize key order or whitespace of
|
|
/// pass-through traffic.
|
|
#[test]
|
|
fn outbound_payload_verbatim_when_unmutated() {
|
|
let original = r#"{ "b" : 1, "a": 2 }"#.to_string();
|
|
let json = pv(&original);
|
|
let out = select_outbound_payload(Some(&json), false, original.clone());
|
|
assert_eq!(
|
|
out, original,
|
|
"unmutated payloads must forward verbatim (exact bytes, not re-serialized)"
|
|
);
|
|
}
|
|
/// A MUTATED payload is re-serialized from the injected/rewritten `Value`
|
|
/// (semantically equal, but no longer the original odd formatting).
|
|
#[test]
|
|
fn outbound_payload_reserialized_when_mutated() {
|
|
let original = r#"{ "b" : 1, "a": 2 }"#.to_string();
|
|
let json = pv(&original);
|
|
let out = select_outbound_payload(Some(&json), true, original.clone());
|
|
assert_ne!(
|
|
out, original,
|
|
"mutated payloads must be re-serialized from the Value, not the stale original"
|
|
);
|
|
assert_eq!(
|
|
pv(&out),
|
|
json,
|
|
"the re-serialized payload must be semantically identical to the mutated Value"
|
|
);
|
|
}
|
|
/// A non-JSON payload (`json = None`) is never parsed or re-serialized —
|
|
/// it passes through untouched, matching the old per-helper parse-failure
|
|
/// behavior.
|
|
#[test]
|
|
fn outbound_payload_non_json_passthrough() {
|
|
let original = "not json".to_string();
|
|
let out = select_outbound_payload(None, false, original.clone());
|
|
assert_eq!(
|
|
out, original,
|
|
"non-JSON payloads must pass through verbatim"
|
|
);
|
|
}
|
|
#[test]
|
|
fn decide_relaunch_is_idempotent_and_directional() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock = temp.path().join("leader.sock");
|
|
let control_state = LeaderServerControlState::new(LeaderServerMetadata {
|
|
pid: std::process::id(),
|
|
socket_path: sock.clone(),
|
|
lock_path: sock.with_extension("lock"),
|
|
socket_suffix: String::new(),
|
|
leader_binary_version: "0.1.100".to_string(),
|
|
});
|
|
let relaunching = AtomicBool::new(false);
|
|
assert!(matches!(
|
|
decide_relaunch_for_update(&control_state, "0.1.100".to_string(), &relaunching),
|
|
Ok(ControlPayload::RelaunchDeclined { .. })
|
|
));
|
|
assert!(!relaunching.load(Ordering::SeqCst));
|
|
assert!(matches!(
|
|
decide_relaunch_for_update(&control_state, "0.1.0".to_string(), &relaunching),
|
|
Ok(ControlPayload::RelaunchDeclined { .. })
|
|
));
|
|
assert!(matches!(
|
|
decide_relaunch_for_update(&control_state, "unknown".to_string(), &relaunching),
|
|
Ok(ControlPayload::RelaunchDeclined { .. })
|
|
));
|
|
assert!(!relaunching.load(Ordering::SeqCst));
|
|
assert!(matches!(
|
|
decide_relaunch_for_update(&control_state, "0.2.0".to_string(), &relaunching),
|
|
Ok(ControlPayload::Relaunching { .. })
|
|
));
|
|
assert!(relaunching.load(Ordering::SeqCst));
|
|
assert!(matches!(
|
|
decide_relaunch_for_update(&control_state, "0.3.0".to_string(), &relaunching),
|
|
Ok(ControlPayload::RelaunchDeclined { .. })
|
|
));
|
|
}
|
|
#[derive(Debug)]
|
|
struct TestAuth;
|
|
impl AuthProvider for TestAuth {
|
|
fn current(&self) -> AuthCredential {
|
|
AuthCredential::bearer("test-token")
|
|
}
|
|
}
|
|
#[tokio::test]
|
|
async fn wait_for_leader_auth_returns_when_already_wired() {
|
|
let ws = WorkspaceControl::new(None);
|
|
ws.auth.send_replace(Some(Arc::new(TestAuth)));
|
|
let cancel = CancellationToken::new();
|
|
let auth = wait_for_leader_auth(&ws, &cancel).await.expect("wired");
|
|
assert!(matches!(auth.current(), AuthCredential::Bearer { .. }));
|
|
}
|
|
#[tokio::test]
|
|
async fn wait_for_leader_auth_resolves_when_wired_late() {
|
|
let ws = Arc::new(WorkspaceControl::new(None));
|
|
let cancel = CancellationToken::new();
|
|
let waiter = {
|
|
let ws = ws.clone();
|
|
let cancel = cancel.clone();
|
|
tokio::spawn(async move { wait_for_leader_auth(&ws, &cancel).await.is_ok() })
|
|
};
|
|
tokio::task::yield_now().await;
|
|
ws.auth.send_replace(Some(Arc::new(TestAuth)));
|
|
assert!(waiter.await.unwrap(), "auth wired late should resolve Ok");
|
|
}
|
|
#[tokio::test]
|
|
async fn workspace_start_errors_when_cancelled_before_auth() {
|
|
let state = default_test_control_state(Path::new("/tmp/grok-ws-auth-test.sock"));
|
|
let cancel = CancellationToken::new();
|
|
cancel.cancel();
|
|
let err = handle_workspace_start(state, None, "/tmp".to_string(), cancel)
|
|
.await
|
|
.unwrap_err();
|
|
assert!(
|
|
err.message.contains("shutting down"),
|
|
"unexpected error: {}",
|
|
err.message
|
|
);
|
|
}
|
|
async fn setup_test_server(
|
|
temp: &TempDir,
|
|
) -> (PathBuf, CancellationToken, mpsc::UnboundedReceiver<String>) {
|
|
let sock_path = temp.path().join("test.sock");
|
|
let handle = spawn_leader_server(sock_path.clone()).await.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
(sock_path, handle.cancel, handle.acp_rx)
|
|
}
|
|
async fn setup_test_server_with_client_count(
|
|
temp: &TempDir,
|
|
) -> (
|
|
PathBuf,
|
|
CancellationToken,
|
|
mpsc::UnboundedReceiver<String>,
|
|
Arc<AtomicUsize>,
|
|
) {
|
|
let sock_path = temp.path().join("test.sock");
|
|
let handle = spawn_leader_server(sock_path.clone()).await.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
(sock_path, handle.cancel, handle.acp_rx, handle.client_count)
|
|
}
|
|
/// Like `setup_test_server` but uses `no_exit_on_disconnect=true` and
|
|
/// exposes `response_tx` for injecting agent responses.
|
|
async fn setup_persistent_server(
|
|
temp: &TempDir,
|
|
) -> (PathBuf, CancellationToken, mpsc::UnboundedSender<String>) {
|
|
let (sock_path, cancel, response_tx, _acp_rx) =
|
|
setup_persistent_server_with_agent(temp).await;
|
|
(sock_path, cancel, response_tx)
|
|
}
|
|
/// Like `setup_persistent_server` but also returns the agent-side receiver
|
|
/// (`acp_rx`) so a test can observe forwarded requests — e.g. to read a
|
|
/// `session/load`'s namespaced id and echo a matching load response, which
|
|
/// is required to complete a load now that live broadcasts to a loading
|
|
/// client are buffered until its load response (see `complete_load`).
|
|
async fn setup_persistent_server_with_agent(
|
|
temp: &TempDir,
|
|
) -> (
|
|
PathBuf,
|
|
CancellationToken,
|
|
mpsc::UnboundedSender<String>,
|
|
mpsc::UnboundedReceiver<String>,
|
|
) {
|
|
let sock_path = temp.path().join("test.sock");
|
|
let (acp_tx, acp_rx) = mpsc::unbounded_channel();
|
|
let (response_tx, response_rx) = mpsc::unbounded_channel();
|
|
let cancel = CancellationToken::new();
|
|
let control_state = default_test_control_state(&sock_path);
|
|
let sock_clone = sock_path.clone();
|
|
let cancel_clone = cancel.clone();
|
|
let (_ready_tx, ready_rx) = watch::channel(true);
|
|
let (shutdown_tx, _shutdown_rx) =
|
|
watch::channel(super::super::protocol::ShutdownReason::Manual);
|
|
tokio::spawn(async move {
|
|
let _ = run_leader_server(
|
|
sock_clone,
|
|
acp_tx,
|
|
response_rx,
|
|
cancel_clone,
|
|
true,
|
|
Arc::new(AtomicUsize::new(0)),
|
|
Arc::new(AtomicBool::new(false)),
|
|
AgentActivity::default(),
|
|
ready_rx,
|
|
shutdown_tx,
|
|
None,
|
|
control_state,
|
|
)
|
|
.await;
|
|
});
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
(sock_path, cancel, response_tx, acp_rx)
|
|
}
|
|
/// Complete an in-flight `session/load` in a test: read the forwarded load
|
|
/// request from the agent channel to learn its leader-assigned namespaced
|
|
/// id, then echo a `LoadSessionResponse` with that id. This routes the
|
|
/// response back to the loading client AND flushes any live notifications
|
|
/// the leader buffered during the load window (live-before-replay guard).
|
|
async fn complete_load(
|
|
acp_rx: &mut mpsc::UnboundedReceiver<String>,
|
|
response_tx: &mpsc::UnboundedSender<String>,
|
|
) {
|
|
loop {
|
|
let forwarded = tokio::time::timeout(Duration::from_secs(1), acp_rx.recv())
|
|
.await
|
|
.expect("timed out waiting for forwarded session/load")
|
|
.expect("agent channel closed");
|
|
let json: serde_json::Value = serde_json::from_str(&forwarded).unwrap();
|
|
if json.get("method").and_then(|m| m.as_str()) == Some("session/load") {
|
|
let id = json.get("id").cloned().unwrap();
|
|
let response = serde_json::json!(
|
|
{ "jsonrpc" : "2.0", "id" : id, "result" : { "models" : [] }, }
|
|
);
|
|
response_tx.send(response.to_string()).unwrap();
|
|
return;
|
|
}
|
|
}
|
|
}
|
|
/// Helper to connect and register a client, returning the split stream.
|
|
async fn connect_and_register(
|
|
sock_path: &std::path::Path,
|
|
client_type: &str,
|
|
) -> (
|
|
tokio::io::ReadHalf<LeaderStream>,
|
|
tokio::io::WriteHalf<LeaderStream>,
|
|
) {
|
|
connect_and_register_with_mode(sock_path, client_type, ClientMode::Stdio).await
|
|
}
|
|
/// Like [`connect_and_register`] but with an explicit [`ClientMode`], for
|
|
/// tests that exercise mode-dependent server behavior (relay demand).
|
|
async fn connect_and_register_with_mode(
|
|
sock_path: &std::path::Path,
|
|
client_type: &str,
|
|
mode: ClientMode,
|
|
) -> (
|
|
tokio::io::ReadHalf<LeaderStream>,
|
|
tokio::io::WriteHalf<LeaderStream>,
|
|
) {
|
|
let stream = LeaderStream::connect(sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: client_type.into(),
|
|
mode,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
(reader, writer)
|
|
}
|
|
#[tokio::test]
|
|
async fn client_registration_flow() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, _acp_rx) = setup_test_server(&temp).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let response: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
match response {
|
|
ServerMessage::Registered {
|
|
client_id,
|
|
ready,
|
|
leader_protocol_version,
|
|
leader_binary_version,
|
|
leader_capabilities,
|
|
} => {
|
|
assert!(ready);
|
|
assert!(client_id > 0);
|
|
assert_eq!(leader_protocol_version, Some(LEADER_PROTOCOL_VERSION));
|
|
assert_eq!(
|
|
leader_binary_version.as_deref(),
|
|
Some(env!("CARGO_PKG_VERSION"))
|
|
);
|
|
let capabilities = leader_capabilities.expect("leader capabilities metadata");
|
|
assert!(capabilities.control_v1);
|
|
assert_eq!(
|
|
capabilities.runtime_cpu_profile,
|
|
CpuProfileManager::new().runtime_cpu_profile()
|
|
);
|
|
}
|
|
_ => panic!("Expected Registered response"),
|
|
}
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn control_requests_bypass_acp_routing() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("test.sock");
|
|
let mut handle = spawn_leader_server(sock_path.clone()).await.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Control {
|
|
request_id: "status-1".into(),
|
|
command: ControlCommand::CpuProfileStatus,
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let response: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
assert!(
|
|
matches!(response, ServerMessage::ControlResult { request_id, result :
|
|
Ok(ControlPayload::CpuProfileStatus { active : false, stopping : false,
|
|
started_at : None, svg_path : None, frequency_hz : None, }), } if request_id
|
|
== "status-1")
|
|
);
|
|
assert!(
|
|
tokio::time::timeout(Duration::from_millis(100), handle.acp_rx.recv())
|
|
.await
|
|
.is_err()
|
|
);
|
|
handle.cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn shutdown_waits_for_in_flight_cpu_profile_stop() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("test.sock");
|
|
let output_path = temp.path().join("shutdown-runtime-profile.folded");
|
|
let control_state = default_test_control_state(&sock_path);
|
|
let stop_handle = {
|
|
let mut manager = control_state.cpu_profile.lock();
|
|
if !manager.runtime_cpu_profile() {
|
|
return;
|
|
}
|
|
let Ok(_) = manager.start(CpuProfileStartOptions {
|
|
output: Some(output_path.clone()),
|
|
frequency_hz: Some(200),
|
|
}) else {
|
|
return;
|
|
};
|
|
manager.take_stop_handle().unwrap()
|
|
};
|
|
let control_state_for_shutdown = control_state.clone();
|
|
let shutdown_wait = tokio::spawn(async move {
|
|
finalize_cpu_profile_on_shutdown(control_state_for_shutdown).await;
|
|
});
|
|
let control_state_for_stop = control_state.clone();
|
|
let in_flight_stop = tokio::spawn(async move {
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let result = tokio::task::spawn_blocking(move || stop_handle.finish())
|
|
.await
|
|
.unwrap()
|
|
.unwrap();
|
|
control_state_for_stop.cpu_profile.lock().complete_stop();
|
|
result
|
|
});
|
|
tokio::time::timeout(Duration::from_secs(5), shutdown_wait)
|
|
.await
|
|
.expect("shutdown wait should complete")
|
|
.unwrap();
|
|
let stop_result = tokio::time::timeout(Duration::from_secs(5), in_flight_stop)
|
|
.await
|
|
.expect("in-flight stop should complete")
|
|
.unwrap();
|
|
assert_eq!(stop_result.svg_path, output_path);
|
|
assert!(output_path.exists());
|
|
assert!(matches!(
|
|
control_state.cpu_profile.lock().status(),
|
|
CpuProfileStatus::Inactive
|
|
));
|
|
}
|
|
#[tokio::test]
|
|
async fn runtime_profile_reports_unsupported_build_end_to_end() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("leader-unsupported.sock");
|
|
let handle = spawn_leader_server(sock_path.clone()).await.unwrap();
|
|
{
|
|
let mut manager = handle.control_state.cpu_profile.lock();
|
|
manager.force_unsupported_for_test();
|
|
}
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let client = super::super::client::LeaderClient::connect(
|
|
sock_path,
|
|
"client",
|
|
ClientMode::Stdio,
|
|
ClientCapabilities::default(),
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let runtime_cpu_profile = client
|
|
.registration()
|
|
.leader_capabilities
|
|
.as_ref()
|
|
.is_some_and(|capabilities| capabilities.runtime_cpu_profile);
|
|
assert!(
|
|
!runtime_cpu_profile,
|
|
"unsupported stub server must report runtime_cpu_profile=false"
|
|
);
|
|
let status = client
|
|
.send_control(ControlCommand::CpuProfileStatus)
|
|
.await
|
|
.unwrap()
|
|
.unwrap();
|
|
assert!(matches!(
|
|
status,
|
|
ControlPayload::CpuProfileStatus {
|
|
active: false,
|
|
stopping: false,
|
|
started_at: None,
|
|
svg_path: None,
|
|
frequency_hz: None,
|
|
}
|
|
));
|
|
let start_err = client
|
|
.send_control(ControlCommand::StartCpuProfile {
|
|
output: None,
|
|
frequency_hz: None,
|
|
})
|
|
.await
|
|
.unwrap()
|
|
.unwrap_err();
|
|
assert_eq!(
|
|
start_err.code,
|
|
crate::cpu_profile::ControlErrorCode::RuntimeProfilingUnsupported
|
|
);
|
|
let stop_err = client
|
|
.send_control(ControlCommand::StopCpuProfile)
|
|
.await
|
|
.unwrap()
|
|
.unwrap_err();
|
|
assert_eq!(
|
|
stop_err.code,
|
|
crate::cpu_profile::ControlErrorCode::ProfileNotActive
|
|
);
|
|
client.cancel();
|
|
handle.cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn ping_pong() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, _acp_rx) = setup_test_server(&temp).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
write_message(&mut writer, &ClientMessage::Ping)
|
|
.await
|
|
.unwrap();
|
|
let response: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
assert!(matches!(response, ServerMessage::Pong));
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn acp_message_forwarding() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, mut acp_rx) = setup_test_server(&temp).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
let payload = r#"{"jsonrpc":"2.0","method":"test"}"#;
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: payload.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let received = acp_rx.recv().await.unwrap();
|
|
assert_eq!(received, payload);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn initialize_gets_client_identifier_injected() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, mut acp_rx) = setup_test_server(&temp).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "grok-tui".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
let payload =
|
|
r#"{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"0.1"}}"#;
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: payload.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let received = acp_rx.recv().await.unwrap();
|
|
let json: serde_json::Value = serde_json::from_str(&received).unwrap();
|
|
assert_eq!(
|
|
json["params"]["_meta"]["clientIdentifier"], "grok-tui",
|
|
"Leader should inject clientIdentifier from IPC registration"
|
|
);
|
|
assert_eq!(json["method"], "initialize");
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn initialize_preserves_existing_client_identifier() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, mut acp_rx) = setup_test_server(&temp).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "grok-tui".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
let payload = r#"{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"0.1","_meta":{"clientIdentifier":"grok-web"}}}"#;
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: payload.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let received = acp_rx.recv().await.unwrap();
|
|
let json: serde_json::Value = serde_json::from_str(&received).unwrap();
|
|
assert_eq!(
|
|
json["params"]["_meta"]["clientIdentifier"], "grok-web",
|
|
"Leader should not override existing clientIdentifier"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[test]
|
|
fn rewrite_request_id_rewrites_requests() {
|
|
let mut json = pv(r#"{"jsonrpc":"2.0","method":"test","id":42,"params":{}}"#);
|
|
let client_id = ClientId(123);
|
|
let (namespaced_id, original_id) = rewrite_request_id(&mut json, client_id).unwrap();
|
|
assert_eq!(original_id, serde_json::json!(42));
|
|
assert_eq!(namespaced_id, "123|42");
|
|
assert_eq!(json["id"], "123|42");
|
|
assert_eq!(json["method"], "test");
|
|
}
|
|
#[test]
|
|
fn is_session_load_request_detects_only_load() {
|
|
assert!(is_session_load_request(&pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"method":"session/load","params":{"sessionId":"s1","cwd":"/tmp"}}"#
|
|
)));
|
|
assert!(!is_session_load_request(&pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"method":"session/new","params":{}}"#
|
|
)));
|
|
assert!(!is_session_load_request(&pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"method":"session/prompt","params":{"sessionId":"s1"}}"#
|
|
)));
|
|
assert!(!is_session_load_request(&pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"result":{}}"#
|
|
)));
|
|
}
|
|
#[test]
|
|
fn is_scheduled_task_inject_prompt_detects_only_inject() {
|
|
assert!(is_scheduled_task_inject_prompt(&pv(
|
|
r#"{"method":"x.ai/scheduled_task_inject_prompt","params":{"sessionId":"s1","taskId":"t1","prompt":"echo hi"}}"#
|
|
)));
|
|
assert!(is_scheduled_task_inject_prompt(&pv(
|
|
r#"{"method":"_x.ai/scheduled_task_inject_prompt","params":{"method":"x.ai/scheduled_task_inject_prompt","params":{"sessionId":"s1","taskId":"t1","prompt":"echo hi"}}}"#
|
|
)));
|
|
assert!(!is_scheduled_task_inject_prompt(&pv(
|
|
r#"{"method":"x.ai/scheduled_task_fired","params":{"sessionId":"s1"}}"#
|
|
)));
|
|
assert!(!is_scheduled_task_inject_prompt(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"s1"}}"#
|
|
)));
|
|
}
|
|
#[test]
|
|
fn is_interaction_request_detects_only_interaction_methods() {
|
|
for m in [
|
|
"session/request_permission",
|
|
"x.ai/ask_user_question",
|
|
"x.ai/exit_plan_mode",
|
|
] {
|
|
let payload = format!(r#"{{"jsonrpc":"2.0","id":1,"method":"{m}","params":{{}}}}"#);
|
|
assert!(
|
|
is_interaction_request(&pv(&payload)),
|
|
"{m} (direct) must be an interaction"
|
|
);
|
|
}
|
|
for m in ["x.ai/ask_user_question", "x.ai/exit_plan_mode"] {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","id":1,"method":"_{m}","params":{{"method":"{m}","params":{{}}}}}}"#
|
|
);
|
|
assert!(
|
|
is_interaction_request(&pv(&payload)),
|
|
"wrapped {m} must be an interaction"
|
|
);
|
|
}
|
|
assert!(!is_interaction_request(&pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"method":"fs/read_text_file","params":{}}"#
|
|
)));
|
|
assert!(!is_interaction_request(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/sessions/changed","params":{}}"#
|
|
)));
|
|
}
|
|
#[test]
|
|
fn extract_interaction_tool_call_id_handles_direct_and_nested() {
|
|
assert_eq!(
|
|
extract_interaction_tool_call_id(&
|
|
pv(r#"{"id":1,"method":"x.ai/ask_user_question","params":{"sessionId":"s","toolCallId":"tc-q"}}"#))
|
|
.as_deref(), Some("tc-q")
|
|
);
|
|
assert_eq!(
|
|
extract_interaction_tool_call_id(&
|
|
pv(r#"{"id":1,"method":"session/request_permission","params":{"sessionId":"s","toolCall":{"toolCallId":"tc-p"}}}"#))
|
|
.as_deref(), Some("tc-p")
|
|
);
|
|
assert_eq!(
|
|
extract_interaction_tool_call_id(&
|
|
pv(r#"{"id":1,"method":"_x.ai/ask_user_question","params":{"method":"x.ai/ask_user_question","params":{"sessionId":"s","toolCallId":"tc-w"}}}"#))
|
|
.as_deref(), Some("tc-w")
|
|
);
|
|
assert_eq!(
|
|
extract_interaction_tool_call_id(&pv(r#"{"params":{}}"#)),
|
|
None
|
|
);
|
|
}
|
|
#[test]
|
|
fn extract_interaction_resolved_tool_call_id_matches_only_resolved() {
|
|
assert_eq!(
|
|
extract_interaction_resolved_tool_call_id(&
|
|
pv(r#"{"method":"x.ai/session_notification","params":{"sessionId":"s","update":{"sessionUpdate":"interaction_resolved","tool_call_id":"tc-r"}}}"#))
|
|
.as_deref(), Some("tc-r")
|
|
);
|
|
assert_eq!(
|
|
extract_interaction_resolved_tool_call_id(&
|
|
pv(r#"{"method":"_x.ai/session_notification","params":{"method":"x.ai/session_notification","params":{"sessionId":"s","update":{"sessionUpdate":"interaction_resolved","tool_call_id":"tc-rw"}}}}"#))
|
|
.as_deref(), Some("tc-rw")
|
|
);
|
|
assert_eq!(
|
|
extract_interaction_resolved_tool_call_id(&pv(
|
|
r#"{"method":"x.ai/session_notification","params":{"sessionId":"s","update":{"sessionUpdate":"pending_interaction","tool_call_id":"tc-r","kind":"permission"}}}"#
|
|
)),
|
|
None
|
|
);
|
|
}
|
|
#[test]
|
|
fn session_load_request_id_matches_response_id_for_buffer_flush() {
|
|
let mut req = pv(
|
|
r#"{"jsonrpc":"2.0","id":7,"method":"session/load","params":{"sessionId":"sess-x","cwd":"/tmp"}}"#,
|
|
);
|
|
assert!(is_session_load_request(&req));
|
|
assert_eq!(extract_session_id(&req).as_deref(), Some("sess-x"));
|
|
let client = ClientId(3);
|
|
let (stored_ns_id, _orig) = rewrite_request_id(&mut req, client).unwrap();
|
|
assert_eq!(stored_ns_id, "3|7");
|
|
assert_eq!(req["id"], stored_ns_id.as_str());
|
|
let mut response = pv(&format!(
|
|
r#"{{"jsonrpc":"2.0","id":"{stored_ns_id}","result":{{"models":[]}}}}"#
|
|
));
|
|
let (parsed_client, raw_response_id) = parse_response_id(&mut response).unwrap();
|
|
assert_eq!(parsed_client, client);
|
|
assert_eq!(raw_response_id, stored_ns_id);
|
|
assert_eq!(response["id"], serde_json::json!(7));
|
|
}
|
|
#[test]
|
|
fn live_buffer_holds_during_load_and_flushes_in_order() {
|
|
let client = ClientId(5);
|
|
let sid = "sess-y".to_string();
|
|
let mut pending_load_by_req: HashMap<String, (ClientId, String)> = HashMap::new();
|
|
let mut load_live_buffer: HashMap<(ClientId, String), Vec<BufferedLive>> = HashMap::new();
|
|
pending_load_by_req.insert("5|1".to_string(), (client, sid.clone()));
|
|
load_live_buffer.entry((client, sid.clone())).or_default();
|
|
for p in ["e1", "e2", "e3"] {
|
|
if let Some(buf) = load_live_buffer.get_mut(&(client, sid.clone())) {
|
|
buf.push((Arc::from(p), None));
|
|
}
|
|
}
|
|
assert_eq!(
|
|
load_live_buffer
|
|
.get(&(client, sid.clone()))
|
|
.unwrap()
|
|
.iter()
|
|
.map(|(p, _)| p.as_ref())
|
|
.collect::<Vec<_>>(),
|
|
["e1", "e2", "e3"]
|
|
);
|
|
let flushed = pending_load_by_req
|
|
.remove("5|1")
|
|
.and_then(|(c, s)| load_live_buffer.remove(&(c, s)))
|
|
.unwrap();
|
|
assert_eq!(
|
|
flushed.iter().map(|(p, _)| p.as_ref()).collect::<Vec<_>>(),
|
|
["e1", "e2", "e3"]
|
|
);
|
|
assert!(pending_load_by_req.is_empty());
|
|
assert!(load_live_buffer.is_empty());
|
|
pending_load_by_req.insert("5|2".to_string(), (client, sid.clone()));
|
|
load_live_buffer.entry((client, sid.clone())).or_default();
|
|
assert!(pending_load_by_req.remove("9|9").is_none());
|
|
assert!(load_live_buffer.contains_key(&(client, sid.clone())));
|
|
pending_load_by_req.retain(|_, (c, _)| *c != client);
|
|
load_live_buffer.retain(|(c, _), _| *c != client);
|
|
assert!(pending_load_by_req.is_empty());
|
|
assert!(load_live_buffer.is_empty());
|
|
}
|
|
/// An `agent_message_chunk` `session/update` carrying `eventId` at
|
|
/// `params._meta.eventId` (the live-broadcast wire shape).
|
|
fn live_chunk(sid: &str, seq: u64) -> String {
|
|
format!(
|
|
r#"{{"jsonrpc":"2.0","method":"session/update","params":{{"sessionId":"{sid}","update":{{"sessionUpdate":"agent_message_chunk","content":{{"type":"text","text":"x"}}}},"_meta":{{"eventId":"{sid}-{seq}"}}}}}}"#
|
|
)
|
|
}
|
|
#[test]
|
|
fn event_seq_of_parses_acp_and_ext_and_handles_missing() {
|
|
let acp = pv(r#"{"params":{"sessionId":"019e-aa","_meta":{"eventId":"019e-aa-42"}}}"#);
|
|
assert_eq!(event_seq_of(&acp), Some(42));
|
|
let ext = pv(
|
|
r#"{"params":{"method":"x.ai/session/update","params":{"sessionId":"019e-aa","_meta":{"eventId":"019e-aa-7"}}}}"#,
|
|
);
|
|
assert_eq!(event_seq_of(&ext), Some(7));
|
|
let none = pv(r#"{"params":{"sessionId":"019e-aa","_meta":{}}}"#);
|
|
assert_eq!(event_seq_of(&none), None);
|
|
}
|
|
/// Regression: on a mid-turn attach, the in-flight turn streams + persists
|
|
/// during the [subscribe -> gate-close] window, so its chunks are BOTH
|
|
/// buffered-live for the loading client AND read back by replay (same
|
|
/// eventId). The post-load flush must drop the buffered copies that replay
|
|
/// already delivered (`event_seq <= replay max`) and forward only the
|
|
/// genuinely-newer tail — so each event reaches the client exactly once.
|
|
#[test]
|
|
fn buffer_flush_drops_replay_overlap_by_event_seq() {
|
|
let client = ClientId(5);
|
|
let sid = "sess-z".to_string();
|
|
let mut load_live_buffer: HashMap<(ClientId, String), Vec<BufferedLive>> = HashMap::new();
|
|
let mut load_replay_max_seq: HashMap<(ClientId, String), u64> = HashMap::new();
|
|
for seq in 7..=21u64 {
|
|
let json = pv(&live_chunk(&sid, seq));
|
|
if let Some(s) = extract_session_id(&json)
|
|
&& let Some(n) = event_seq_of(&json)
|
|
{
|
|
let e = load_replay_max_seq.entry((client, s)).or_insert(0);
|
|
*e = (*e).max(n);
|
|
}
|
|
}
|
|
assert_eq!(load_replay_max_seq.get(&(client, sid.clone())), Some(&21));
|
|
let buf = load_live_buffer.entry((client, sid.clone())).or_default();
|
|
for seq in 7..=23u64 {
|
|
let payload = live_chunk(&sid, seq);
|
|
let event_seq = event_seq_of(&pv(&payload));
|
|
buf.push((payload.into(), event_seq));
|
|
}
|
|
let cutoff: Option<u64> = load_replay_max_seq.remove(&(client, sid.clone()));
|
|
let buffered = load_live_buffer.remove(&(client, sid.clone())).unwrap();
|
|
let mut forwarded: Vec<u64> = Vec::new();
|
|
for (_, buffered_seq) in &buffered {
|
|
if let Some(c) = cutoff
|
|
&& buffered_seq.is_some_and(|s| s <= c)
|
|
{
|
|
continue;
|
|
}
|
|
if let Some(s) = buffered_seq {
|
|
forwarded.push(*s);
|
|
}
|
|
}
|
|
assert_eq!(
|
|
forwarded,
|
|
vec![22, 23],
|
|
"only the post-replay tail is forwarded (overlap 7..=21 dropped)"
|
|
);
|
|
}
|
|
/// Edge case: a fresh process's very first event has `event_seq == 0`. The
|
|
/// cutoff must be an `Option` (not a `> 0` sentinel), so a genuine max of 0
|
|
/// still drops the buffered-live seq-0 duplicate instead of forwarding it.
|
|
#[test]
|
|
fn buffer_flush_drops_replay_overlap_at_seq_zero() {
|
|
let client = ClientId(5);
|
|
let sid = "sess-0".to_string();
|
|
let mut load_live_buffer: HashMap<(ClientId, String), Vec<BufferedLive>> = HashMap::new();
|
|
let mut load_replay_max_seq: HashMap<(ClientId, String), u64> = HashMap::new();
|
|
let json = pv(&live_chunk(&sid, 0));
|
|
if let Some(s) = extract_session_id(&json)
|
|
&& let Some(n) = event_seq_of(&json)
|
|
{
|
|
let e = load_replay_max_seq.entry((client, s)).or_insert(0);
|
|
*e = (*e).max(n);
|
|
}
|
|
assert_eq!(load_replay_max_seq.get(&(client, sid.clone())), Some(&0));
|
|
let buf = load_live_buffer.entry((client, sid.clone())).or_default();
|
|
for seq in [0u64, 1] {
|
|
let payload = live_chunk(&sid, seq);
|
|
let event_seq = event_seq_of(&pv(&payload));
|
|
buf.push((payload.into(), event_seq));
|
|
}
|
|
let cutoff: Option<u64> = load_replay_max_seq.remove(&(client, sid.clone()));
|
|
assert_eq!(
|
|
cutoff,
|
|
Some(0),
|
|
"a genuine cutoff of 0 must be Some(0), not absent"
|
|
);
|
|
let buffered = load_live_buffer.remove(&(client, sid.clone())).unwrap();
|
|
let mut forwarded: Vec<u64> = Vec::new();
|
|
for (_, buffered_seq) in &buffered {
|
|
if let Some(c) = cutoff
|
|
&& buffered_seq.is_some_and(|s| s <= c)
|
|
{
|
|
continue;
|
|
}
|
|
if let Some(s) = buffered_seq {
|
|
forwarded.push(*s);
|
|
}
|
|
}
|
|
assert_eq!(
|
|
forwarded,
|
|
vec![1],
|
|
"seq-0 duplicate dropped, seq-1 tail forwarded (Option cutoff, not > 0)"
|
|
);
|
|
}
|
|
#[test]
|
|
fn rewrite_request_id_skips_responses_with_result() {
|
|
let mut json = pv(r#"{"jsonrpc":"2.0","result":{"content":"hello"},"id":42}"#);
|
|
let before = json.clone();
|
|
assert!(rewrite_request_id(&mut json, ClientId(123)).is_none());
|
|
assert_eq!(json, before, "payload unchanged");
|
|
}
|
|
#[test]
|
|
fn rewrite_request_id_skips_responses_with_error() {
|
|
let mut json =
|
|
pv(r#"{"jsonrpc":"2.0","error":{"code":-32600,"message":"Invalid"},"id":5}"#);
|
|
let before = json.clone();
|
|
assert!(rewrite_request_id(&mut json, ClientId(123)).is_none());
|
|
assert_eq!(json, before, "payload unchanged");
|
|
}
|
|
#[test]
|
|
fn rewrite_request_id_handles_notifications() {
|
|
let mut json = pv(r#"{"jsonrpc":"2.0","method":"session/update","params":{}}"#);
|
|
let before = json.clone();
|
|
assert!(rewrite_request_id(&mut json, ClientId(123)).is_none());
|
|
assert_eq!(json, before, "payload unchanged");
|
|
assert!(json.get("id").is_none());
|
|
}
|
|
#[test]
|
|
fn rewrite_request_id_handles_string_ids() {
|
|
let mut json = pv(r#"{"jsonrpc":"2.0","method":"test","id":"abc-123"}"#);
|
|
let (namespaced_id, original_id) = rewrite_request_id(&mut json, ClientId(456)).unwrap();
|
|
assert_eq!(original_id, serde_json::json!("abc-123"));
|
|
assert_eq!(namespaced_id, "456|\"abc-123\"");
|
|
assert_eq!(json["id"], "456|\"abc-123\"");
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_adds_yolo_mode_to_session_new() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: true,
|
|
default_model: None,
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json["params"]["_meta"]["yoloMode"], true);
|
|
}
|
|
/// Leader capabilities.auto_mode seeds `_meta.autoMode` on session/new
|
|
/// (the real ConnectFlags.default_auto_mode entry path).
|
|
#[test]
|
|
fn inject_capabilities_adds_auto_mode_to_session_new() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
auto_mode: true,
|
|
yolo_mode: false,
|
|
default_model: None,
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json["params"]["_meta"]["autoMode"], true);
|
|
assert!(json["params"]["_meta"].get("yoloMode").is_none());
|
|
}
|
|
/// session/load also receives autoMode (reconnect path).
|
|
#[test]
|
|
fn inject_capabilities_adds_auto_mode_to_session_load() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"sessionId":"sess-1"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_load
|
|
);
|
|
let caps = ClientCapabilities {
|
|
auto_mode: true,
|
|
yolo_mode: false,
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"grok-tui",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json["params"]["_meta"]["autoMode"], true);
|
|
}
|
|
/// Yolo suppresses autoMode injection even when auto_mode capability is set.
|
|
#[test]
|
|
fn inject_capabilities_yolo_suppresses_auto_mode() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
auto_mode: true,
|
|
yolo_mode: true,
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json["params"]["_meta"]["yoloMode"], true);
|
|
assert!(
|
|
json["params"]["_meta"].get("autoMode").is_none(),
|
|
"yolo must not also inject autoMode"
|
|
);
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_skips_non_session_new() {
|
|
let mut json = pv(r#"{"jsonrpc":"2.0","method":"other/method","id":1,"params":{}}"#);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: true,
|
|
default_model: None,
|
|
..Default::default()
|
|
};
|
|
assert!(!inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert!(json["params"].get("_meta").is_none());
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_skips_when_yolo_mode_false() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: false,
|
|
default_model: None,
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
let before = json.clone();
|
|
assert!(!inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_preserves_existing_meta() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp","_meta":{{"foo":"bar"}}}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: true,
|
|
default_model: None,
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json["params"]["_meta"]["foo"], "bar");
|
|
assert_eq!(json["params"]["_meta"]["yoloMode"], true);
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_adds_default_model_to_session_new() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: false,
|
|
default_model: Some("grok-3-fast".to_string()),
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json["params"]["_meta"]["modelId"], "grok-3-fast");
|
|
assert!(json["params"]["_meta"].get("yoloMode").is_none());
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_adds_both_yolo_and_model() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: true,
|
|
default_model: Some("grok-3-fast".to_string()),
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json["params"]["_meta"]["yoloMode"], true);
|
|
assert_eq!(json["params"]["_meta"]["modelId"], "grok-3-fast");
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_does_not_override_existing_model_id() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp","_meta":{{"modelId":"custom-model"}}}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: false,
|
|
default_model: Some("grok-3-fast".to_string()),
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
inject_capabilities_into_session_new(&mut json, &caps, "", ClientId(1));
|
|
assert_eq!(json["params"]["_meta"]["modelId"], "custom-model");
|
|
}
|
|
#[test]
|
|
fn extract_yolo_mode_change_returns_value() {
|
|
let payload =
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/yolo_mode_changed","params":{"yolo_mode":true}}"#;
|
|
assert_eq!(extract_yolo_mode_change(&pv(payload)), Some(true));
|
|
let payload =
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/yolo_mode_changed","params":{"yolo_mode":false}}"#;
|
|
assert_eq!(extract_yolo_mode_change(&pv(payload)), Some(false));
|
|
}
|
|
#[test]
|
|
fn extract_yolo_mode_change_returns_none_for_other_methods() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"other/method","params":{"yolo_mode":true}}"#;
|
|
assert_eq!(extract_yolo_mode_change(&pv(payload)), None);
|
|
}
|
|
/// Branch 1: an explicit `auto_mode` flag wins, even over `permission_mode`.
|
|
#[test]
|
|
fn extract_auto_mode_change_explicit_flag_wins() {
|
|
let payload =
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/yolo_mode_changed","params":{"auto_mode":true}}"#;
|
|
assert_eq!(extract_auto_mode_change(&pv(payload)), Some(true));
|
|
let payload =
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/yolo_mode_changed","params":{"auto_mode":false}}"#;
|
|
assert_eq!(extract_auto_mode_change(&pv(payload)), Some(false));
|
|
let payload = r#"{"jsonrpc":"2.0","method":"x.ai/yolo_mode_changed","params":{"auto_mode":false,"permission_mode":"auto"}}"#;
|
|
assert_eq!(extract_auto_mode_change(&pv(payload)), Some(false));
|
|
}
|
|
/// Branch 2: with no explicit flag, derive from `permission_mode`.
|
|
#[test]
|
|
fn extract_auto_mode_change_derives_from_permission_mode() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"x.ai/yolo_mode_changed","params":{"permission_mode":"auto"}}"#;
|
|
assert_eq!(extract_auto_mode_change(&pv(payload)), Some(true));
|
|
for mode in ["ask", "always-approve", "default"] {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"x.ai/yolo_mode_changed","params":{{"permission_mode":"{mode}"}}}}"#
|
|
);
|
|
assert_eq!(
|
|
extract_auto_mode_change(&pv(&payload)),
|
|
Some(false),
|
|
"permission_mode={mode} must clear auto"
|
|
);
|
|
}
|
|
}
|
|
/// Branch 3: None when there's no auto signal — wrong method, or a bare yolo
|
|
/// toggle (no `auto_mode`, no `permission_mode`) must NOT change auto state.
|
|
#[test]
|
|
fn extract_auto_mode_change_returns_none_when_no_auto_signal() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"other/method","params":{"auto_mode":true}}"#;
|
|
assert_eq!(extract_auto_mode_change(&pv(payload)), None);
|
|
let payload =
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/yolo_mode_changed","params":{"yolo_mode":true}}"#;
|
|
assert_eq!(extract_auto_mode_change(&pv(payload)), None);
|
|
}
|
|
#[test]
|
|
fn extract_model_id_from_set_model_returns_value() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"sessionId":"sess-123","modelId":"grok-3-fast"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_set_model
|
|
);
|
|
assert_eq!(
|
|
extract_model_id_from_set_model(&pv(&payload)),
|
|
Some("grok-3-fast".to_string())
|
|
);
|
|
}
|
|
#[test]
|
|
fn extract_model_id_from_set_model_handles_snake_case() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"session_id":"sess-123","model_id":"grok-3"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_set_model
|
|
);
|
|
assert_eq!(
|
|
extract_model_id_from_set_model(&pv(&payload)),
|
|
Some("grok-3".to_string())
|
|
);
|
|
}
|
|
#[test]
|
|
fn extract_model_id_from_set_model_returns_none_for_other_methods() {
|
|
let payload =
|
|
r#"{"jsonrpc":"2.0","method":"other/method","id":1,"params":{"modelId":"grok-3"}}"#;
|
|
assert_eq!(extract_model_id_from_set_model(&pv(payload)), None);
|
|
}
|
|
#[test]
|
|
fn extract_model_id_from_set_model_returns_none_for_empty_model() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"sessionId":"sess-123","modelId":""}}}}"#,
|
|
AGENT_METHOD_NAMES.session_set_model
|
|
);
|
|
assert_eq!(extract_model_id_from_set_model(&pv(&payload)), None);
|
|
}
|
|
#[test]
|
|
fn extract_model_id_from_set_model_returns_none_for_missing_model() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"sessionId":"sess-123"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_set_model
|
|
);
|
|
assert_eq!(extract_model_id_from_set_model(&pv(&payload)), None);
|
|
}
|
|
#[test]
|
|
fn patch_initialize_response_patches_current_model_id() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"result":{"meta":{"modelState":{"currentModelId":"grok-3","availableModels":[]}}}}"#,
|
|
);
|
|
let default_model = Some("grok-3-fast".to_string());
|
|
assert!(patch_initialize_response_model(&mut json, &default_model));
|
|
assert_eq!(
|
|
json["result"]["meta"]["modelState"]["currentModelId"],
|
|
"grok-3-fast"
|
|
);
|
|
}
|
|
#[test]
|
|
fn patch_initialize_response_preserves_other_fields() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"result":{"meta":{"grokShell":true,"modelState":{"currentModelId":"grok-3","availableModels":[{"modelId":"grok-3"},{"modelId":"grok-3-fast"}]}}}}"#,
|
|
);
|
|
let default_model = Some("grok-3-fast".to_string());
|
|
assert!(patch_initialize_response_model(&mut json, &default_model));
|
|
assert_eq!(json["result"]["meta"]["grokShell"], true);
|
|
assert_eq!(
|
|
json["result"]["meta"]["modelState"]["currentModelId"],
|
|
"grok-3-fast"
|
|
);
|
|
assert_eq!(
|
|
json["result"]["meta"]["modelState"]["availableModels"]
|
|
.as_array()
|
|
.unwrap()
|
|
.len(),
|
|
2
|
|
);
|
|
}
|
|
#[test]
|
|
fn patch_initialize_response_noop_when_no_default_model() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"result":{"meta":{"modelState":{"currentModelId":"grok-3"}}}}"#,
|
|
);
|
|
let before = json.clone();
|
|
assert!(!patch_initialize_response_model(&mut json, &None));
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn patch_initialize_response_noop_when_empty_default_model() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"result":{"meta":{"modelState":{"currentModelId":"grok-3"}}}}"#,
|
|
);
|
|
let before = json.clone();
|
|
assert!(!patch_initialize_response_model(
|
|
&mut json,
|
|
&Some("".to_string())
|
|
));
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn patch_initialize_response_noop_when_already_matches() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"result":{"meta":{"modelState":{"currentModelId":"grok-3"}}}}"#,
|
|
);
|
|
let before = json.clone();
|
|
assert!(!patch_initialize_response_model(
|
|
&mut json,
|
|
&Some("grok-3".to_string())
|
|
));
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn patch_initialize_response_noop_for_non_initialize_response() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","id":1,"result":{"session_id":"sess-1","models":{"currentModelId":"grok-3","availableModels":[]}}}"#,
|
|
);
|
|
let before = json.clone();
|
|
assert!(!patch_initialize_response_model(
|
|
&mut json,
|
|
&Some("grok-3-fast".to_string())
|
|
));
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn extract_session_id_from_result_works() {
|
|
let payload = r#"{"jsonrpc":"2.0","result":{"session_id":"sess-123"},"id":1}"#;
|
|
assert_eq!(
|
|
extract_session_id_from_result(&pv(payload)),
|
|
Some("sess-123".to_string())
|
|
);
|
|
let payload = r#"{"jsonrpc":"2.0","result":{"sessionId":"sess-456"},"id":1}"#;
|
|
assert_eq!(
|
|
extract_session_id_from_result(&pv(payload)),
|
|
Some("sess-456".to_string())
|
|
);
|
|
}
|
|
#[test]
|
|
fn extract_session_id_from_result_returns_none_for_other_responses() {
|
|
let payload = r#"{"jsonrpc":"2.0","result":{"other":"value"},"id":1}"#;
|
|
assert_eq!(extract_session_id_from_result(&pv(payload)), None);
|
|
let payload = r#"{"jsonrpc":"2.0","error":{"code":-1,"message":"fail"},"id":1}"#;
|
|
assert_eq!(extract_session_id_from_result(&pv(payload)), None);
|
|
let payload = r#"{"jsonrpc":"2.0","method":"test","params":{"session_id":"abc"},"id":1}"#;
|
|
assert_eq!(extract_session_id_from_result(&pv(payload)), None);
|
|
}
|
|
#[test]
|
|
fn extract_session_id_from_params_works() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"session/notification","params":{"session_id":"sess-789"}}"#;
|
|
assert_eq!(
|
|
extract_session_id(&pv(payload)),
|
|
Some("sess-789".to_string())
|
|
);
|
|
let payload = r#"{"jsonrpc":"2.0","method":"session/notification","params":{"sessionId":"sess-abc"}}"#;
|
|
assert_eq!(
|
|
extract_session_id(&pv(payload)),
|
|
Some("sess-abc".to_string())
|
|
);
|
|
}
|
|
#[test]
|
|
fn extract_session_id_from_nested_params_works() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"_x.ai/session_notification","params":{"method":"x.ai/session_notification","params":{"sessionId":"sess-nested"}}}"#;
|
|
assert_eq!(
|
|
extract_session_id(&pv(payload)),
|
|
Some("sess-nested".to_string())
|
|
);
|
|
let payload = r#"{"jsonrpc":"2.0","method":"_x.ai/fs_notify","params":{"method":"x.ai/fs_notify","params":{"session_id":"sess-nested-2","event":{}}}}"#;
|
|
assert_eq!(
|
|
extract_session_id(&pv(payload)),
|
|
Some("sess-nested-2".to_string())
|
|
);
|
|
let payload = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"top-level","params":{"sessionId":"nested"}}}"#;
|
|
assert_eq!(
|
|
extract_session_id(&pv(payload)),
|
|
Some("top-level".to_string())
|
|
);
|
|
}
|
|
#[test]
|
|
fn extract_session_id_from_prompt_complete_works() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"x.ai/session/prompt_complete","params":{"sessionId":"sess-prompt"}}"#;
|
|
assert_eq!(
|
|
extract_session_id_from_prompt_complete(&pv(payload)),
|
|
Some("sess-prompt".to_string())
|
|
);
|
|
}
|
|
#[test]
|
|
fn extract_session_id_from_prompt_complete_ignores_other_methods() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-prompt"}}"#;
|
|
assert_eq!(extract_session_id_from_prompt_complete(&pv(payload)), None);
|
|
}
|
|
#[test]
|
|
fn extract_child_session_event_spawned() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"parent","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-1"}}}"#;
|
|
match extract_child_session_event(&pv(payload)) {
|
|
Some(ChildSessionEvent::Spawned(id)) => assert_eq!(id, "child-1"),
|
|
other => panic!("Expected Spawned, got {:?}", other),
|
|
}
|
|
}
|
|
#[test]
|
|
fn extract_child_session_event_finished() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"parent","update":{"sessionUpdate":"subagent_finished","child_session_id":"child-2"}}}"#;
|
|
match extract_child_session_event(&pv(payload)) {
|
|
Some(ChildSessionEvent::Finished(id)) => assert_eq!(id, "child-2"),
|
|
other => panic!("Expected Finished, got {:?}", other),
|
|
}
|
|
}
|
|
#[test]
|
|
fn extract_child_session_event_nested_ext_notification() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"_x.ai/session_notification","params":{"method":"x.ai/session_notification","params":{"sessionId":"parent","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-3"}}}}"#;
|
|
match extract_child_session_event(&pv(payload)) {
|
|
Some(ChildSessionEvent::Spawned(id)) => assert_eq!(id, "child-3"),
|
|
other => panic!("Expected Spawned, got {:?}", other),
|
|
}
|
|
}
|
|
#[test]
|
|
fn extract_child_session_event_nested_ext_notification_finished() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"_x.ai/session_notification","params":{"method":"x.ai/session_notification","params":{"sessionId":"parent","update":{"sessionUpdate":"subagent_finished","child_session_id":"child-4"}}}}"#;
|
|
match extract_child_session_event(&pv(payload)) {
|
|
Some(ChildSessionEvent::Finished(id)) => assert_eq!(id, "child-4"),
|
|
other => panic!("Expected Finished, got {:?}", other),
|
|
}
|
|
}
|
|
#[test]
|
|
fn extract_child_session_event_none_for_other_updates() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"parent","update":{"sessionUpdate":"message_delta","content":"hello"}}}"#;
|
|
assert!(extract_child_session_event(&pv(payload)).is_none());
|
|
}
|
|
#[test]
|
|
fn extract_child_session_event_none_without_child_id() {
|
|
let payload = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"parent","update":{"sessionUpdate":"subagent_spawned"}}}"#;
|
|
assert!(extract_child_session_event(&pv(payload)).is_none());
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_skips_empty_default_model() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: false,
|
|
default_model: Some("".to_string()),
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
let before = json.clone();
|
|
assert!(!inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_skips_empty_model_with_yolo_mode() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: true,
|
|
default_model: Some("".to_string()),
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json["params"]["_meta"]["yoloMode"], true);
|
|
assert!(json["params"]["_meta"].get("modelId").is_none());
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_no_model_no_yolo_returns_unchanged() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp","_meta":{{"yoloMode":true}}}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: false,
|
|
default_model: None,
|
|
..Default::default()
|
|
};
|
|
let mut json = pv(&payload);
|
|
let before = json.clone();
|
|
assert!(!inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"",
|
|
ClientId(1)
|
|
));
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_adds_client_identifier_to_session_new() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities::default();
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"grok-code-extension",
|
|
ClientId(1),
|
|
));
|
|
assert_eq!(
|
|
json["params"]["_meta"]["clientIdentifier"],
|
|
"grok-code-extension"
|
|
);
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_does_not_override_existing_client_identifier() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"cwd":"/tmp","_meta":{{"clientIdentifier":"custom-client"}}}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
let caps = ClientCapabilities::default();
|
|
let mut json = pv(&payload);
|
|
inject_capabilities_into_session_new(&mut json, &caps, "grok-tui", ClientId(1));
|
|
assert_eq!(json["params"]["_meta"]["clientIdentifier"], "custom-client");
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_adds_client_identifier_to_session_load() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"sessionId":"sess-1"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_load
|
|
);
|
|
let caps = ClientCapabilities::default();
|
|
let mut json = pv(&payload);
|
|
assert!(inject_capabilities_into_session_new(
|
|
&mut json,
|
|
&caps,
|
|
"grok-code-extension",
|
|
ClientId(1),
|
|
));
|
|
assert_eq!(
|
|
json["params"]["_meta"]["clientIdentifier"],
|
|
"grok-code-extension"
|
|
);
|
|
assert!(json["params"]["_meta"].get("yoloMode").is_none());
|
|
assert!(json["params"]["_meta"].get("modelId").is_none());
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_adds_leader_client_id_to_session_load() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"sessionId":"sess-1"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_load
|
|
);
|
|
let caps = ClientCapabilities::default();
|
|
let mut json = pv(&payload);
|
|
inject_capabilities_into_session_new(&mut json, &caps, "grok-tui", ClientId(42));
|
|
assert_eq!(
|
|
json["params"]["_meta"]["x.ai/leaderClientId"].as_u64(),
|
|
Some(42)
|
|
);
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_does_not_override_existing_leader_client_id() {
|
|
let payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"sessionId":"sess-1","_meta":{{"x.ai/leaderClientId":7}}}}}}"#,
|
|
AGENT_METHOD_NAMES.session_load
|
|
);
|
|
let caps = ClientCapabilities::default();
|
|
let mut json = pv(&payload);
|
|
inject_capabilities_into_session_new(&mut json, &caps, "grok-tui", ClientId(42));
|
|
assert_eq!(
|
|
json["params"]["_meta"]["x.ai/leaderClientId"].as_u64(),
|
|
Some(7)
|
|
);
|
|
}
|
|
#[test]
|
|
fn extract_target_client_id_some_when_meta_present() {
|
|
let direct = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"sess-1","_meta":{"x.ai/leaderClientId":9}}}"#;
|
|
assert_eq!(extract_target_client_id(&pv(direct)), Some(ClientId(9)));
|
|
let nested = r#"{"jsonrpc":"2.0","method":"_x.ai/session/update","params":{"params":{"sessionId":"sess-1","_meta":{"x.ai/leaderClientId":11}}}}"#;
|
|
assert_eq!(extract_target_client_id(&pv(nested)), Some(ClientId(11)));
|
|
}
|
|
#[test]
|
|
fn extract_target_client_id_none_when_absent() {
|
|
let no_meta =
|
|
r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"sess-1"}}"#;
|
|
assert_eq!(extract_target_client_id(&pv(no_meta)), None);
|
|
let no_key = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"sess-1","_meta":{"isReplay":true}}}"#;
|
|
assert_eq!(extract_target_client_id(&pv(no_key)), None);
|
|
}
|
|
#[test]
|
|
fn inject_yolo_notification_adds_client_identifier() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/yolo_mode_changed","params":{"yolo_mode":true}}"#,
|
|
);
|
|
assert!(inject_client_identity_into_yolo_notification(
|
|
&mut json, "grok-tui"
|
|
));
|
|
assert_eq!(json["params"]["clientIdentifier"], "grok-tui");
|
|
assert_eq!(json["params"]["yolo_mode"], true);
|
|
}
|
|
#[test]
|
|
fn inject_yolo_notification_skips_non_yolo_methods() {
|
|
let mut json = pv(r#"{"jsonrpc":"2.0","method":"x.ai/other","params":{"data":1}}"#);
|
|
let before = json.clone();
|
|
assert!(!inject_client_identity_into_yolo_notification(
|
|
&mut json, "grok-tui"
|
|
));
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn inject_client_identity_adds_identifier_to_initialize() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"0.1"}}"#,
|
|
);
|
|
let (mutated, was_initialize) =
|
|
inject_client_identity_into_initialize(&mut json, "grok-tui");
|
|
assert!(was_initialize, "should have detected an initialize message");
|
|
assert!(mutated, "should have injected the identifier");
|
|
assert_eq!(json["params"]["_meta"]["clientIdentifier"], "grok-tui");
|
|
}
|
|
#[test]
|
|
fn inject_client_identity_does_not_override_existing() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"0.1","_meta":{"clientIdentifier":"grok-web"}}}"#,
|
|
);
|
|
let (mutated, was_initialize) =
|
|
inject_client_identity_into_initialize(&mut json, "grok-tui");
|
|
assert!(was_initialize, "should have detected an initialize message");
|
|
assert!(!mutated, "existing identifier means nothing was injected");
|
|
assert_eq!(json["params"]["_meta"]["clientIdentifier"], "grok-web");
|
|
}
|
|
#[test]
|
|
fn inject_client_identity_skips_non_initialize() {
|
|
let mut json =
|
|
pv(r#"{"jsonrpc":"2.0","method":"session/new","id":1,"params":{"cwd":"/tmp"}}"#);
|
|
let before = json.clone();
|
|
let (mutated, was_initialize) =
|
|
inject_client_identity_into_initialize(&mut json, "grok-tui");
|
|
assert!(
|
|
!was_initialize,
|
|
"session/new should not be detected as initialize"
|
|
);
|
|
assert!(!mutated);
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn inject_client_identity_skips_empty_client_type() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"0.1"}}"#,
|
|
);
|
|
let before = json.clone();
|
|
let (mutated, was_initialize) = inject_client_identity_into_initialize(&mut json, "");
|
|
assert!(
|
|
!was_initialize,
|
|
"empty client_type means no injection, not an initialize"
|
|
);
|
|
assert!(!mutated);
|
|
assert_eq!(json, before);
|
|
}
|
|
#[test]
|
|
fn inject_client_identity_preserves_existing_meta() {
|
|
let mut json = pv(
|
|
r#"{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"0.1","_meta":{"foo":"bar"}}}"#,
|
|
);
|
|
let (mutated, was_initialize) =
|
|
inject_client_identity_into_initialize(&mut json, "grok-code-extension");
|
|
assert!(was_initialize, "should have detected an initialize message");
|
|
assert!(mutated);
|
|
assert_eq!(json["params"]["_meta"]["foo"], "bar");
|
|
assert_eq!(
|
|
json["params"]["_meta"]["clientIdentifier"],
|
|
"grok-code-extension"
|
|
);
|
|
}
|
|
#[test]
|
|
fn version_mismatch_notification_contains_correct_fields() {
|
|
let payload = make_version_mismatch_notification("0.1.157", "0.1.150")
|
|
.expect("should produce notification");
|
|
let json: serde_json::Value = serde_json::from_str(&payload).unwrap();
|
|
assert_eq!(json["method"], "x.ai/leader/version_mismatch");
|
|
assert_eq!(json["params"]["clientVersion"], "0.1.157");
|
|
assert_eq!(json["params"]["leaderVersion"], "0.1.150");
|
|
assert!(
|
|
json["params"]["message"]
|
|
.as_str()
|
|
.unwrap_or("")
|
|
.contains("0.1.157"),
|
|
"message should mention the client version"
|
|
);
|
|
}
|
|
#[test]
|
|
fn version_mismatch_notification_is_none_when_versions_match() {
|
|
assert!(
|
|
make_version_mismatch_notification("0.1.150", "0.1.150").is_none(),
|
|
"matching versions must not produce a notification"
|
|
);
|
|
}
|
|
#[test]
|
|
fn version_mismatch_notification_is_none_for_unknown_leader_version() {
|
|
assert!(
|
|
make_version_mismatch_notification("0.1.150", "unknown").is_none(),
|
|
"unknown leader version (dev build) must not produce a notification"
|
|
);
|
|
}
|
|
/// Verify that a session/setModel request updates the client's default_model
|
|
/// capability, so the next session/new injects the updated model.
|
|
#[tokio::test]
|
|
async fn set_model_updates_default_model_for_next_session_new() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, mut acp_rx) = setup_test_server(&temp).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities {
|
|
yolo_mode: false,
|
|
default_model: Some("grok-original".to_string()),
|
|
..Default::default()
|
|
},
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
let set_model_payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":1,"params":{{"sessionId":"sess-1","modelId":"grok-4.5"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_set_model
|
|
);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: set_model_payload,
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _ = acp_rx.recv().await.unwrap();
|
|
let session_new_payload = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"{}","id":2,"params":{{"cwd":"/tmp"}}}}"#,
|
|
AGENT_METHOD_NAMES.session_new
|
|
);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: session_new_payload,
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let forwarded = acp_rx.recv().await.unwrap();
|
|
let json: serde_json::Value = serde_json::from_str(&forwarded).unwrap();
|
|
assert_eq!(
|
|
json["params"]["_meta"]["modelId"], "grok-4.5",
|
|
"Leader should inject the updated model after session/setModel, not the stale registration model"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn client_count_starts_at_zero() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (_sock_path, cancel, _acp_rx, client_count) =
|
|
setup_test_server_with_client_count(&temp).await;
|
|
assert_eq!(
|
|
client_count.load(Ordering::Relaxed),
|
|
0,
|
|
"client_count should start at 0"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn client_count_increments_on_connect() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, _acp_rx, client_count) =
|
|
setup_test_server_with_client_count(&temp).await;
|
|
let (_reader1, _writer1) = connect_and_register(&sock_path, "client-1").await;
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
assert_eq!(
|
|
client_count.load(Ordering::Relaxed),
|
|
1,
|
|
"client_count should be 1 after one client connects"
|
|
);
|
|
let (_reader2, _writer2) = connect_and_register(&sock_path, "client-2").await;
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
assert_eq!(
|
|
client_count.load(Ordering::Relaxed),
|
|
2,
|
|
"client_count should be 2 after two clients connect"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn client_count_decrements_on_disconnect() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, _acp_rx, client_count) =
|
|
setup_test_server_with_client_count(&temp).await;
|
|
let (_reader1, mut writer1) = connect_and_register(&sock_path, "client-1").await;
|
|
let (_reader2, _writer2) = connect_and_register(&sock_path, "client-2").await;
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
assert_eq!(client_count.load(Ordering::Relaxed), 2);
|
|
write_message(&mut writer1, &ClientMessage::Disconnect)
|
|
.await
|
|
.unwrap();
|
|
drop(_reader1);
|
|
drop(writer1);
|
|
tokio::time::sleep(Duration::from_millis(200)).await;
|
|
assert_eq!(
|
|
client_count.load(Ordering::Relaxed),
|
|
1,
|
|
"client_count should be 1 after one client disconnects"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn client_count_returns_to_zero_after_all_disconnect() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("test.sock");
|
|
let (acp_tx, _acp_rx) = mpsc::unbounded_channel();
|
|
let (_response_tx, response_rx) = mpsc::unbounded_channel();
|
|
let server_cancel = CancellationToken::new();
|
|
let client_count = Arc::new(AtomicUsize::new(0));
|
|
let agent_busy = Arc::new(AtomicBool::new(false));
|
|
let control_state = default_test_control_state(&sock_path);
|
|
let sock_clone = sock_path.clone();
|
|
let cancel_clone = server_cancel.clone();
|
|
let count_clone = client_count.clone();
|
|
let busy_clone = agent_busy.clone();
|
|
tokio::spawn(async move {
|
|
let _ = run_leader_server(
|
|
sock_clone,
|
|
acp_tx,
|
|
response_rx,
|
|
cancel_clone,
|
|
true,
|
|
count_clone,
|
|
busy_clone,
|
|
AgentActivity::default(),
|
|
watch::channel(true).1,
|
|
watch::channel(super::super::protocol::ShutdownReason::Manual).0,
|
|
None,
|
|
control_state,
|
|
)
|
|
.await;
|
|
});
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
{
|
|
let (_reader, mut writer) = connect_and_register(&sock_path, "temp-client").await;
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
assert_eq!(client_count.load(Ordering::Relaxed), 1);
|
|
write_message(&mut writer, &ClientMessage::Disconnect)
|
|
.await
|
|
.unwrap();
|
|
}
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
assert_eq!(
|
|
client_count.load(Ordering::Relaxed),
|
|
0,
|
|
"client_count should return to 0 after all clients disconnect"
|
|
);
|
|
server_cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn client_count_not_incremented_before_registration() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (_sock_path, cancel, _acp_rx, client_count) =
|
|
setup_test_server_with_client_count(&temp).await;
|
|
let _stream = LeaderStream::connect(&_sock_path).await.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
assert_eq!(
|
|
client_count.load(Ordering::Relaxed),
|
|
0,
|
|
"client_count should remain 0 for unregistered connections"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn fallback_routing_forwards_notifications_but_drops_responses() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("test.sock");
|
|
let (acp_tx, _acp_rx) = mpsc::unbounded_channel();
|
|
let (response_tx, response_rx) = mpsc::unbounded_channel();
|
|
let server_cancel = CancellationToken::new();
|
|
let client_count = Arc::new(AtomicUsize::new(0));
|
|
let control_state = default_test_control_state(&sock_path);
|
|
let sock_clone = sock_path.clone();
|
|
let cancel_clone = server_cancel.clone();
|
|
let count_clone = client_count.clone();
|
|
tokio::spawn(async move {
|
|
let _ = run_leader_server(
|
|
sock_clone,
|
|
acp_tx,
|
|
response_rx,
|
|
cancel_clone,
|
|
true,
|
|
count_clone,
|
|
Arc::new(AtomicBool::new(false)),
|
|
AgentActivity::default(),
|
|
watch::channel(true).1,
|
|
watch::channel(super::super::protocol::ShutdownReason::Manual).0,
|
|
None,
|
|
control_state,
|
|
)
|
|
.await;
|
|
});
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"test","id":99}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(r#"{"jsonrpc":"2.0","result":{"ok":true},"id":42}"#.to_string())
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"agent/progress","params":{"status":"working"}}"#
|
|
.to_string(),
|
|
)
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let msg: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader))
|
|
.await
|
|
.expect("should receive notification")
|
|
.unwrap();
|
|
match msg {
|
|
ServerMessage::Acp { payload } => {
|
|
let json: serde_json::Value = serde_json::from_str(&payload).unwrap();
|
|
assert_eq!(
|
|
json["method"], "agent/progress",
|
|
"Should receive the notification, not the relay response"
|
|
);
|
|
}
|
|
other => panic!("Expected Acp message, got {:?}", other),
|
|
}
|
|
server_cancel.cancel();
|
|
}
|
|
/// Relay-originated session notifications must be dropped, not forwarded
|
|
/// to the last active IPC client.
|
|
#[tokio::test]
|
|
async fn relay_session_notification_not_forwarded_to_ipc_client() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let (mut reader, mut writer) = connect_and_register(&sock_path, "test").await;
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"test","id":99}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"relay-sess-xyz","data":"from-relay"}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"agent/progress","params":{"status":"working"}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let msg: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader))
|
|
.await
|
|
.expect("should receive the session-less notification")
|
|
.unwrap();
|
|
match msg {
|
|
ServerMessage::Acp { payload } => {
|
|
let json: serde_json::Value = serde_json::from_str(&payload).unwrap();
|
|
assert_eq!(json["method"], "agent/progress");
|
|
assert!(json["params"].get("sessionId").is_none());
|
|
}
|
|
other => panic!("Expected Acp message, got {:?}", other),
|
|
}
|
|
cancel.cancel();
|
|
}
|
|
/// When a client disconnects while its session streams, notifications for
|
|
/// that session must NOT leak to another client via `last_active_client`.
|
|
#[tokio::test]
|
|
async fn dead_client_session_notification_not_leaked_to_other_client() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let (reader_a, mut writer_a) = connect_and_register(&sock_path, "test-a").await;
|
|
write_message(
|
|
&mut writer_a,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"session/prompt","id":1,"params":{"sessionId":"sess-A","prompt":[]}}"#
|
|
.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
drop(writer_a);
|
|
drop(reader_a);
|
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "test-b").await;
|
|
write_message(
|
|
&mut writer_b,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"initialize","id":2,"params":{}}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"sess-A","sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"leaked content"}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"agent/progress","params":{"status":"working"}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let msg: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader_b))
|
|
.await
|
|
.expect("should receive the session-less notification")
|
|
.unwrap();
|
|
match msg {
|
|
ServerMessage::Acp { payload } => {
|
|
let json: serde_json::Value = serde_json::from_str(&payload).unwrap();
|
|
assert_eq!(json["method"], "agent/progress");
|
|
}
|
|
other => panic!("Expected Acp message, got {:?}", other),
|
|
}
|
|
cancel.cancel();
|
|
}
|
|
/// `ext/notification` with nested sessionId (params.params.sessionId) must
|
|
/// route to the session owner, not fall through to `last_active_client`.
|
|
#[tokio::test]
|
|
async fn ext_notification_with_nested_session_id_routes_correctly() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "test-a").await;
|
|
write_message(
|
|
&mut writer_a,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"session/prompt","id":1,"params":{"sessionId":"sess-A","prompt":[]}}"#
|
|
.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "test-b").await;
|
|
write_message(
|
|
&mut writer_b,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"initialize","id":2,"params":{}}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"_x.ai/session_notification","params":{"method":"x.ai/session_notification","params":{"sessionId":"sess-A","update":{"sessionUpdate":"retry_state","attempt":1,"maxRetries":3,"reason":"transient"}}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let msg: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader_a))
|
|
.await
|
|
.expect("client A should receive the ext/notification")
|
|
.unwrap();
|
|
match msg {
|
|
ServerMessage::Acp { payload } => {
|
|
let json: serde_json::Value = serde_json::from_str(&payload).unwrap();
|
|
assert_eq!(json["method"], "_x.ai/session_notification");
|
|
}
|
|
other => panic!("Expected Acp message, got {:?}", other),
|
|
}
|
|
let timeout_result: Result<Result<ServerMessage, _>, _> =
|
|
tokio::time::timeout(Duration::from_millis(100), read_message(&mut reader_b)).await;
|
|
assert!(
|
|
timeout_result.is_err(),
|
|
"Client B should NOT receive session A's notification"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn server_sends_shutting_down_before_shutdown() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("test.sock");
|
|
let (acp_tx, _acp_rx) = mpsc::unbounded_channel();
|
|
let (_response_tx, response_rx) = mpsc::unbounded_channel();
|
|
let cancel = CancellationToken::new();
|
|
let client_count = Arc::new(AtomicUsize::new(0));
|
|
let control_state = default_test_control_state(&sock_path);
|
|
let cancel_clone = cancel.clone();
|
|
let sock_clone = sock_path.clone();
|
|
let cc = client_count.clone();
|
|
tokio::spawn(async move {
|
|
let _ = run_leader_server(
|
|
sock_clone,
|
|
acp_tx,
|
|
response_rx,
|
|
cancel_clone,
|
|
true,
|
|
cc,
|
|
Arc::new(AtomicBool::new(false)),
|
|
AgentActivity::default(),
|
|
watch::channel(true).1,
|
|
watch::channel(super::super::protocol::ShutdownReason::Manual).0,
|
|
None,
|
|
control_state,
|
|
)
|
|
.await;
|
|
});
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (mut reader, _writer) = connect_and_register(&sock_path, "test").await;
|
|
cancel.cancel();
|
|
let msg1: ServerMessage =
|
|
tokio::time::timeout(Duration::from_secs(5), read_message(&mut reader))
|
|
.await
|
|
.expect("should receive ShuttingDown")
|
|
.unwrap();
|
|
match msg1 {
|
|
ServerMessage::ShuttingDown { reason, delay_ms } => {
|
|
assert_eq!(
|
|
reason,
|
|
super::super::protocol::ShutdownReason::Manual,
|
|
"Reason should be Manual"
|
|
);
|
|
assert_eq!(delay_ms, 0, "delay_ms should be 0 (immediate shutdown)");
|
|
}
|
|
other => panic!("Expected ShuttingDown, got {:?}", other),
|
|
}
|
|
let msg2: ServerMessage =
|
|
tokio::time::timeout(Duration::from_secs(5), read_message(&mut reader))
|
|
.await
|
|
.expect("should receive Shutdown")
|
|
.unwrap();
|
|
assert!(
|
|
matches!(msg2, ServerMessage::Shutdown),
|
|
"Expected Shutdown, got {:?}",
|
|
msg2
|
|
);
|
|
}
|
|
#[tokio::test]
|
|
async fn agent_busy_set_when_request_forwarded() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("busy_test.sock");
|
|
let mut handle = spawn_leader_server(sock_path.clone()).await.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
assert!(
|
|
!handle.agent_busy.load(Ordering::Relaxed),
|
|
"agent_busy should be false initially"
|
|
);
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"test/ping","id":1}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let forwarded = handle.acp_rx.recv().await.unwrap();
|
|
assert!(forwarded.contains("test/ping"));
|
|
assert!(
|
|
handle.agent_busy.load(Ordering::Relaxed),
|
|
"agent_busy should be true after forwarding a request"
|
|
);
|
|
handle.cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn agent_busy_cleared_when_response_received() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("busy_clear.sock");
|
|
let mut handle = spawn_leader_server(sock_path.clone()).await.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"test/ping","id":42}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let forwarded = handle.acp_rx.recv().await.unwrap();
|
|
let json: serde_json::Value = serde_json::from_str(&forwarded).unwrap();
|
|
let namespaced_id = json["id"].as_str().unwrap().to_string();
|
|
assert!(handle.agent_busy.load(Ordering::Relaxed));
|
|
let response = format!(
|
|
r#"{{"jsonrpc":"2.0","result":{{"ok":true}},"id":"{}"}}"#,
|
|
namespaced_id
|
|
);
|
|
handle.response_tx.send(response).unwrap();
|
|
let client_resp: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
assert!(matches!(client_resp, ServerMessage::Acp { .. }));
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
assert!(
|
|
!handle.agent_busy.load(Ordering::Relaxed),
|
|
"agent_busy should be false after response is routed"
|
|
);
|
|
handle.cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn agent_busy_tracks_multiple_pending_requests() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("busy_multi.sock");
|
|
let mut handle = spawn_leader_server(sock_path.clone()).await.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"test/a","id":1}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"test/b","id":2}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let fwd1 = handle.acp_rx.recv().await.unwrap();
|
|
let fwd2 = handle.acp_rx.recv().await.unwrap();
|
|
let id1 = serde_json::from_str::<serde_json::Value>(&fwd1).unwrap()["id"]
|
|
.as_str()
|
|
.unwrap()
|
|
.to_string();
|
|
let id2 = serde_json::from_str::<serde_json::Value>(&fwd2).unwrap()["id"]
|
|
.as_str()
|
|
.unwrap()
|
|
.to_string();
|
|
assert!(handle.agent_busy.load(Ordering::Relaxed));
|
|
handle
|
|
.response_tx
|
|
.send(format!(
|
|
r#"{{"jsonrpc":"2.0","result":{{}},"id":"{}"}}"#,
|
|
id1
|
|
))
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
assert!(
|
|
handle.agent_busy.load(Ordering::Relaxed),
|
|
"agent_busy should still be true with one request pending"
|
|
);
|
|
handle
|
|
.response_tx
|
|
.send(format!(
|
|
r#"{{"jsonrpc":"2.0","result":{{}},"id":"{}"}}"#,
|
|
id2
|
|
))
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
assert!(
|
|
!handle.agent_busy.load(Ordering::Relaxed),
|
|
"agent_busy should be false after all responses received"
|
|
);
|
|
handle.cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn agent_busy_clears_when_client_disconnects_mid_request() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("busy_disconnect.sock");
|
|
let (acp_tx, mut acp_rx) = mpsc::unbounded_channel();
|
|
let (response_tx, response_rx) = mpsc::unbounded_channel();
|
|
let cancel = CancellationToken::new();
|
|
let client_count = Arc::new(AtomicUsize::new(0));
|
|
let agent_busy = Arc::new(AtomicBool::new(false));
|
|
let control_state = default_test_control_state(&sock_path);
|
|
let sock_clone = sock_path.clone();
|
|
let cancel_clone = cancel.clone();
|
|
let count_clone = client_count.clone();
|
|
let busy_clone = agent_busy.clone();
|
|
tokio::spawn(async move {
|
|
let _ = run_leader_server(
|
|
sock_clone,
|
|
acp_tx,
|
|
response_rx,
|
|
cancel_clone,
|
|
true,
|
|
count_clone,
|
|
busy_clone,
|
|
AgentActivity::default(),
|
|
watch::channel(true).1,
|
|
watch::channel(super::super::protocol::ShutdownReason::Manual).0,
|
|
None,
|
|
control_state,
|
|
)
|
|
.await;
|
|
});
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let namespaced_id = {
|
|
let stream = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: "test".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let _: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"test/slow","id":1}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let forwarded = acp_rx.recv().await.unwrap();
|
|
let json: serde_json::Value = serde_json::from_str(&forwarded).unwrap();
|
|
let id = json["id"].as_str().unwrap().to_string();
|
|
assert!(
|
|
agent_busy.load(Ordering::Relaxed),
|
|
"should be busy after request"
|
|
);
|
|
write_message(&mut writer, &ClientMessage::Disconnect)
|
|
.await
|
|
.unwrap();
|
|
id
|
|
};
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
assert!(
|
|
agent_busy.load(Ordering::Relaxed),
|
|
"agent_busy should still be true after client disconnect (request still pending)"
|
|
);
|
|
response_tx
|
|
.send(format!(
|
|
r#"{{"jsonrpc":"2.0","result":{{"done":true}},"id":"{}"}}"#,
|
|
namespaced_id
|
|
))
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
assert!(
|
|
!agent_busy.load(Ordering::Relaxed),
|
|
"agent_busy should be false after response arrives (even though client disconnected)"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// Regression: bounded(256) client channel + try_send silently dropped
|
|
/// notifications during session replay bursts. Unbounded channel fixes this.
|
|
#[tokio::test]
|
|
async fn high_throughput_replay_no_drops() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader, mut writer) = connect_and_register(&sock_path, "grok-tui").await;
|
|
let load_req = r#"{"jsonrpc":"2.0","method":"session/load","id":1,"params":{"session_id":"sess_replay"}}"#;
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: load_req.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader).await;
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
const REPLAY_COUNT: usize = 500;
|
|
for i in 0..REPLAY_COUNT {
|
|
let notification = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"session/notification","params":{{"session_id":"sess_replay","updates":[{{"type":"message_start","message_id":"msg_{i}"}}]}}}}"#,
|
|
);
|
|
response_tx.send(notification).unwrap();
|
|
}
|
|
let mut received = 0usize;
|
|
let deadline = tokio::time::Instant::now() + Duration::from_secs(5);
|
|
loop {
|
|
let remaining = deadline - tokio::time::Instant::now();
|
|
if remaining.is_zero() {
|
|
break;
|
|
}
|
|
match tokio::time::timeout(remaining, read_message::<_, ServerMessage>(&mut reader))
|
|
.await
|
|
{
|
|
Ok(Ok(ServerMessage::Acp { .. })) => {
|
|
received += 1;
|
|
if received == REPLAY_COUNT {
|
|
break;
|
|
}
|
|
}
|
|
Ok(Ok(_)) => {}
|
|
Ok(Err(_)) | Err(_) => break,
|
|
}
|
|
}
|
|
assert_eq!(
|
|
received, REPLAY_COUNT,
|
|
"All {REPLAY_COUNT} replay notifications must arrive, got {received}"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// When a client disconnects after interacting with a session, the server
|
|
/// sends an `x.ai/internal/evict_sessions` notification through acp_tx
|
|
/// so the agent can release session memory.
|
|
#[tokio::test]
|
|
async fn evict_sessions_notification_on_disconnect() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("test.sock");
|
|
let (acp_tx, mut acp_rx) = mpsc::unbounded_channel();
|
|
let (_response_tx, response_rx) = mpsc::unbounded_channel();
|
|
let cancel = CancellationToken::new();
|
|
let control_state = default_test_control_state(&sock_path);
|
|
let sock_clone = sock_path.clone();
|
|
let cancel_clone = cancel.clone();
|
|
tokio::spawn(async move {
|
|
let _ = run_leader_server(
|
|
sock_clone,
|
|
acp_tx,
|
|
response_rx,
|
|
cancel_clone,
|
|
true,
|
|
Arc::new(AtomicUsize::new(0)),
|
|
Arc::new(AtomicBool::new(false)),
|
|
AgentActivity::default(),
|
|
watch::channel(true).1,
|
|
watch::channel(super::super::protocol::ShutdownReason::Manual).0,
|
|
None,
|
|
control_state,
|
|
)
|
|
.await;
|
|
});
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (mut _reader, mut writer) = connect_and_register(&sock_path, "test-client").await;
|
|
let msg = r#"{"jsonrpc":"2.0","method":"session/load","id":1,"params":{"sessionId":"sess-evict-test"}}"#;
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: msg.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let _ = acp_rx.recv().await;
|
|
write_message(&mut writer, &ClientMessage::Disconnect)
|
|
.await
|
|
.unwrap();
|
|
drop(_reader);
|
|
drop(writer);
|
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
|
let eviction_msg = tokio::time::timeout(Duration::from_secs(1), acp_rx.recv())
|
|
.await
|
|
.expect("should receive eviction notification")
|
|
.expect("channel should not be closed");
|
|
let json: serde_json::Value =
|
|
serde_json::from_str(&eviction_msg).expect("should be valid JSON");
|
|
assert_eq!(json["method"], "x.ai/internal/evict_sessions");
|
|
let session_ids = json["params"]["sessionIds"]
|
|
.as_array()
|
|
.expect("sessionIds should be an array");
|
|
assert!(
|
|
session_ids
|
|
.iter()
|
|
.any(|v| v.as_str() == Some("sess-evict-test")),
|
|
"eviction should include the session we interacted with, got: {session_ids:?}"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// When a client disconnects without interacting with any sessions,
|
|
/// no eviction notification should be sent.
|
|
#[tokio::test]
|
|
async fn no_eviction_when_client_has_no_sessions() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("test.sock");
|
|
let (acp_tx, mut acp_rx) = mpsc::unbounded_channel();
|
|
let (_response_tx, response_rx) = mpsc::unbounded_channel();
|
|
let cancel = CancellationToken::new();
|
|
let control_state = default_test_control_state(&sock_path);
|
|
let sock_clone = sock_path.clone();
|
|
let cancel_clone = cancel.clone();
|
|
tokio::spawn(async move {
|
|
let _ = run_leader_server(
|
|
sock_clone,
|
|
acp_tx,
|
|
response_rx,
|
|
cancel_clone,
|
|
true,
|
|
Arc::new(AtomicUsize::new(0)),
|
|
Arc::new(AtomicBool::new(false)),
|
|
AgentActivity::default(),
|
|
watch::channel(true).1,
|
|
watch::channel(super::super::protocol::ShutdownReason::Manual).0,
|
|
None,
|
|
control_state,
|
|
)
|
|
.await;
|
|
});
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (mut _reader, mut writer) = connect_and_register(&sock_path, "idle-client").await;
|
|
write_message(&mut writer, &ClientMessage::Disconnect)
|
|
.await
|
|
.unwrap();
|
|
drop(_reader);
|
|
drop(writer);
|
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
|
assert!(
|
|
acp_rx.try_recv().is_err(),
|
|
"no eviction notification should be sent for clients with no sessions"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// Read the next `ServerMessage::Acp` payload for a client, ignoring other
|
|
/// server messages, with a short deadline. Returns `None` on timeout.
|
|
async fn next_acp_payload(reader: &mut tokio::io::ReadHalf<LeaderStream>) -> Option<String> {
|
|
let deadline = tokio::time::Instant::now() + Duration::from_millis(800);
|
|
loop {
|
|
let remaining = deadline - tokio::time::Instant::now();
|
|
if remaining.is_zero() {
|
|
return None;
|
|
}
|
|
match tokio::time::timeout(remaining, read_message::<_, ServerMessage>(reader)).await {
|
|
Ok(Ok(ServerMessage::Acp { payload })) => return Some(payload),
|
|
Ok(Ok(_)) => continue,
|
|
Ok(Err(_)) | Err(_) => return None,
|
|
}
|
|
}
|
|
}
|
|
/// Drain up to a few ACP payloads looking for one containing `needle`.
|
|
/// Returns it if found within the window, else `None` (so a "must NOT
|
|
/// receive" assertion can use `.is_none()`).
|
|
async fn next_acp_payload_matching(
|
|
reader: &mut tokio::io::ReadHalf<LeaderStream>,
|
|
needle: &str,
|
|
) -> Option<String> {
|
|
for _ in 0..8 {
|
|
match next_acp_payload(reader).await {
|
|
Some(p) if p.contains(needle) => return Some(p),
|
|
Some(_) => continue,
|
|
None => return None,
|
|
}
|
|
}
|
|
None
|
|
}
|
|
async fn load_session(writer: &mut tokio::io::WriteHalf<LeaderStream>, session_id: &str) {
|
|
let msg = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"session/load","id":1,"params":{{"sessionId":"{session_id}"}}}}"#
|
|
);
|
|
write_message(writer, &ClientMessage::Acp { payload: msg })
|
|
.await
|
|
.unwrap();
|
|
}
|
|
/// Regression (live-before-replay race): a live `session/notification` that
|
|
/// arrives WHILE a viewer's `session/load` is in flight must be BUFFERED —
|
|
/// not delivered early (which would bump the client's eventId highwater and
|
|
/// make the subsequent lower-eventId replay get deduped away) — and then
|
|
/// flushed, in order, AFTER the load response.
|
|
#[tokio::test]
|
|
async fn live_broadcast_during_load_is_buffered_then_flushed_after_response() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader, mut writer) = connect_and_register(&sock_path, "viewer").await;
|
|
load_session(&mut writer, "sess-buf").await;
|
|
let forwarded = tokio::time::timeout(Duration::from_secs(1), acp_rx.recv())
|
|
.await
|
|
.expect("timed out waiting for forwarded load")
|
|
.expect("agent channel closed");
|
|
let load_id = serde_json::from_str::<serde_json::Value>(&forwarded)
|
|
.unwrap()
|
|
.get("id")
|
|
.cloned()
|
|
.unwrap();
|
|
let live = r#"{"jsonrpc":"2.0","method":"session/notification","params":{"sessionId":"sess-buf","updates":[{"type":"message_start","message_id":"live1"}]}}"#;
|
|
response_tx.send(live.to_string()).unwrap();
|
|
let early = tokio::time::timeout(
|
|
Duration::from_millis(250),
|
|
read_message::<_, ServerMessage>(&mut reader),
|
|
)
|
|
.await;
|
|
assert!(
|
|
early.is_err(),
|
|
"live broadcast must be buffered until the load response, got {early:?}"
|
|
);
|
|
let response = serde_json::json!(
|
|
{ "jsonrpc" : "2.0", "id" : load_id, "result" : { "models" : [] }, }
|
|
);
|
|
response_tx.send(response.to_string()).unwrap();
|
|
let first = next_acp_payload(&mut reader).await;
|
|
assert!(
|
|
first.as_deref().is_some_and(|p| p.contains("\"models\"")),
|
|
"first message after load must be the load response, got {first:?}"
|
|
);
|
|
let second = next_acp_payload(&mut reader).await;
|
|
assert!(
|
|
second.as_deref().is_some_and(|p| p.contains("live1")),
|
|
"buffered live notif must arrive (in order) after the load response, got {second:?}"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// Two clients load the same session; a `session/notification` (no `id`)
|
|
/// must reach BOTH (broadcast), while a reverse-request (`id` + `method`)
|
|
/// reaches ONLY the driver. The second client's `session/load` must not
|
|
/// black out the first (join-not-steal).
|
|
#[tokio::test]
|
|
async fn two_clients_one_session_broadcast_and_driver() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-multi").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-multi").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_b).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let notif = r#"{"jsonrpc":"2.0","method":"session/notification","params":{"sessionId":"sess-multi","updates":[{"type":"message_start","message_id":"m1"}]}}"#;
|
|
response_tx.send(notif.to_string()).unwrap();
|
|
let got_a = next_acp_payload(&mut reader_a).await;
|
|
let got_b = next_acp_payload(&mut reader_b).await;
|
|
assert!(
|
|
got_a.as_deref().is_some_and(|p| p.contains("m1")),
|
|
"client A must receive the broadcast notification, got {got_a:?}"
|
|
);
|
|
assert!(
|
|
got_b.as_deref().is_some_and(|p| p.contains("m1")),
|
|
"client B must receive the broadcast notification (no blackout), got {got_b:?}"
|
|
);
|
|
let req = r#"{"jsonrpc":"2.0","id":42,"method":"fs/read_text_file","params":{"sessionId":"sess-multi","path":"/tmp/x"}}"#;
|
|
response_tx.send(req.to_string()).unwrap();
|
|
let req_a = next_acp_payload(&mut reader_a).await;
|
|
let req_b = next_acp_payload(&mut reader_b).await;
|
|
assert!(
|
|
req_a
|
|
.as_deref()
|
|
.is_some_and(|p| p.contains("read_text_file")),
|
|
"driver A must receive the reverse-request, got {req_a:?}"
|
|
);
|
|
assert!(
|
|
req_b.is_none(),
|
|
"non-driver B must NOT receive the reverse-request, got {req_b:?}"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A `x.ai/scheduled_task_inject_prompt` (cron `/loop` fire) must be routed
|
|
/// to the SINGLE session driver, not fanned out to every subscriber. If it
|
|
/// broadcast, each attached dashboard would enqueue + try to drive the same
|
|
/// cron turn (phantom `#N` queue rows, competing drivers, stuck turns). The
|
|
/// other clients render the resulting turn from the broadcast deltas.
|
|
#[tokio::test]
|
|
async fn scheduled_task_inject_prompt_routes_to_driver_only() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-cron").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-cron").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_b).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let inject = r#"{"method":"_x.ai/scheduled_task_inject_prompt","params":{"method":"x.ai/scheduled_task_inject_prompt","params":{"sessionId":"sess-cron","taskId":"task-1","prompt":"echo hello","humanSchedule":"every 1m"}}}"#;
|
|
response_tx.send(inject.to_string()).unwrap();
|
|
let got_a = next_acp_payload(&mut reader_a).await;
|
|
let got_b = next_acp_payload(&mut reader_b).await;
|
|
assert!(
|
|
got_a
|
|
.as_deref()
|
|
.is_some_and(|p| p.contains("scheduled_task_inject_prompt")),
|
|
"driver A must receive the cron inject_prompt, got {got_a:?}"
|
|
);
|
|
assert!(
|
|
got_b.is_none(),
|
|
"non-driver B must NOT receive the cron inject_prompt, got {got_b:?}"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A blocking interaction reverse-request (permission / `ask_user_question` /
|
|
/// plan-approval) is SHARED: broadcast to every subscriber so any client can
|
|
/// render + answer the modal. Contrast
|
|
/// with `two_clients_one_session_broadcast_and_driver`, where an ordinary
|
|
/// reverse-request reaches the driver only.
|
|
#[tokio::test]
|
|
async fn interaction_request_broadcasts_to_all_subscribers() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-int").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-int").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_b).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let req = r#"{"jsonrpc":"2.0","id":501,"method":"_x.ai/ask_user_question","params":{"method":"x.ai/ask_user_question","params":{"sessionId":"sess-int","toolCallId":"tc-q","questions":[]}}}"#;
|
|
response_tx.send(req.to_string()).unwrap();
|
|
let got_a = next_acp_payload_matching(&mut reader_a, "ask_user_question").await;
|
|
let got_b = next_acp_payload_matching(&mut reader_b, "ask_user_question").await;
|
|
assert!(
|
|
got_a.is_some(),
|
|
"driver A must receive the shared interaction"
|
|
);
|
|
assert!(
|
|
got_b.is_some(),
|
|
"subscriber B must ALSO receive the shared interaction (not driver-only)"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A client that attaches WHILE an interaction is pending must render it too:
|
|
/// the leader caches the issued interaction and replays it to the new
|
|
/// subscriber after its `session/load` completes.
|
|
#[tokio::test]
|
|
async fn pending_interaction_replayed_to_late_joiner() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-int").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let req = r#"{"jsonrpc":"2.0","id":601,"method":"_x.ai/ask_user_question","params":{"method":"x.ai/ask_user_question","params":{"sessionId":"sess-int","toolCallId":"tc-late","questions":[]}}}"#;
|
|
response_tx.send(req.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "ask_user_question").await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-int").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let replayed = next_acp_payload_matching(&mut reader_b, "ask_user_question").await;
|
|
assert!(
|
|
replayed.is_some(),
|
|
"a late-joiner must receive the replayed still-pending interaction"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// Like [`connect_and_register`] but also returns the server-assigned
|
|
/// `ClientId` (needed to address targeted replay payloads at the client).
|
|
async fn connect_register_get_id(
|
|
sock_path: &std::path::Path,
|
|
client_type: &str,
|
|
) -> (
|
|
tokio::io::ReadHalf<LeaderStream>,
|
|
tokio::io::WriteHalf<LeaderStream>,
|
|
ClientId,
|
|
) {
|
|
let stream = LeaderStream::connect(sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: client_type.into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let msg: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
let ServerMessage::Registered { client_id, .. } = msg else {
|
|
panic!("expected Registered, got {msg:?}");
|
|
};
|
|
(reader, writer, ClientId(client_id))
|
|
}
|
|
/// A client that reattaches AFTER a subagent spawned is backfilled into
|
|
/// the child route when its parent `session/load` response lands: the
|
|
/// parent→child index survives the disconnect eviction (which only
|
|
/// empties subscriber sets), so live child updates resume without any
|
|
/// replayed spawn line. Driver inheritance is pinned too: a driver-only
|
|
/// child reverse-request must reach the reattached client.
|
|
#[tokio::test]
|
|
async fn reattached_client_backfilled_into_child_routes() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-sub").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-sub","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-sub"}}}"#;
|
|
response_tx.send(spawned_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a, "subagent_spawned")
|
|
.await
|
|
.is_some(),
|
|
"sanity: A receives the live spawn"
|
|
);
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (mut reader_a2, mut writer_a2) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a2, "sess-sub").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a2).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-sub","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"CHILD_LIVE_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a2, "CHILD_LIVE_DELTA")
|
|
.await
|
|
.is_some(),
|
|
"live child updates must reach the reattached client via backfill"
|
|
);
|
|
let child_reverse = r#"{"jsonrpc":"2.0","id":777,"method":"x.ai/child_thing","params":{"sessionId":"child-sub"}}"#;
|
|
response_tx.send(child_reverse.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a2, "child_thing")
|
|
.await
|
|
.is_some(),
|
|
"child reverse-requests must reach the backfilled driver"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A loading client receives the child route from the targeted REPLAYED
|
|
/// `subagent_spawned` alone (fresh-leader relaunch: no live spawn ever
|
|
/// crossed this server instance, the index is empty, only replay lines
|
|
/// describe the subagent).
|
|
#[tokio::test]
|
|
async fn replayed_spawn_registers_child_route_for_loading_client() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a, a_id) =
|
|
connect_register_get_id(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-fresh").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_replay = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{{"sessionId":"sess-fresh","_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}},"update":{{"sessionUpdate":"subagent_spawned","child_session_id":"child-fresh"}}}}}}"#,
|
|
a_id.0
|
|
);
|
|
response_tx.send(spawned_replay).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a, "subagent_spawned")
|
|
.await
|
|
.is_some(),
|
|
"the replayed spawn row reaches the loading client"
|
|
);
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-fresh","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"CHILD_FRESH_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a, "CHILD_FRESH_DELTA")
|
|
.await
|
|
.is_some(),
|
|
"the replayed spawn must register the live child route"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A client that attaches to the parent while another client already holds
|
|
/// a live child route is backfilled into that route (child sets are
|
|
/// spawn-time snapshots; joining the parent must join its descendants).
|
|
#[tokio::test]
|
|
async fn late_attacher_backfilled_into_existing_child_routes() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-sub2").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-sub2","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-sub2"}}}"#;
|
|
response_tx.send(spawned_live.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-sub2").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_b).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-sub2","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"CHILD_LIVE_DELTA2"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a, "CHILD_LIVE_DELTA2")
|
|
.await
|
|
.is_some(),
|
|
"A (in the spawn-time snapshot) still receives child updates"
|
|
);
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_b, "CHILD_LIVE_DELTA2")
|
|
.await
|
|
.is_some(),
|
|
"the late attacher must be backfilled into the child route"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A replayed `subagent_finished` unsubscribes ONLY its target client:
|
|
/// another client's live child route must survive one client's history
|
|
/// replay (full teardown is reserved for the LIVE finish).
|
|
#[tokio::test]
|
|
async fn replayed_finished_does_not_tear_down_live_child_route() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-tear").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-tear","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-tear"}}}"#;
|
|
response_tx.send(spawned_live.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
let (mut reader_b, mut writer_b, b_id) =
|
|
connect_register_get_id(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-tear").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_b).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let finished_replay = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{{"sessionId":"sess-tear","_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}},"update":{{"sessionUpdate":"subagent_finished","child_session_id":"child-tear"}}}}}}"#,
|
|
b_id.0
|
|
);
|
|
response_tx.send(finished_replay).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_b, "subagent_finished").await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-tear","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"CHILD_TEAR_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a, "CHILD_TEAR_DELTA")
|
|
.await
|
|
.is_some(),
|
|
"A's live child route must survive B's replayed finished"
|
|
);
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_b, "CHILD_TEAR_DELTA")
|
|
.await
|
|
.is_none(),
|
|
"B was unsubscribed by ITS replayed finished"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// Backfill walks the index depth-first: a nested child (spawned under a
|
|
/// CHILD session) is also joined when a client attaches to the root
|
|
/// parent.
|
|
#[tokio::test]
|
|
async fn backfill_covers_nested_children() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-nest").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_child = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-nest","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-nest"}}}"#;
|
|
response_tx.send(spawned_child.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
let spawned_grandchild = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"child-nest","update":{"sessionUpdate":"subagent_spawned","child_session_id":"grandchild-nest"}}}"#;
|
|
response_tx.send(spawned_grandchild.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "grandchild-nest").await;
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (mut reader_a2, mut writer_a2) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a2, "sess-nest").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a2).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let grandchild_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"grandchild-nest","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"GRANDCHILD_DELTA"}}}}"#;
|
|
response_tx.send(grandchild_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a2, "GRANDCHILD_DELTA")
|
|
.await
|
|
.is_some(),
|
|
"backfill must subscribe the client to nested descendants"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// Re-parenting on an INTERMEDIATE finish: root → A → B (both live). A
|
|
/// finishes LIVE while B keeps running. A new client loading the ROOT must
|
|
/// still be backfilled into B's live route — `prune_child_route` promotes B
|
|
/// onto A's parent so the forward-only root walk reaches it. Without
|
|
/// re-parenting the root→A edge is gone and B's subtree is orphaned.
|
|
#[tokio::test]
|
|
async fn intermediate_finish_reparents_live_grandchild_for_root_backfill() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-rep").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_a = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-rep","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-a"}}}"#;
|
|
response_tx.send(spawned_a.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "child-a").await;
|
|
let spawned_b = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"child-a","update":{"sessionUpdate":"subagent_spawned","child_session_id":"grandchild-b"}}}"#;
|
|
response_tx.send(spawned_b.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "grandchild-b").await;
|
|
let finished_a = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-rep","update":{"sessionUpdate":"subagent_finished","child_session_id":"child-a"}}}"#;
|
|
response_tx.send(finished_a.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_finished").await;
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (mut reader_a2, mut writer_a2) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a2, "sess-rep").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a2).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let grandchild_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"grandchild-b","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"LIVE_GRANDCHILD_AFTER_A_FINISH"}}}}"#;
|
|
response_tx.send(grandchild_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a2, "LIVE_GRANDCHILD_AFTER_A_FINISH")
|
|
.await
|
|
.is_some(),
|
|
"an intermediate finish must re-parent the live grandchild so root backfill still reaches it"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// The LIVE `subagent_finished` still tears the route down globally and
|
|
/// prunes the index: after it, a reattaching client is NOT backfilled
|
|
/// into the dead child (no leaked routes for finished subagents).
|
|
#[tokio::test]
|
|
async fn live_finished_prunes_index_so_reattach_skips_dead_child() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-dead").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-dead","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-dead"}}}"#;
|
|
response_tx.send(spawned_live.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
let finished_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-dead","update":{"sessionUpdate":"subagent_finished","child_session_id":"child-dead"}}}"#;
|
|
response_tx.send(finished_live.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_finished").await;
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (mut reader_a2, mut writer_a2) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a2, "sess-dead").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a2).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-dead","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"DEAD_CHILD_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a2, "DEAD_CHILD_DELTA")
|
|
.await
|
|
.is_none(),
|
|
"a finished child's route must not be resurrected by reattach"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// Symmetric twin of `live_finished_prunes_index_so_reattach_skips_dead_child`
|
|
/// for the no-subscribers case: the parent goes fully detached (every
|
|
/// client disconnects, the index edge survives), THEN a live
|
|
/// `subagent_finished` arrives. It is relay-classified (no subscribers) and
|
|
/// dropped — but it must still prune the index edge, so a reattaching
|
|
/// client's `session/load` backfill does not resurrect the dead child.
|
|
#[tokio::test]
|
|
async fn detached_live_finished_prunes_index_so_reattach_skips_dead_child() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-detach").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-detach","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-detach"}}}"#;
|
|
response_tx.send(spawned_live.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let finished_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-detach","update":{"sessionUpdate":"subagent_finished","child_session_id":"child-detach"}}}"#;
|
|
response_tx.send(finished_live.to_string()).unwrap();
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let (mut reader_a2, mut writer_a2) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a2, "sess-detach").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a2).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-detach","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"DETACHED_DEAD_CHILD_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a2, "DETACHED_DEAD_CHILD_DELTA")
|
|
.await
|
|
.is_none(),
|
|
"a detached live finish must prune the edge — the dead child's route \
|
|
must not be resurrected by reattach backfill"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A loader disconnecting between a dead child's replayed spawn and
|
|
/// replayed finish must not leak the index edge: the orphan-drop arm still
|
|
/// prunes when nothing holds the route, so a later attacher is not
|
|
/// backfilled into the dead child.
|
|
#[tokio::test]
|
|
async fn mid_burst_disconnect_still_prunes_dead_child_route() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a, a_id) =
|
|
connect_register_get_id(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-leak").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_replay = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{{"sessionId":"sess-leak","_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}},"update":{{"sessionUpdate":"subagent_spawned","child_session_id":"child-leak"}}}}}}"#,
|
|
a_id.0
|
|
);
|
|
response_tx.send(spawned_replay).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let finished_replay = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{{"sessionId":"sess-leak","_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}},"update":{{"sessionUpdate":"subagent_finished","child_session_id":"child-leak"}}}}}}"#,
|
|
a_id.0
|
|
);
|
|
response_tx.send(finished_replay).unwrap();
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let (mut reader_a2, mut writer_a2) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a2, "sess-leak").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a2).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-leak","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"LEAKED_CHILD_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a2, "LEAKED_CHILD_DELTA")
|
|
.await
|
|
.is_none(),
|
|
"an orphaned replayed finish must prune the edge — the dead child's \
|
|
route must not be resurrected by reattach backfill"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// An ORPHANED replayed finish (its target already vanished) must leave a
|
|
/// route other clients hold untouched — the orphan branch prunes only
|
|
/// when nothing holds the route. An always-prune mutation of that guard
|
|
/// would let one dead client's stale replay burst tear down A's live
|
|
/// route.
|
|
#[tokio::test]
|
|
async fn orphaned_replayed_finished_leaves_held_route_untouched() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-hold").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-hold","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-hold"}}}"#;
|
|
response_tx.send(spawned_live.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
let (reader_b, writer_b, b_id) = connect_register_get_id(&sock_path, "client-b").await;
|
|
drop(reader_b);
|
|
drop(writer_b);
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let finished_replay = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{{"sessionId":"sess-hold","_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}},"update":{{"sessionUpdate":"subagent_finished","child_session_id":"child-hold"}}}}}}"#,
|
|
b_id.0
|
|
);
|
|
response_tx.send(finished_replay).unwrap();
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-hold","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"HELD_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a, "HELD_DELTA")
|
|
.await
|
|
.is_some(),
|
|
"an orphaned replayed finish must not prune a route A still holds"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A replayed spawn UNIONS the loading client into an existing live route:
|
|
/// a regression to the live arm's snapshot-replace would tear down the
|
|
/// holder's route on someone else's history replay (the symmetric twin of
|
|
/// `replayed_finished_does_not_tear_down_live_child_route`).
|
|
#[tokio::test]
|
|
async fn replayed_spawn_unions_into_existing_live_route() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-union").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-union","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-union"}}}"#;
|
|
response_tx.send(spawned_live.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
let (mut reader_b, mut writer_b, b_id) =
|
|
connect_register_get_id(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-union").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_b).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_replay = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{{"sessionId":"sess-union","_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}},"update":{{"sessionUpdate":"subagent_spawned","child_session_id":"child-union"}}}}}}"#,
|
|
b_id.0
|
|
);
|
|
response_tx.send(spawned_replay).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_b, "subagent_spawned").await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-union","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"UNION_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a, "UNION_DELTA")
|
|
.await
|
|
.is_some(),
|
|
"A's live route must survive B's replayed spawn (union, not replace)"
|
|
);
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_b, "UNION_DELTA")
|
|
.await
|
|
.is_some(),
|
|
"B is in the route too"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A replayed finish that removes the LAST subscriber prunes the route,
|
|
/// driver, and index edge — a later attacher must not be backfilled into
|
|
/// a child whose finish was only ever observed via replay.
|
|
#[tokio::test]
|
|
async fn replayed_finished_last_subscriber_prunes_dead_child() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a, a_id) =
|
|
connect_register_get_id(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-last").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_replay = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{{"sessionId":"sess-last","_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}},"update":{{"sessionUpdate":"subagent_spawned","child_session_id":"child-last"}}}}}}"#,
|
|
a_id.0
|
|
);
|
|
response_tx.send(spawned_replay).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
let finished_replay = format!(
|
|
r#"{{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{{"sessionId":"sess-last","_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}},"update":{{"sessionUpdate":"subagent_finished","child_session_id":"child-last"}}}}}}"#,
|
|
a_id.0
|
|
);
|
|
response_tx.send(finished_replay).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_finished").await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-last").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_b).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-last","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"LAST_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_b, "LAST_DELTA")
|
|
.await
|
|
.is_none(),
|
|
"the last-subscriber replayed finish must prune the edge"
|
|
);
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_a, "LAST_DELTA")
|
|
.await
|
|
.is_none(),
|
|
"A was unsubscribed by its own replayed finish"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// Isolates the REQUEST-side backfill call site: it subscribes the loader
|
|
/// to live children the moment the `session/load` request passes through,
|
|
/// so a child delta arriving MID-LOAD (post-request, pre-response) is
|
|
/// delivered instead of dropped as subscriber-less. With only the
|
|
/// response-side site the delta would be lost before the response lands.
|
|
#[tokio::test]
|
|
async fn mid_load_child_delta_reaches_loader_via_request_side_backfill() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-midload").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let spawned_live = r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-midload","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-midload"}}}"#;
|
|
response_tx.send(spawned_live.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "subagent_spawned").await;
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-midload").await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let child_live = r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"child-midload","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"MIDLOAD_DELTA"}}}}"#;
|
|
response_tx.send(child_live.to_string()).unwrap();
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
assert!(
|
|
next_acp_payload_matching(&mut reader_b, "MIDLOAD_DELTA")
|
|
.await
|
|
.is_some(),
|
|
"a mid-load child delta must reach the loader (request-side backfill)"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// A pending interaction must SURVIVE a full client disconnect and be
|
|
/// replayed on reconnect. A session with a pending interaction has a running
|
|
/// turn (the tool awaits the answer), so the agent keeps it resident across
|
|
/// the disconnect with the reverse-request still parked
|
|
/// (`session_has_live_work`). The leader must therefore NOT drop its
|
|
/// interaction cache on detach — otherwise the reconnecting client gets no
|
|
/// modal while the agent is still waiting. Regression for the "modal vanishes
|
|
/// on reconnect" bug.
|
|
#[tokio::test]
|
|
async fn pending_interaction_survives_disconnect_and_replays_on_reconnect() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-int").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let req = r#"{"jsonrpc":"2.0","id":801,"method":"_x.ai/ask_user_question","params":{"method":"x.ai/ask_user_question","params":{"sessionId":"sess-int","toolCallId":"tc-reconnect","questions":[]}}}"#;
|
|
response_tx.send(req.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "ask_user_question").await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(80)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-int").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let replayed = next_acp_payload_matching(&mut reader_b, "ask_user_question").await;
|
|
assert!(
|
|
replayed.is_some(),
|
|
"a still-pending interaction must survive a full disconnect and replay on reconnect"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// An interaction raised while the session has NO subscriber (a session
|
|
/// started from the dashboard whose turn hit `ask_user_question` before
|
|
/// anyone entered it, or a reverse-request that races ahead of the
|
|
/// `session/new`/`session/load` response that registers the subscriber) must
|
|
/// still be cached, so the FIRST client to attach gets the modal replayed.
|
|
/// Regression for the "entered the session, modal never appears, turn stuck
|
|
/// Waiting" bug — the cache insert used to be gated on an existing subscriber.
|
|
#[tokio::test]
|
|
async fn interaction_raised_with_no_subscriber_is_cached_and_replayed_on_first_attach() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let req = r#"{"jsonrpc":"2.0","id":901,"method":"_x.ai/ask_user_question","params":{"method":"x.ai/ask_user_question","params":{"sessionId":"sess-int","toolCallId":"tc-nosub","questions":[]}}}"#;
|
|
response_tx.send(req.to_string()).unwrap();
|
|
tokio::time::sleep(Duration::from_millis(40)).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-int").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let replayed = next_acp_payload_matching(&mut reader_a, "ask_user_question").await;
|
|
assert!(
|
|
replayed.is_some(),
|
|
"an interaction raised with no subscriber must be cached and replayed to the first client that attaches"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// Once an interaction resolves (first-answer-wins → `InteractionResolved`),
|
|
/// the leader evicts it from the replay cache, so a client that attaches
|
|
/// afterwards does NOT get a stale modal.
|
|
#[tokio::test]
|
|
async fn resolved_interaction_not_replayed_to_late_joiner() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx, mut acp_rx) =
|
|
setup_persistent_server_with_agent(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-int").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let _ = next_acp_payload(&mut reader_a).await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let req = r#"{"jsonrpc":"2.0","id":701,"method":"_x.ai/ask_user_question","params":{"method":"x.ai/ask_user_question","params":{"sessionId":"sess-int","toolCallId":"tc-ev","questions":[]}}}"#;
|
|
response_tx.send(req.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "ask_user_question").await;
|
|
let resolved = r#"{"method":"_x.ai/session_notification","params":{"method":"x.ai/session_notification","params":{"sessionId":"sess-int","update":{"sessionUpdate":"interaction_resolved","tool_call_id":"tc-ev"}}}}"#;
|
|
response_tx.send(resolved.to_string()).unwrap();
|
|
let _ = next_acp_payload_matching(&mut reader_a, "interaction_resolved").await;
|
|
tokio::time::sleep(Duration::from_millis(30)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-int").await;
|
|
complete_load(&mut acp_rx, &response_tx).await;
|
|
let replayed = next_acp_payload_matching(&mut reader_b, "ask_user_question").await;
|
|
assert!(
|
|
replayed.is_none(),
|
|
"a resolved interaction must NOT be replayed to a late-joiner (evicted)"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// When the driver disconnects but another subscriber remains, the session
|
|
/// is NOT evicted and the driver role transfers to the remaining client.
|
|
#[tokio::test]
|
|
async fn driver_disconnect_transfers_not_evicts() {
|
|
let temp = TempDir::new().unwrap();
|
|
let sock_path = temp.path().join("test.sock");
|
|
let (acp_tx, mut acp_rx) = mpsc::unbounded_channel();
|
|
let (response_tx, response_rx) = mpsc::unbounded_channel();
|
|
let cancel = CancellationToken::new();
|
|
let control_state = default_test_control_state(&sock_path);
|
|
let sock_clone = sock_path.clone();
|
|
let cancel_clone = cancel.clone();
|
|
tokio::spawn(async move {
|
|
let _ = run_leader_server(
|
|
sock_clone,
|
|
acp_tx,
|
|
response_rx,
|
|
cancel_clone,
|
|
true,
|
|
Arc::new(AtomicUsize::new(0)),
|
|
Arc::new(AtomicBool::new(false)),
|
|
AgentActivity::default(),
|
|
watch::channel(true).1,
|
|
watch::channel(super::super::protocol::ShutdownReason::Manual).0,
|
|
None,
|
|
control_state,
|
|
)
|
|
.await;
|
|
});
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
let (reader_a, mut writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
load_session(&mut writer_a, "sess-xfer").await;
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
load_session(&mut writer_b, "sess-xfer").await;
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
while acp_rx.try_recv().is_ok() {}
|
|
write_message(&mut writer_a, &ClientMessage::Disconnect)
|
|
.await
|
|
.unwrap();
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(80)).await;
|
|
assert!(
|
|
acp_rx.try_recv().is_err(),
|
|
"session must NOT be evicted while another subscriber remains"
|
|
);
|
|
let req = r#"{"jsonrpc":"2.0","id":7,"method":"fs/read_text_file","params":{"sessionId":"sess-xfer","path":"/tmp/x"}}"#;
|
|
response_tx.send(req.to_string()).unwrap();
|
|
let req_b = next_acp_payload(&mut reader_b).await;
|
|
assert!(
|
|
req_b
|
|
.as_deref()
|
|
.is_some_and(|p| p.contains("read_text_file")),
|
|
"after driver disconnect, B should become driver and receive the reverse-request, got {req_b:?}"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// `x.ai/sessions/changed` is a machine-wide roster notification with no
|
|
/// sessionId; it must broadcast to every registered client (not just the
|
|
/// last-active one) so all open dashboards stay in sync.
|
|
#[tokio::test]
|
|
async fn roster_changed_broadcasts_to_all_clients() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let (mut reader_a, _writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
let (mut reader_b, _writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
let changed = r#"{"jsonrpc":"2.0","method":"x.ai/sessions/changed","params":{"upserted":[{"sessionId":"sess-roster","cwd":"/repo","isWorktree":false,"yolo":false,"activity":"working","resident":true,"lastChangeUnixMs":1,"origin":{"kind":"local"}}],"removed":[]}}"#;
|
|
response_tx.send(changed.to_string()).unwrap();
|
|
let got_a = next_acp_payload(&mut reader_a).await;
|
|
let got_b = next_acp_payload(&mut reader_b).await;
|
|
assert!(
|
|
got_a.as_deref().is_some_and(|p| p.contains("sess-roster")),
|
|
"client A must receive the roster broadcast, got {got_a:?}"
|
|
);
|
|
assert!(
|
|
got_b.as_deref().is_some_and(|p| p.contains("sess-roster")),
|
|
"client B must receive the roster broadcast, got {got_b:?}"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// `x.ai/models/update` is a machine-wide catalog notification with no
|
|
/// sessionId; it must broadcast to every registered client so every model
|
|
/// picker refreshes after a config.toml / models_cache.json hot-reload —
|
|
/// not just the last-active client. Uses the production wire form: agent
|
|
/// ext notifications arrive `_`-prefixed (`_x.ai/models/update`).
|
|
#[tokio::test]
|
|
async fn models_update_broadcasts_to_all_clients() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let (mut reader_a, _writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
let (mut reader_b, _writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
let update = r#"{"jsonrpc":"2.0","method":"_x.ai/models/update","params":{"currentModelId":"grok-new","availableModels":[{"modelId":"grok-new","name":"Grok New"}]}}"#;
|
|
response_tx.send(update.to_string()).unwrap();
|
|
let got_a = next_acp_payload(&mut reader_a).await;
|
|
let got_b = next_acp_payload(&mut reader_b).await;
|
|
assert!(
|
|
got_a.as_deref().is_some_and(|p| p.contains("grok-new")),
|
|
"client A must receive the models broadcast, got {got_a:?}"
|
|
);
|
|
assert!(
|
|
got_b.as_deref().is_some_and(|p| p.contains("grok-new")),
|
|
"client B must receive the models broadcast, got {got_b:?}"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// `x.ai/mcp/servers_updated` is a machine-wide MCP-catalog notification
|
|
/// with no sessionId (session-agnostic by design); it must broadcast to
|
|
/// every registered client so managed connectors don't vanish from clients
|
|
/// that weren't last-active when the post-initialize background fetch
|
|
/// resolved. Uses the production wire form (`_`-prefixed ext notification
|
|
/// with the real method nested in params).
|
|
#[tokio::test]
|
|
async fn mcp_servers_updated_broadcasts_to_all_clients() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let (mut reader_a, _writer_a) = connect_and_register(&sock_path, "client-a").await;
|
|
let (mut reader_b, _writer_b) = connect_and_register(&sock_path, "client-b").await;
|
|
tokio::time::sleep(Duration::from_millis(20)).await;
|
|
let update = r#"{"jsonrpc":"2.0","method":"_x.ai/mcp/servers_updated","params":{"method":"x.ai/mcp/servers_updated","params":{"mcpServers":[{"name":"grok_com_slack","source":"managed"}]}}}"#;
|
|
response_tx.send(update.to_string()).unwrap();
|
|
let got_a = next_acp_payload(&mut reader_a).await;
|
|
let got_b = next_acp_payload(&mut reader_b).await;
|
|
assert!(
|
|
got_a
|
|
.as_deref()
|
|
.is_some_and(|p| p.contains("grok_com_slack")),
|
|
"client A must receive the MCP catalog broadcast, got {got_a:?}"
|
|
);
|
|
assert!(
|
|
got_b
|
|
.as_deref()
|
|
.is_some_and(|p| p.contains("grok_com_slack")),
|
|
"client B must receive the MCP catalog broadcast, got {got_b:?}"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
/// The broadcast classifier must accept both wire forms (`_`-prefixed
|
|
/// production ext notifications and direct methods) for the machine-wide
|
|
/// set, and reject sessionful / unrelated methods.
|
|
#[test]
|
|
fn machine_wide_broadcast_classifier_matches_both_wire_forms() {
|
|
assert!(is_machine_wide_broadcast_notification(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/sessions/changed","params":{}}"#
|
|
)));
|
|
assert!(is_machine_wide_broadcast_notification(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/models/update","params":{}}"#
|
|
)));
|
|
assert!(is_machine_wide_broadcast_notification(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/mcp/servers_updated","params":{}}"#
|
|
)));
|
|
assert!(is_machine_wide_broadcast_notification(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"_x.ai/sessions/changed","params":{}}"#
|
|
)));
|
|
assert!(is_machine_wide_broadcast_notification(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"_x.ai/models/update","params":{}}"#
|
|
)));
|
|
assert!(is_machine_wide_broadcast_notification(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"_x.ai/mcp/servers_updated","params":{"method":"x.ai/mcp/servers_updated","params":{"mcpServers":[]}}}"#
|
|
)));
|
|
assert!(!is_machine_wide_broadcast_notification(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"s"}}"#
|
|
)));
|
|
assert!(!is_machine_wide_broadcast_notification(&pv(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/settings/update","params":{}}"#
|
|
)));
|
|
}
|
|
/// Verify that the leader injects `codeNavEnabled: true` into session/new
|
|
/// when the client registered with `code_nav_enabled: true`.
|
|
#[test]
|
|
fn inject_capabilities_sets_code_nav_enabled_true() {
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: false,
|
|
default_model: None,
|
|
client_version: None,
|
|
code_nav_enabled: true,
|
|
..Default::default()
|
|
};
|
|
let payload = r#"{"jsonrpc":"2.0","method":"session/new","id":1,"params":{"cwd":"/repo","_meta":{}}}"#;
|
|
let mut json = pv(payload);
|
|
inject_capabilities_into_session_new(&mut json, &caps, "grok-web", ClientId(1));
|
|
assert_eq!(
|
|
json["params"]["_meta"]["codeNavEnabled"],
|
|
serde_json::json!(true),
|
|
"leader must inject codeNavEnabled=true for code-nav-capable client"
|
|
);
|
|
}
|
|
/// Verify that the leader injects `codeNavEnabled: false` when the client
|
|
/// did NOT register with `code_nav_enabled` — preventing a prior eligible
|
|
/// client's shared state from bleeding into this client's sessions.
|
|
#[test]
|
|
fn inject_capabilities_sets_code_nav_enabled_false() {
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: false,
|
|
default_model: None,
|
|
client_version: None,
|
|
code_nav_enabled: false,
|
|
..Default::default()
|
|
};
|
|
let payload = r#"{"jsonrpc":"2.0","method":"session/new","id":1,"params":{"cwd":"/repo","_meta":{"clientIdentifier":"grok-tui"}}}"#;
|
|
let mut json = pv(payload);
|
|
inject_capabilities_into_session_new(&mut json, &caps, "grok-tui", ClientId(1));
|
|
assert_eq!(
|
|
json["params"]["_meta"]["codeNavEnabled"],
|
|
serde_json::json!(false),
|
|
"leader must inject codeNavEnabled=false for client without code-nav capability"
|
|
);
|
|
}
|
|
/// Verify that `codeNavEnabled` is also injected into `session/load` so
|
|
/// reconnect sessions inherit the correct per-client capability.
|
|
#[test]
|
|
fn inject_capabilities_injects_code_nav_into_session_load() {
|
|
let caps = ClientCapabilities {
|
|
yolo_mode: false,
|
|
default_model: None,
|
|
client_version: None,
|
|
code_nav_enabled: true,
|
|
..Default::default()
|
|
};
|
|
let payload = r#"{"jsonrpc":"2.0","method":"session/load","id":2,"params":{"sessionId":"abc","cwd":"/repo","_meta":{}}}"#;
|
|
let mut json = pv(payload);
|
|
inject_capabilities_into_session_new(&mut json, &caps, "grok-web", ClientId(1));
|
|
assert_eq!(
|
|
json["params"]["_meta"]["codeNavEnabled"],
|
|
serde_json::json!(true),
|
|
"leader must inject codeNavEnabled into session/load for reconnect isolation"
|
|
);
|
|
}
|
|
/// Verify leader-mode client isolation: two clients with different code-nav
|
|
/// capabilities get independent `codeNavEnabled` values injected into their
|
|
/// session/new requests.
|
|
#[test]
|
|
fn inject_capabilities_two_clients_stay_isolated() {
|
|
let web_caps = ClientCapabilities {
|
|
code_nav_enabled: true,
|
|
..Default::default()
|
|
};
|
|
let tui_caps = ClientCapabilities {
|
|
code_nav_enabled: false,
|
|
..Default::default()
|
|
};
|
|
let session_new = r#"{"jsonrpc":"2.0","method":"session/new","id":1,"params":{"cwd":"/repo","_meta":{}}}"#;
|
|
let mut web_json = pv(session_new);
|
|
inject_capabilities_into_session_new(&mut web_json, &web_caps, "grok-web", ClientId(1));
|
|
let mut tui_json = pv(session_new);
|
|
inject_capabilities_into_session_new(&mut tui_json, &tui_caps, "grok-tui", ClientId(2));
|
|
assert_eq!(
|
|
web_json["params"]["_meta"]["codeNavEnabled"],
|
|
serde_json::json!(true)
|
|
);
|
|
assert_eq!(
|
|
tui_json["params"]["_meta"]["codeNavEnabled"],
|
|
serde_json::json!(false)
|
|
);
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_terminal_and_fs_per_client() {
|
|
let web_caps = ClientCapabilities {
|
|
terminal: true,
|
|
fs_read: true,
|
|
fs_write: true,
|
|
..Default::default()
|
|
};
|
|
let tui_caps = ClientCapabilities {
|
|
terminal: false,
|
|
fs_read: false,
|
|
fs_write: false,
|
|
..Default::default()
|
|
};
|
|
let session_new = r#"{"jsonrpc":"2.0","method":"session/new","id":1,"params":{"cwd":"/repo","_meta":{}}}"#;
|
|
let mut web_json = pv(session_new);
|
|
inject_capabilities_into_session_new(&mut web_json, &web_caps, "grok-web", ClientId(1));
|
|
let mut tui_json = pv(session_new);
|
|
inject_capabilities_into_session_new(&mut tui_json, &tui_caps, "grok-tui", ClientId(2));
|
|
assert_eq!(
|
|
web_json["params"]["_meta"]["clientTerminal"],
|
|
serde_json::json!(true)
|
|
);
|
|
assert_eq!(
|
|
web_json["params"]["_meta"]["clientFsRead"],
|
|
serde_json::json!(true)
|
|
);
|
|
assert_eq!(
|
|
web_json["params"]["_meta"]["clientFsWrite"],
|
|
serde_json::json!(true)
|
|
);
|
|
assert_eq!(
|
|
tui_json["params"]["_meta"]["clientTerminal"],
|
|
serde_json::json!(false)
|
|
);
|
|
assert_eq!(
|
|
tui_json["params"]["_meta"]["clientFsRead"],
|
|
serde_json::json!(false)
|
|
);
|
|
assert_eq!(
|
|
tui_json["params"]["_meta"]["clientFsWrite"],
|
|
serde_json::json!(false)
|
|
);
|
|
}
|
|
#[test]
|
|
fn inject_capabilities_terminal_into_session_load() {
|
|
let caps = ClientCapabilities {
|
|
terminal: true,
|
|
fs_read: false,
|
|
fs_write: false,
|
|
..Default::default()
|
|
};
|
|
let session_load = r#"{"jsonrpc":"2.0","method":"session/load","id":2,"params":{"sessionId":"sess-1","_meta":{}}}"#;
|
|
let mut json = pv(session_load);
|
|
inject_capabilities_into_session_new(&mut json, &caps, "grok-web", ClientId(1));
|
|
assert_eq!(
|
|
json["params"]["_meta"]["clientTerminal"],
|
|
serde_json::json!(true)
|
|
);
|
|
assert_eq!(
|
|
json["params"]["_meta"]["clientFsRead"],
|
|
serde_json::json!(false)
|
|
);
|
|
assert_eq!(
|
|
json["params"]["_meta"]["clientFsWrite"],
|
|
serde_json::json!(false)
|
|
);
|
|
}
|
|
#[tokio::test]
|
|
async fn subagent_child_session_routed_after_spawned() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let (mut reader, mut writer) = connect_and_register(&sock_path, "test").await;
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"session/prompt","id":1,"params":{"sessionId":"sess-parent","prompt":[]}}"#
|
|
.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-parent","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-123"}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let _: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader))
|
|
.await
|
|
.unwrap()
|
|
.unwrap();
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"child-123","update":{"sessionUpdate":"message_delta","content":"hello"}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let msg: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader))
|
|
.await
|
|
.expect("child session notification should reach parent owner")
|
|
.unwrap();
|
|
match msg {
|
|
ServerMessage::Acp { payload } => {
|
|
let json: serde_json::Value = serde_json::from_str(&payload).unwrap();
|
|
assert_eq!(json["params"]["sessionId"], "child-123");
|
|
}
|
|
other => panic!("Expected Acp, got {:?}", other),
|
|
}
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn subagent_child_session_cleaned_up_on_finished() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let (mut reader, mut writer) = connect_and_register(&sock_path, "test").await;
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"session/prompt","id":1,"params":{"sessionId":"sess-parent","prompt":[]}}"#
|
|
.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-parent","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-456"}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let _: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader))
|
|
.await
|
|
.unwrap()
|
|
.unwrap();
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"sess-parent","update":{"sessionUpdate":"subagent_finished","child_session_id":"child-456"}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let _: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader))
|
|
.await
|
|
.unwrap()
|
|
.unwrap();
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"child-456","update":{"sessionUpdate":"message_delta"}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let timeout_result: Result<Result<ServerMessage, _>, _> =
|
|
tokio::time::timeout(Duration::from_millis(100), read_message(&mut reader)).await;
|
|
assert!(
|
|
timeout_result.is_err(),
|
|
"Notification for finished child session should not be routed"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn subagent_child_session_not_leaked_to_other_client() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let (mut reader_a, mut writer_a) = connect_and_register(&sock_path, "test-a").await;
|
|
write_message(
|
|
&mut writer_a,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"session/prompt","id":1,"params":{"sessionId":"sess-parent","prompt":[]}}"#
|
|
.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"_x.ai/session_notification","params":{"method":"x.ai/session_notification","params":{"sessionId":"sess-parent","update":{"sessionUpdate":"subagent_spawned","child_session_id":"child-789"}}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let _: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader_a))
|
|
.await
|
|
.unwrap()
|
|
.unwrap();
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "test-b").await;
|
|
write_message(
|
|
&mut writer_b,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"initialize","id":2,"params":{}}"#.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"x.ai/session_notification","params":{"sessionId":"child-789","update":{"sessionUpdate":"message_delta"}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let msg: ServerMessage =
|
|
tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader_a))
|
|
.await
|
|
.expect("Client A should receive child session notification")
|
|
.unwrap();
|
|
assert!(matches!(msg, ServerMessage::Acp { .. }));
|
|
let timeout_result: Result<Result<ServerMessage, _>, _> =
|
|
tokio::time::timeout(Duration::from_millis(100), read_message(&mut reader_b)).await;
|
|
assert!(
|
|
timeout_result.is_err(),
|
|
"Client B should NOT receive child session notification"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn leader_client_id_unicasts_to_target_only() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
async fn register_capture(
|
|
sock_path: &std::path::Path,
|
|
client_type: &str,
|
|
) -> (
|
|
tokio::io::ReadHalf<LeaderStream>,
|
|
tokio::io::WriteHalf<LeaderStream>,
|
|
u64,
|
|
) {
|
|
let stream = LeaderStream::connect(sock_path).await.unwrap();
|
|
let (mut reader, mut writer) = tokio::io::split(stream);
|
|
write_message(
|
|
&mut writer,
|
|
&ClientMessage::Register {
|
|
client_type: client_type.into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let msg: ServerMessage = read_message(&mut reader).await.unwrap();
|
|
let client_id = match msg {
|
|
ServerMessage::Registered { client_id, .. } => client_id,
|
|
other => panic!("Expected Registered, got {:?}", other),
|
|
};
|
|
(reader, writer, client_id)
|
|
}
|
|
let (mut reader_a, _writer_a, id_a) = register_capture(&sock_path, "test-a").await;
|
|
let (mut reader_b, _writer_b, _id_b) = register_capture(&sock_path, "test-b").await;
|
|
response_tx
|
|
.send(
|
|
format!(
|
|
r#"{{"jsonrpc":"2.0","method":"session/update","params":{{"sessionId":"sess-1","update":{{"sessionUpdate":"agent_message_chunk"}},"_meta":{{"x.ai/leaderClientId":{}}}}}}}"#,
|
|
id_a
|
|
),
|
|
)
|
|
.unwrap();
|
|
let msg = tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader_a))
|
|
.await
|
|
.expect("Client A should receive the tagged replay notification")
|
|
.unwrap();
|
|
assert!(matches!(msg, ServerMessage::Acp { .. }));
|
|
let timeout_result: Result<Result<ServerMessage, _>, _> =
|
|
tokio::time::timeout(Duration::from_millis(100), read_message(&mut reader_b)).await;
|
|
assert!(
|
|
timeout_result.is_err(),
|
|
"Client B must not receive a notification tagged for client A"
|
|
);
|
|
cancel.cancel();
|
|
}
|
|
#[tokio::test]
|
|
async fn leader_client_id_dropped_when_target_disconnected() {
|
|
let temp = TempDir::new().unwrap();
|
|
let (sock_path, cancel, response_tx) = setup_persistent_server(&temp).await;
|
|
let stream_a = LeaderStream::connect(&sock_path).await.unwrap();
|
|
let (mut reader_a, mut writer_a) = tokio::io::split(stream_a);
|
|
write_message(
|
|
&mut writer_a,
|
|
&ClientMessage::Register {
|
|
client_type: "test-a".into(),
|
|
mode: ClientMode::Stdio,
|
|
capabilities: ClientCapabilities::default(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
let id_a = match read_message(&mut reader_a).await.unwrap() {
|
|
ServerMessage::Registered { client_id, .. } => client_id,
|
|
other => panic!("Expected Registered, got {:?}", other),
|
|
};
|
|
let (mut reader_b, mut writer_b) = connect_and_register(&sock_path, "test-b").await;
|
|
write_message(
|
|
&mut writer_b,
|
|
&ClientMessage::Acp {
|
|
payload: r#"{"jsonrpc":"2.0","method":"session/prompt","id":1,"params":{"sessionId":"sess-1","prompt":[]}}"#
|
|
.into(),
|
|
},
|
|
)
|
|
.await
|
|
.unwrap();
|
|
tokio::time::sleep(Duration::from_millis(50)).await;
|
|
drop(reader_a);
|
|
drop(writer_a);
|
|
tokio::time::sleep(Duration::from_millis(100)).await;
|
|
response_tx
|
|
.send(
|
|
format!(
|
|
r#"{{"jsonrpc":"2.0","method":"session/update","params":{{"sessionId":"sess-1","update":{{"sessionUpdate":"agent_message_chunk"}},"_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}}}}}}"#,
|
|
id_a
|
|
),
|
|
)
|
|
.unwrap();
|
|
response_tx
|
|
.send(
|
|
format!(
|
|
r#"{{"jsonrpc":"2.0","method":"_x.ai/session/update","params":{{"params":{{"sessionId":"sess-1","update":{{"sessionUpdate":"hook_annotation","message":"m"}},"_meta":{{"isReplay":true,"x.ai/leaderClientId":{}}}}}}}}}"#,
|
|
id_a
|
|
),
|
|
)
|
|
.unwrap();
|
|
let timeout_result: Result<Result<ServerMessage, _>, _> =
|
|
tokio::time::timeout(Duration::from_millis(150), read_message(&mut reader_b)).await;
|
|
assert!(
|
|
timeout_result.is_err(),
|
|
"A targeted replay line for a disconnected loader must be dropped, not broadcast"
|
|
);
|
|
response_tx
|
|
.send(
|
|
r#"{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"sess-1","update":{"sessionUpdate":"agent_message_chunk"}}}"#
|
|
.into(),
|
|
)
|
|
.unwrap();
|
|
let msg = tokio::time::timeout(Duration::from_millis(200), read_message(&mut reader_b))
|
|
.await
|
|
.expect("Subscriber B should still receive untagged live notifications")
|
|
.unwrap();
|
|
assert!(matches!(msg, ServerMessage::Acp { .. }));
|
|
cancel.cancel();
|
|
}
|
|
}
|