Workers, verifiers, and serial node goals can now surface out-of-scope work
as line-anchored 'DISCOVERED: <text>' markers (fence-stripped and
placeholder-filtered — the templates' own examples are fenced so verbatim
echoes never parse; worker summaries embedded in verifier prompts get the
marker neutralized alongside NODE_RESULT/NODE_VERDICT). Discoveries queue on
the orchestration as persisted state and fold into the graph at dispatch
boundaries: a replanner subagent produces a strictly APPEND-ONLY appendix,
validated against the live graph (existing-id deps allowed; edges onto
gn-final rejected — they would cycle the moment the final-gating extension
lands; Blocks deps on Failed/Blocked nodes rejected as DeadDep so the
attempt-2 feedback loop repairs the artifact). Installing an appendix bumps
plan_version, freezes an immutable graph.baseline.v{N}.json next to the
prior versions, extends gn-final's gate (demoting a Ready final back to
Waiting), and recomputes readiness.
DiscoveredFrom edges are audit metadata, never scheduling gates: an origin
is always terminal at replan time, so gating on it is either a no-op or a
permanent wedge — and a failed node's discoveries are still real work.
Replanning is bounded by KIGI_GRAPH_REPLAN_CAP (default 3; 0 disables it
quietly): past the cap, after the final node has achieved, or on replan
failure, discoveries drain to history only — a working graph is never
paused for a failed enhancement pass, and it always converges. The budget
gate now precedes the replan boundary (a budget-dead graph keeps its
discoveries queued for a later --budget top-up instead of spending two
replanner runs first), and both planner runners delete stale artifacts
before spawning so a child that responds without writing can never get a
previous pass's file validated as its own output.
Tests: validate_replan unit coverage (existing-id resolution,
DiscoveredFrom dedup, collisions, dead deps vs dead origins, terminal-node
edges, combined-graph cycles), tracker appendix/regate/audit-edge tests,
and two e2e flows — discovery → replan → appended node runs to Achieved
with both baselines frozen, and cap-0 draining to history while the graph
still converges. kigi-shell 4935 lib tests green; workspace clippy clean.
985 lines
43 KiB
Rust
985 lines
43 KiB
Rust
//! Graph mode orchestration seam: a deterministic DAG scheduler layered
|
|
//! over the goal engine.
|
|
//!
|
|
//! Every node executes as one ordinary goal (planner → worker loop →
|
|
//! adversarial verifier), so the agentic loop lives inside the node and
|
|
//! this module only does the deterministic parts: decompose (graph
|
|
//! planner + static validation), launch the next `Ready` node as a fresh
|
|
//! goal, observe the goal's terminal state at the in-turn loop's
|
|
//! `EndTurn` boundary, advance the DAG, and persist every transition via
|
|
//! [`PersistenceMsg::GraphModeState`].
|
|
//!
|
|
//! Failure semantics: goal-side auto-pauses cascade to the graph at the
|
|
//! single chokepoint (`auto_pause_goal_if_active_inner` in `goal.rs`),
|
|
//! and the node budget is always armed with the REMAINING graph budget,
|
|
//! so a mid-node overrun trips the goal engine's own enforcement and is
|
|
//! mirrored here as a graph-level `BudgetLimited`.
|
|
|
|
use std::sync::Arc;
|
|
|
|
use super::super::goal_planner::{ChannelSpawner, GoalPlannerSpawner};
|
|
use super::super::goal_tracker::{GoalPauseReason, GoalStatus};
|
|
use super::super::graph_planner::{
|
|
GRAPH_PLANNER_SUBAGENT_DESCRIPTION, GraphPlannerInputs, GraphPlannerOutcome, run_graph_planner,
|
|
};
|
|
use super::super::graph_tracker::{GraphNode, NodeStatus};
|
|
use super::super::persistence::PersistenceMsg;
|
|
use super::SessionActor;
|
|
|
|
/// Outcome of `/graph <objective>` and `/graph resume` interception in
|
|
/// `handle_prompt`: either flow into inference with a system reminder
|
|
/// (mirrors [`GoalResumeOutcome`](super::goal_support::GoalResumeOutcome))
|
|
/// or print a terminal message and end the turn.
|
|
pub(super) enum GraphSetupOutcome {
|
|
Inference { reminder: String, user_msg: String },
|
|
Message(String),
|
|
}
|
|
|
|
/// Compose the goal objective for one graph node. The node spec is the
|
|
/// contract; the graph context line keeps the node model from wandering
|
|
/// into other nodes' scope.
|
|
pub(super) fn node_goal_objective(
|
|
graph_objective: &str,
|
|
node: &GraphNode,
|
|
position: usize,
|
|
total: usize,
|
|
) -> String {
|
|
format!(
|
|
"[Graph node {position}/{total}: {title}]\n\
|
|
{spec}\n\n\
|
|
This node is one unit of a larger graph objective:\n\
|
|
{graph_objective}\n\n\
|
|
Complete ONLY this node's scope; other nodes cover the rest. If you \
|
|
find NECESSARY work outside this node's scope, do NOT do it — list \
|
|
each item on its own line as `DISCOVERED: <description>` in your \
|
|
final summary; the harness turns these into new graph nodes.",
|
|
title = node.title,
|
|
spec = node.spec,
|
|
)
|
|
}
|
|
|
|
/// Snake-case wire form of a graph/goal status (matches the
|
|
/// `GoalUpdated` vocabulary the pager already parses).
|
|
fn graph_status_str(status: GoalStatus) -> &'static str {
|
|
match status {
|
|
GoalStatus::Active => "active",
|
|
GoalStatus::UserPaused => "user_paused",
|
|
GoalStatus::BackOffPaused => "back_off_paused",
|
|
GoalStatus::NoProgressPaused => "no_progress_paused",
|
|
GoalStatus::InfraPaused => "infra_paused",
|
|
GoalStatus::Blocked => "blocked",
|
|
GoalStatus::BudgetLimited => "budget_limited",
|
|
GoalStatus::Complete => "complete",
|
|
}
|
|
}
|
|
|
|
fn graph_event_as_str(event: &super::super::graph_tracker::GraphEvent) -> &'static str {
|
|
use super::super::graph_tracker::GraphEvent;
|
|
match event {
|
|
GraphEvent::GraphCreated => "graph_created",
|
|
GraphEvent::PlanningStarted => "planning_started",
|
|
GraphEvent::PlanningCompleted => "planning_completed",
|
|
GraphEvent::PlanningFailed => "planning_failed",
|
|
GraphEvent::NodeStarted => "node_started",
|
|
GraphEvent::NodeAchieved => "node_achieved",
|
|
GraphEvent::NodeFailed => "node_failed",
|
|
GraphEvent::GraphPaused => "graph_paused",
|
|
GraphEvent::GraphResumed => "graph_resumed",
|
|
GraphEvent::GraphCompleted => "graph_completed",
|
|
GraphEvent::GraphCleared => "graph_cleared",
|
|
GraphEvent::BudgetExceeded => "budget_exceeded",
|
|
GraphEvent::Unknown => "unknown",
|
|
}
|
|
}
|
|
|
|
/// Build the pager-facing `GraphUpdated` badge payload from a snapshot.
|
|
pub(crate) fn build_graph_updated(
|
|
state: &super::super::graph_tracker::GraphOrchestration,
|
|
) -> crate::extensions::notification::SessionUpdate {
|
|
use super::super::goal_tracker::GoalPhase;
|
|
let count = |s: NodeStatus| state.nodes.iter().filter(|n| n.status == s).count() as u32;
|
|
let current = state
|
|
.current_node
|
|
.as_deref()
|
|
.and_then(|id| state.nodes.iter().find(|n| n.id == id));
|
|
crate::extensions::notification::SessionUpdate::GraphUpdated {
|
|
graph_id: state.graph_id.clone(),
|
|
objective: state.objective.clone(),
|
|
status: graph_status_str(state.status).to_owned(),
|
|
phase: match state.phase {
|
|
GoalPhase::Idle => "idle",
|
|
GoalPhase::Planning => "planning",
|
|
GoalPhase::Executing => "executing",
|
|
}
|
|
.to_owned(),
|
|
plan_version: state.plan_version,
|
|
total_nodes: state.nodes.len() as u32,
|
|
achieved_nodes: count(NodeStatus::Achieved),
|
|
failed_nodes: count(NodeStatus::Failed) + count(NodeStatus::Blocked),
|
|
running_nodes: count(NodeStatus::Running) + count(NodeStatus::Verifying),
|
|
current_node: current.map(|n| n.id.clone()),
|
|
current_node_title: current.map(|n| n.title.clone()),
|
|
token_budget: state.token_budget,
|
|
tokens_spent: state.tokens_spent_nodes,
|
|
last_event: state
|
|
.history
|
|
.last()
|
|
.map(|e| graph_event_as_str(&e.event).to_owned()),
|
|
pause_message: state.pause_message.clone(),
|
|
}
|
|
}
|
|
|
|
/// `status: "cleared"` sentinel — the pager drops its graph state.
|
|
pub(crate) fn build_graph_cleared() -> crate::extensions::notification::SessionUpdate {
|
|
crate::extensions::notification::SessionUpdate::GraphUpdated {
|
|
graph_id: String::new(),
|
|
objective: String::new(),
|
|
status: "cleared".to_owned(),
|
|
phase: "idle".to_owned(),
|
|
plan_version: 0,
|
|
total_nodes: 0,
|
|
achieved_nodes: 0,
|
|
failed_nodes: 0,
|
|
running_nodes: 0,
|
|
current_node: None,
|
|
current_node_title: None,
|
|
token_budget: None,
|
|
tokens_spent: 0,
|
|
last_event: None,
|
|
pause_message: None,
|
|
}
|
|
}
|
|
|
|
impl SessionActor {
|
|
/// Graph feature flag AND the goal harness (nodes execute as goals).
|
|
pub(super) fn graph_harness_enabled(&self) -> bool {
|
|
self.graph_enabled && self.goal_harness_enabled()
|
|
}
|
|
|
|
/// Send the current graph snapshot (or a tombstone after `clear`) to
|
|
/// the persistence actor, AND notify the pager: every graph
|
|
/// transition is both a checkpoint and a `GraphUpdated` badge tick
|
|
/// (single chokepoint — no transition can persist without also
|
|
/// updating the UI, and vice versa).
|
|
pub(crate) fn persist_graph_state(&self) {
|
|
let snapshot = self.graph_tracker.lock().snapshot().cloned();
|
|
let update = match &snapshot {
|
|
Some(state) => build_graph_updated(state),
|
|
None => build_graph_cleared(),
|
|
};
|
|
let _ = self
|
|
.notifications
|
|
.persistence_tx
|
|
.send(PersistenceMsg::GraphModeState(snapshot));
|
|
self.goal_notify_sender().send_update(update);
|
|
}
|
|
|
|
/// True when the graph occupies the goal engine (any non-terminal
|
|
/// state). `/goal` commands are refused in this window.
|
|
pub(super) fn graph_owns_goal_engine(&self) -> bool {
|
|
self.graph_tracker
|
|
.lock()
|
|
.status()
|
|
.is_some_and(|s| s == GoalStatus::Active || s.is_paused())
|
|
}
|
|
|
|
/// Apply `/graph <objective>` — create the graph, run the graph
|
|
/// planner (one validation retry with feedback), install the DAG,
|
|
/// freeze the immutable baseline, and launch the first node.
|
|
pub(super) async fn setup_graph(
|
|
&self,
|
|
objective: &str,
|
|
token_budget: Option<i64>,
|
|
) -> GraphSetupOutcome {
|
|
{
|
|
let goal_status = self.goal_tracker.lock().status();
|
|
if matches!(goal_status, Some(s) if s == GoalStatus::Active || s.is_paused()) {
|
|
return GraphSetupOutcome::Message(
|
|
"A goal is active or paused. Use /goal clear first, then /graph <objective>."
|
|
.to_owned(),
|
|
);
|
|
}
|
|
let graph_status = self.graph_tracker.lock().status();
|
|
// Refuse over ANYTHING non-Complete — including BudgetLimited,
|
|
// which is resumable and must never be silently replaced.
|
|
if matches!(graph_status, Some(s) if s != GoalStatus::Complete) {
|
|
return GraphSetupOutcome::Message(
|
|
"A graph is already set. Use /graph status, /graph resume \
|
|
[--budget <tokens>], or /graph clear."
|
|
.to_owned(),
|
|
);
|
|
}
|
|
}
|
|
|
|
let graph_id = uuid::Uuid::new_v4().to_string();
|
|
tracing::info!(%graph_id, "graph: created, planning started");
|
|
self.graph_tracker.lock().create_graph(
|
|
graph_id,
|
|
objective.to_owned(),
|
|
token_budget,
|
|
chrono::Utc::now().to_rfc3339(),
|
|
);
|
|
self.persist_graph_state();
|
|
|
|
let nodes = match self.run_graph_planning(objective).await {
|
|
Ok(nodes) => nodes,
|
|
Err(reason) => {
|
|
tracing::warn!(%reason, "graph: planning failed; pausing");
|
|
{
|
|
let mut tracker = self.graph_tracker.lock();
|
|
tracker.record_planning_failed(reason.clone());
|
|
tracker.pause_with_message(
|
|
GoalPauseReason::Infra,
|
|
format!("Graph planning failed: {reason}"),
|
|
);
|
|
}
|
|
self.persist_graph_state();
|
|
return GraphSetupOutcome::Message(format!(
|
|
"Graph planning failed: {reason}\n\
|
|
Use /graph resume to retry or /graph clear to abandon."
|
|
));
|
|
}
|
|
};
|
|
|
|
// Freeze the immutable v1 baseline BEFORE anything executes
|
|
// (SGH: plans are immutable within a version boundary). Failing
|
|
// to write the audit baseline is an infra failure, not ignorable.
|
|
if let Err(err) = self.write_graph_baseline(&nodes).await {
|
|
let reason = format!("failed to write graph baseline: {err}");
|
|
tracing::warn!(%reason, "graph: baseline write failed; pausing");
|
|
{
|
|
let mut tracker = self.graph_tracker.lock();
|
|
tracker.record_planning_failed(reason.clone());
|
|
tracker.pause_with_message(GoalPauseReason::Infra, reason.clone());
|
|
}
|
|
self.persist_graph_state();
|
|
return GraphSetupOutcome::Message(format!(
|
|
"{reason}\nUse /graph resume to retry or /graph clear to abandon."
|
|
));
|
|
}
|
|
|
|
let total = nodes.len();
|
|
self.graph_tracker.lock().install_nodes(nodes);
|
|
self.persist_graph_state();
|
|
tracing::info!(total, "graph: DAG installed, launching first node");
|
|
|
|
match self.drive_graph().await {
|
|
Some(reminder) => GraphSetupOutcome::Inference {
|
|
reminder,
|
|
user_msg: format!(
|
|
"Graph created: {total} nodes (incl. final verification). Starting work."
|
|
),
|
|
},
|
|
// The dispatch loop already settled (parallel completion,
|
|
// pause, budget) and messaged; point at the status view.
|
|
None => GraphSetupOutcome::Message(
|
|
"Graph did not enter a serial node. See /graph status.".to_owned(),
|
|
),
|
|
}
|
|
}
|
|
|
|
/// One planning pass with a single validation retry: an `Invalid`
|
|
/// artifact re-runs the planner once with the exact validation error
|
|
/// as CONTEXT; infra failures fail closed immediately.
|
|
async fn run_graph_planning(&self, objective: &str) -> Result<Vec<GraphNode>, String> {
|
|
let Some(event_tx) = self.tool_context.subagent_event_tx.clone() else {
|
|
return Err("no subagent coordinator channel".to_owned());
|
|
};
|
|
let graph_file = self.graph_tracker.lock().artifacts_dir().join("graph.json");
|
|
let parent_prompt_id = self
|
|
.current_prompt_id
|
|
.lock()
|
|
.expect("current_prompt_id mutex poisoned")
|
|
.clone();
|
|
// Verbatim mirror-child fork on the parent model, same rationale
|
|
// as the goal planner (radix-cache reuse).
|
|
let spawner: Arc<dyn GoalPlannerSpawner> = Arc::new(ChannelSpawner {
|
|
event_tx,
|
|
parent_session_id: self.session_id_string(),
|
|
parent_prompt_id,
|
|
cwd: Some(self.tool_context.cwd.as_str().to_owned()),
|
|
role_override: Default::default(),
|
|
events: Some(self.events.writer()),
|
|
});
|
|
let tool_names = self.resolve_inherit_role_tool_names().await;
|
|
|
|
let mut feedback = String::new();
|
|
for attempt in 1..=2u32 {
|
|
tracing::info!(
|
|
attempt,
|
|
role = GRAPH_PLANNER_SUBAGENT_DESCRIPTION,
|
|
"graph planner: firing"
|
|
);
|
|
match run_graph_planner(
|
|
spawner.clone(),
|
|
GraphPlannerInputs {
|
|
objective,
|
|
feedback: &feedback,
|
|
graph_file: &graph_file,
|
|
tool_names: &tool_names,
|
|
inherit_tool_names: &tool_names,
|
|
},
|
|
)
|
|
.await
|
|
{
|
|
GraphPlannerOutcome::Planned(nodes) => return Ok(nodes),
|
|
GraphPlannerOutcome::Invalid { reason } if attempt == 1 => {
|
|
tracing::warn!(%reason, "graph planner: invalid artifact; retrying with feedback");
|
|
feedback = format!(
|
|
"Your previous graph JSON failed validation:\n{reason}\n\
|
|
Rewrite the file fixing exactly this."
|
|
);
|
|
}
|
|
GraphPlannerOutcome::Invalid { reason } => {
|
|
return Err(format!("graph failed validation twice: {reason}"));
|
|
}
|
|
GraphPlannerOutcome::FailClosed { reason } => return Err(reason),
|
|
}
|
|
}
|
|
unreachable!("planning loop returns on every branch by attempt 2")
|
|
}
|
|
|
|
/// Write the immutable plan baseline for the current version.
|
|
/// `create_new` guarantees a frozen baseline is never overwritten —
|
|
/// an existing file is the infra failure it looks like.
|
|
pub(super) async fn write_graph_baseline(&self, nodes: &[GraphNode]) -> std::io::Result<()> {
|
|
let path = {
|
|
let tracker = self.graph_tracker.lock();
|
|
let version = tracker.snapshot().map(|s| s.plan_version).unwrap_or(1);
|
|
tracker.baseline_path(version)
|
|
};
|
|
if let Some(parent) = path.parent() {
|
|
tokio::fs::create_dir_all(parent).await?;
|
|
}
|
|
let json = serde_json::to_vec_pretty(nodes)
|
|
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?;
|
|
let mut file = tokio::fs::OpenOptions::new()
|
|
.write(true)
|
|
.create_new(true)
|
|
.open(&path)
|
|
.await?;
|
|
tokio::io::AsyncWriteExt::write_all(&mut file, &json).await
|
|
}
|
|
|
|
/// Launch the next `Ready` node as a fresh goal on a clean engine.
|
|
/// Returns the node goal's system reminder to seed/continue the turn,
|
|
/// or `None` when nothing is launchable (all done / wedged / budget).
|
|
pub(super) async fn launch_next_graph_node(&self) -> Option<String> {
|
|
let (node_id, node_objective, position, total) = {
|
|
let tracker = self.graph_tracker.lock();
|
|
let node = tracker.next_ready_node()?;
|
|
let snapshot = tracker.snapshot()?;
|
|
let position = snapshot
|
|
.nodes
|
|
.iter()
|
|
.position(|n| n.id == node.id)
|
|
.unwrap_or(0)
|
|
+ 1;
|
|
(
|
|
node.id.clone(),
|
|
node_goal_objective(&snapshot.objective, node, position, snapshot.nodes.len()),
|
|
position,
|
|
snapshot.nodes.len(),
|
|
)
|
|
};
|
|
let node_budget = self.graph_tracker.lock().remaining_budget();
|
|
if node_budget == Some(0) {
|
|
tracing::warn!(%node_id, "graph: budget exhausted before node start");
|
|
self.graph_tracker.lock().budget_limit();
|
|
self.persist_graph_state();
|
|
self.send_slash_command_output(
|
|
"Graph token budget exhausted. Top up with /graph resume --budget <tokens>, \
|
|
or /graph clear to abandon.",
|
|
)
|
|
.await;
|
|
return None;
|
|
}
|
|
|
|
// Fresh engine per node: the previous node's goal state, token
|
|
// records, and pending classifier claims must not leak.
|
|
self.reset_goal_engine_state().await;
|
|
// Mark Running BEFORE the long setup_goal await (its planner run
|
|
// can take minutes): a cancel landing inside it then finds
|
|
// consistent bookkeeping — node Running, cascade pauses the
|
|
// graph, resume retries the node — instead of a Ready node whose
|
|
// goal is already live.
|
|
self.graph_tracker
|
|
.lock()
|
|
.mark_node_running(&node_id, String::new());
|
|
self.persist_graph_state();
|
|
let reminder = self.setup_goal(&node_objective, node_budget).await;
|
|
let goal_id = self
|
|
.goal_tracker
|
|
.lock()
|
|
.snapshot()
|
|
.map(|o| o.goal_id.clone())
|
|
.unwrap_or_default();
|
|
if let Some(node) = self
|
|
.graph_tracker
|
|
.lock()
|
|
.snapshot_mut()
|
|
.and_then(|s| s.nodes.iter_mut().find(|n| n.id == node_id))
|
|
{
|
|
node.goal_id = Some(goal_id.clone());
|
|
}
|
|
self.persist_graph_state();
|
|
// setup_goal fails closed on planner errors by pausing the node
|
|
// goal — and the chokepoint cascade pauses the graph with it. Do
|
|
// NOT hand the now-stale "Start now" reminder to inference.
|
|
if self.goal_tracker.lock().status() != Some(GoalStatus::Active) {
|
|
tracing::warn!(
|
|
%node_id,
|
|
goal_status = ?self.goal_tracker.lock().status(),
|
|
"graph: node goal not active after setup; ending turn instead of launching"
|
|
);
|
|
return None;
|
|
}
|
|
tracing::info!(%node_id, %goal_id, position, total, "graph: node launched");
|
|
Some(reminder)
|
|
}
|
|
|
|
/// Graph seam for the in-turn loop, called when the goal loop decided
|
|
/// `EndTurn`. Reads the node goal's terminal state, advances the DAG,
|
|
/// and returns `Some(reminder)` to keep the turn alive on the next
|
|
/// node — or `None` to genuinely end the turn (graph done, paused,
|
|
/// budget-limited, or not in graph mode at all).
|
|
pub(super) async fn run_graph_round_end(&self) -> Option<String> {
|
|
if !self.graph_harness_enabled() || !self.graph_tracker.lock().is_active() {
|
|
return None;
|
|
}
|
|
let goal_status = self.goal_tracker.lock().status();
|
|
match goal_status {
|
|
Some(GoalStatus::Complete) => self.advance_graph_after_node_complete().await,
|
|
Some(GoalStatus::BudgetLimited) => {
|
|
// Node goals are armed with the remaining graph budget,
|
|
// so a node-level trip IS the graph-level trip. The
|
|
// enforce-side cascade normally handles this; mirroring
|
|
// here is idempotent (budget_limit is Active-only), and
|
|
// the two charge sites are mutually exclusive with it.
|
|
tracing::warn!("graph: node goal budget-limited; graph budget-limited");
|
|
let current_tokens = self.chat_state_handle.get_total_tokens().await as i64;
|
|
let node_tokens = self.goal_tokens_used(current_tokens);
|
|
{
|
|
let mut tracker = self.graph_tracker.lock();
|
|
// Budget integrity: the tripped node's partial burn
|
|
// was still spent — charge it before the demotion
|
|
// clears current_node.
|
|
if let Some(node_id) = tracker.current_node_id().map(str::to_owned) {
|
|
tracker.charge_node_tokens(&node_id, node_tokens);
|
|
}
|
|
tracker.budget_limit();
|
|
}
|
|
self.persist_graph_state();
|
|
None
|
|
}
|
|
Some(s) if s.is_paused() => {
|
|
// The auto-pause chokepoint cascade normally paused the
|
|
// graph before we got here (making is_active() false and
|
|
// returning early above). Reaching this arm means a pause
|
|
// path bypassed the chokepoint — mirror it, loudly.
|
|
tracing::warn!(status = ?s, "graph: node goal paused without cascade; mirroring");
|
|
let reason = match s {
|
|
GoalStatus::BackOffPaused => GoalPauseReason::BackOff,
|
|
GoalStatus::NoProgressPaused => GoalPauseReason::NoProgress,
|
|
GoalStatus::InfraPaused => GoalPauseReason::Infra,
|
|
GoalStatus::Blocked => GoalPauseReason::Verification,
|
|
_ => GoalPauseReason::User,
|
|
};
|
|
let message = self
|
|
.goal_tracker
|
|
.lock()
|
|
.snapshot()
|
|
.and_then(|o| o.pause_message.clone());
|
|
let mut tracker = self.graph_tracker.lock();
|
|
match message {
|
|
Some(msg) => tracker.pause_with_message(reason, msg),
|
|
None => tracker.pause(reason),
|
|
};
|
|
drop(tracker);
|
|
self.persist_graph_state();
|
|
None
|
|
}
|
|
other => {
|
|
// EndTurn with an Active goal cannot happen (an Active
|
|
// goal always yields Continue), and a missing goal while
|
|
// a node is Running is a launch bug. Pause loudly rather
|
|
// than leaving a self-driving graph with no engine.
|
|
tracing::error!(
|
|
goal_status = ?other,
|
|
current_node = ?self.graph_tracker.lock().current_node_id(),
|
|
"graph: inconsistent engine state at round end; pausing graph"
|
|
);
|
|
self.graph_tracker.lock().pause_with_message(
|
|
GoalPauseReason::Infra,
|
|
format!("Inconsistent goal engine state at round end: {other:?}"),
|
|
);
|
|
self.persist_graph_state();
|
|
None
|
|
}
|
|
}
|
|
}
|
|
|
|
/// The `Complete` arm of the seam: harvest the node's cost, archive
|
|
/// its goal artifacts, mark it achieved, and either finish the graph
|
|
/// or launch the next node.
|
|
async fn advance_graph_after_node_complete(&self) -> Option<String> {
|
|
let Some(node_id) = self
|
|
.graph_tracker
|
|
.lock()
|
|
.current_node_id()
|
|
.map(str::to_owned)
|
|
else {
|
|
tracing::error!("graph: goal completed but no current node; pausing graph");
|
|
self.graph_tracker.lock().pause_with_message(
|
|
GoalPauseReason::Infra,
|
|
"Goal completed with no current graph node".to_owned(),
|
|
);
|
|
self.persist_graph_state();
|
|
return None;
|
|
};
|
|
|
|
let current_tokens = self.chat_state_handle.get_total_tokens().await as i64;
|
|
let node_tokens = self.goal_tokens_used(current_tokens);
|
|
let rounds = self
|
|
.goal_tracker
|
|
.lock()
|
|
.snapshot()
|
|
.map(|o| o.total_worker_rounds)
|
|
.unwrap_or(0);
|
|
// Serial-path discovery capture: the node's final assistant text
|
|
// may carry `DISCOVERED:` items (same contract as the parallel
|
|
// workers, zero extra tool surface).
|
|
if let Some(text) = self.chat_state_handle.get_last_assistant_text().await {
|
|
let found = super::graph_workers::parse_discovered_lines(&text);
|
|
if !found.is_empty() {
|
|
let ds: Vec<super::super::graph_tracker::Discovery> = found
|
|
.into_iter()
|
|
.map(|description| super::super::graph_tracker::Discovery {
|
|
from_node: node_id.clone(),
|
|
description,
|
|
})
|
|
.collect();
|
|
self.graph_tracker.lock().queue_discoveries(ds);
|
|
}
|
|
}
|
|
self.archive_node_artifacts(&node_id).await;
|
|
tracing::info!(%node_id, rounds, node_tokens, "graph: node achieved");
|
|
self.graph_tracker
|
|
.lock()
|
|
.mark_node_achieved(&node_id, rounds, node_tokens);
|
|
self.persist_graph_state();
|
|
self.drive_graph().await
|
|
}
|
|
|
|
/// THE single dispatch loop, shared by setup/advance/resume: run
|
|
/// parallel batches while ≥2 nodes are `Ready` (and the cap allows),
|
|
/// then launch the next serial node on the goal engine and return
|
|
/// its reminder — or settle the graph (complete / wedged / paused /
|
|
/// budget) and return `None`. With `graph_concurrency == 1` this
|
|
/// degenerates to exactly the G0 serial behavior.
|
|
pub(super) async fn drive_graph(&self) -> Option<String> {
|
|
loop {
|
|
if !self.graph_tracker.lock().is_active() {
|
|
// Paused/limited during a batch (cancel cascade) or by a
|
|
// serial-launch failure — the pauser already messaged.
|
|
return None;
|
|
}
|
|
// Replan boundary: fold queued discoveries into the graph
|
|
// (bounded; failure degrades to history-only).
|
|
self.maybe_replan_graph().await;
|
|
if self.graph_tracker.lock().remaining_budget() == Some(0) {
|
|
tracing::warn!("graph: budget exhausted at dispatch");
|
|
self.graph_tracker.lock().budget_limit();
|
|
self.persist_graph_state();
|
|
self.send_slash_command_output(
|
|
"Graph token budget exhausted. Top up with /graph resume --budget <tokens>, \
|
|
or /graph clear to abandon.",
|
|
)
|
|
.await;
|
|
return None;
|
|
}
|
|
let ready: Vec<String> = self
|
|
.graph_tracker
|
|
.lock()
|
|
.snapshot()
|
|
.map(|s| {
|
|
s.nodes
|
|
.iter()
|
|
.filter(|n| n.status == NodeStatus::Ready)
|
|
.map(|n| n.id.clone())
|
|
.collect()
|
|
})
|
|
.unwrap_or_default();
|
|
if ready.is_empty() {
|
|
if self.graph_tracker.lock().all_achieved() {
|
|
let (total, objective) = {
|
|
let mut tracker = self.graph_tracker.lock();
|
|
tracker.complete();
|
|
let snapshot = tracker.snapshot();
|
|
(
|
|
snapshot.map(|s| s.nodes.len()).unwrap_or(0),
|
|
snapshot.map(|s| s.objective.clone()).unwrap_or_default(),
|
|
)
|
|
};
|
|
self.persist_graph_state();
|
|
self.reset_goal_engine_state().await;
|
|
tracing::info!(total, "graph: complete");
|
|
self.send_slash_command_output(&format!(
|
|
"Graph complete: all {total} nodes achieved (final verification \
|
|
included).\nObjective: {objective}"
|
|
))
|
|
.await;
|
|
} else if self.graph_tracker.lock().is_wedged() {
|
|
tracing::warn!("graph: wedged (no runnable node, work remaining)");
|
|
self.graph_tracker.lock().pause_with_message(
|
|
GoalPauseReason::Verification,
|
|
"No runnable node left: a dependency chain failed".to_owned(),
|
|
);
|
|
self.persist_graph_state();
|
|
self.send_slash_command_output(
|
|
"Graph blocked: a dependency chain failed and no runnable node is \
|
|
left. See /graph status.",
|
|
)
|
|
.await;
|
|
} else {
|
|
tracing::error!("graph: active with no ready node and work remaining");
|
|
self.graph_tracker.lock().pause_with_message(
|
|
GoalPauseReason::Infra,
|
|
"Scheduler found no runnable node while work remains".to_owned(),
|
|
);
|
|
self.persist_graph_state();
|
|
// Never pause invisibly: the user must know the
|
|
// scheduler stopped and why.
|
|
self.send_slash_command_output(
|
|
"Graph paused: scheduler found no runnable node while work \
|
|
remains. See /graph status.",
|
|
)
|
|
.await;
|
|
}
|
|
return None;
|
|
}
|
|
// Parallel batches need per-node worktrees, which need a git
|
|
// repo. Outside one, degrade to serial (the goal engine works
|
|
// anywhere) instead of letting N workers share one tree.
|
|
let parallel_possible = kigi_workspace::session::git::find_git_root_from_path(
|
|
self.tool_context.cwd.as_path(),
|
|
)
|
|
.is_ok();
|
|
let cap = if parallel_possible {
|
|
self.graph_concurrency as usize
|
|
} else {
|
|
1
|
|
};
|
|
if cap <= 1 || ready.len() == 1 {
|
|
return self.launch_next_graph_node().await;
|
|
}
|
|
let batch: Vec<String> = ready.into_iter().take(cap).collect();
|
|
// No goal is Active during a batch, so arm the interrupt
|
|
// gate explicitly: a queued user prompt must stack FIFO
|
|
// behind the batch instead of cancelling the graph turn
|
|
// (the in-turn loop re-syncs the gate from goal status at
|
|
// every round start).
|
|
self.set_goal_loop_active_resource(true).await;
|
|
self.run_graph_parallel_batch(batch).await;
|
|
// Loop: the batch resolved nodes (achieved/failed); recompute
|
|
// and keep driving — another batch, a serial tail (gn-final),
|
|
// or settle.
|
|
}
|
|
}
|
|
|
|
/// Copy the node goal's durable artifacts (plan, plan baseline,
|
|
/// strategy note) into the node's archive dir before the engine is
|
|
/// cleared for the next node. Sources come from the ORCHESTRATION's
|
|
/// own claims (`plan_file`/`plan_baseline_file`/`last_strategy_path`),
|
|
/// never from bare path probing — a stale file left by a previous
|
|
/// node must not be archived as this node's work. Best-effort: a
|
|
/// failed copy loses audit detail, never progress — but always logs.
|
|
async fn archive_node_artifacts(&self, node_id: &str) {
|
|
let (sources, dst_dir) = {
|
|
let goal = self.goal_tracker.lock();
|
|
let graph = self.graph_tracker.lock();
|
|
let mut sources: Vec<std::path::PathBuf> = Vec::new();
|
|
if let Some(o) = goal.snapshot() {
|
|
if let Some(p) = &o.plan_file {
|
|
sources.push(p.clone());
|
|
}
|
|
if let Some(p) = &o.plan_baseline_file {
|
|
sources.push(p.clone());
|
|
}
|
|
if let Some(p) = &o.last_strategy_path {
|
|
sources.push(std::path::PathBuf::from(p));
|
|
}
|
|
}
|
|
(sources, graph.node_archive_dir(node_id))
|
|
};
|
|
if sources.is_empty() {
|
|
return;
|
|
}
|
|
if let Err(err) = tokio::fs::create_dir_all(&dst_dir).await {
|
|
tracing::warn!(%node_id, %err, "graph: failed to create node archive dir");
|
|
return;
|
|
}
|
|
for src in sources {
|
|
if !src.is_file() {
|
|
continue;
|
|
}
|
|
let Some(name) = src.file_name() else {
|
|
continue;
|
|
};
|
|
if let Err(err) = tokio::fs::copy(&src, dst_dir.join(name)).await {
|
|
tracing::warn!(%node_id, src = %src.display(), %err, "graph: artifact archive copy failed");
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Clear all goal-engine session state (tracker, streaks, task ids,
|
|
/// token records, pending classifier claims) and tell the pager.
|
|
/// Shared by `/goal clear`, graph node boundaries, and `/graph clear`.
|
|
pub(super) async fn reset_goal_engine_state(&self) {
|
|
self.goal_tracker.lock().clear();
|
|
self.goal_continuation_streak
|
|
.store(0, std::sync::atomic::Ordering::Relaxed);
|
|
self.goal_blocked_streak
|
|
.store(0, std::sync::atomic::Ordering::Relaxed);
|
|
self.goal_turn_task_ids.lock().clear();
|
|
self.subagent_token_records.lock().clear();
|
|
self.clear_pending_classifier_completions();
|
|
let update = crate::session::goal_orchestrator::build_goal_cleared();
|
|
self.send_xai_notification(update).await;
|
|
}
|
|
|
|
/// Apply `/graph resume`: re-arm a paused graph. If the current node's
|
|
/// goal is still in the engine and paused, resume it (planner retry
|
|
/// included); otherwise (post-restart empty engine) launch the next
|
|
/// `Ready` node fresh — the verifier gates completion, so re-running
|
|
/// a node is always safe.
|
|
pub(super) async fn resume_graph(&self, extra_budget: Option<i64>) -> GraphSetupOutcome {
|
|
use super::goal_support::GoalResumeOutcome;
|
|
let status = self.graph_tracker.lock().status();
|
|
match status {
|
|
None => GraphSetupOutcome::Message(
|
|
"No graph is set. Use /graph <objective> to start one.".to_owned(),
|
|
),
|
|
Some(GoalStatus::Active) => {
|
|
GraphSetupOutcome::Message("Graph is already running.".to_owned())
|
|
}
|
|
Some(GoalStatus::Complete) => GraphSetupOutcome::Message(
|
|
"Graph is already complete. Use /graph <objective> to start a new one.".to_owned(),
|
|
),
|
|
Some(GoalStatus::BudgetLimited) => {
|
|
let Some(extra) = extra_budget else {
|
|
return GraphSetupOutcome::Message(
|
|
"Graph is budget-limited. Top up with /graph resume --budget \
|
|
<tokens>, or /graph clear to abandon."
|
|
.to_owned(),
|
|
);
|
|
};
|
|
if !self.graph_tracker.lock().resume_budget_limited(extra) {
|
|
return GraphSetupOutcome::Message(
|
|
"Budget top-up must be a positive token count.".to_owned(),
|
|
);
|
|
}
|
|
self.persist_graph_state();
|
|
tracing::info!(extra, "graph: resumed with budget top-up");
|
|
match self.drive_graph().await {
|
|
Some(reminder) => GraphSetupOutcome::Inference {
|
|
reminder,
|
|
user_msg: format!("Graph resumed with {extra} fresh budget tokens."),
|
|
},
|
|
None => GraphSetupOutcome::Message(
|
|
"Graph resumed with fresh budget but did not enter a serial node. \
|
|
See /graph status."
|
|
.to_owned(),
|
|
),
|
|
}
|
|
}
|
|
Some(s) if s.is_paused() => {
|
|
if extra_budget.is_some() {
|
|
// Never silently discard an explicit flag.
|
|
return GraphSetupOutcome::Message(
|
|
"--budget only applies to a budget-limited graph; this graph is \
|
|
paused. Use /graph resume (no flags) to continue, or /graph \
|
|
clear to abandon."
|
|
.to_owned(),
|
|
);
|
|
}
|
|
{
|
|
let mut tracker = self.graph_tracker.lock();
|
|
tracker.resume();
|
|
// A cancel during a parallel batch left its nodes
|
|
// marked Running with no executor. Demote every
|
|
// orphaned in-flight node back to Ready, keeping only
|
|
// the node whose goal actually lives in the engine —
|
|
// otherwise resume wedges forever (nothing Ready,
|
|
// nothing achieved, is_wedged false).
|
|
let keep = self
|
|
.goal_tracker
|
|
.lock()
|
|
.snapshot()
|
|
.is_some()
|
|
.then(|| tracker.current_node_id().map(str::to_owned))
|
|
.flatten();
|
|
tracker.demote_orphaned_in_flight(keep.as_deref());
|
|
}
|
|
self.persist_graph_state();
|
|
tracing::info!("graph: resumed");
|
|
// Planning never finished? Re-plan before touching nodes.
|
|
let needs_planning = self
|
|
.graph_tracker
|
|
.lock()
|
|
.snapshot()
|
|
.is_some_and(|s| s.nodes.is_empty());
|
|
if needs_planning {
|
|
let objective = self
|
|
.graph_tracker
|
|
.lock()
|
|
.objective()
|
|
.map(str::to_owned)
|
|
.unwrap_or_default();
|
|
return match self.finish_planning_on_resume(&objective).await {
|
|
Ok(Some(reminder)) => GraphSetupOutcome::Inference {
|
|
reminder,
|
|
user_msg: "Graph resumed; planning retried.".to_owned(),
|
|
},
|
|
Ok(None) => GraphSetupOutcome::Message(
|
|
"Graph resumed but no node could start. See /graph status.".to_owned(),
|
|
),
|
|
Err(msg) => GraphSetupOutcome::Message(msg),
|
|
};
|
|
}
|
|
let goal_paused = self
|
|
.goal_tracker
|
|
.lock()
|
|
.status()
|
|
.is_some_and(|s| s.is_paused());
|
|
if goal_paused {
|
|
match self.resume_goal().await {
|
|
GoalResumeOutcome::Inference { reminder, user_msg } => {
|
|
GraphSetupOutcome::Inference {
|
|
reminder,
|
|
user_msg: format!("Graph resumed. {user_msg}"),
|
|
}
|
|
}
|
|
GoalResumeOutcome::Message(msg) => {
|
|
// The node goal re-paused (e.g. planner failed
|
|
// again); the cascade re-paused the graph.
|
|
GraphSetupOutcome::Message(format!("Graph resume: {msg}"))
|
|
}
|
|
}
|
|
} else {
|
|
match self.drive_graph().await {
|
|
Some(reminder) => GraphSetupOutcome::Inference {
|
|
reminder,
|
|
user_msg: "Graph resumed.".to_owned(),
|
|
},
|
|
None => GraphSetupOutcome::Message(
|
|
"Graph resumed but did not enter a serial node. See /graph status."
|
|
.to_owned(),
|
|
),
|
|
}
|
|
}
|
|
}
|
|
Some(other) => GraphSetupOutcome::Message(format!(
|
|
"Graph is in an unexpected state ({other:?}); use /graph status."
|
|
)),
|
|
}
|
|
}
|
|
|
|
/// Resume path for a graph that paused during planning: retry the
|
|
/// planner, install on success, launch the first node.
|
|
async fn finish_planning_on_resume(&self, objective: &str) -> Result<Option<String>, String> {
|
|
match self.run_graph_planning(objective).await {
|
|
Ok(nodes) => {
|
|
if let Err(err) = self.write_graph_baseline(&nodes).await {
|
|
let reason = format!("failed to write graph baseline: {err}");
|
|
self.graph_tracker
|
|
.lock()
|
|
.pause_with_message(GoalPauseReason::Infra, reason.clone());
|
|
self.persist_graph_state();
|
|
return Err(reason);
|
|
}
|
|
self.graph_tracker.lock().install_nodes(nodes);
|
|
self.persist_graph_state();
|
|
Ok(self.drive_graph().await)
|
|
}
|
|
Err(reason) => {
|
|
{
|
|
let mut tracker = self.graph_tracker.lock();
|
|
tracker.record_planning_failed(reason.clone());
|
|
tracker.pause_with_message(
|
|
GoalPauseReason::Infra,
|
|
format!("Graph planning failed: {reason}"),
|
|
);
|
|
}
|
|
self.persist_graph_state();
|
|
Err(format!(
|
|
"Graph planning failed again: {reason}\nUse /graph resume to retry."
|
|
))
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Render the `/graph status` tree.
|
|
pub(super) async fn graph_status_message(&self) -> String {
|
|
let current_tokens = self.chat_state_handle.get_total_tokens().await as i64;
|
|
let node_goal_tokens = self.goal_tokens_used(current_tokens);
|
|
let tracker = self.graph_tracker.lock();
|
|
let Some(s) = tracker.snapshot() else {
|
|
return "No graph is set. Use /graph <objective> to start one.".to_owned();
|
|
};
|
|
let achieved = s
|
|
.nodes
|
|
.iter()
|
|
.filter(|n| n.status == NodeStatus::Achieved)
|
|
.count();
|
|
let mut buf = format!(
|
|
"Graph: {}\nStatus: {:?} | Phase: {:?} | Plan v{}\nNodes: {achieved}/{} achieved\n",
|
|
s.objective,
|
|
s.status,
|
|
s.phase,
|
|
s.plan_version,
|
|
s.nodes.len(),
|
|
);
|
|
for node in &s.nodes {
|
|
let glyph = match node.status {
|
|
NodeStatus::Achieved => "[x]",
|
|
NodeStatus::Running | NodeStatus::Verifying => "[>]",
|
|
NodeStatus::Ready => "[ ]",
|
|
NodeStatus::Waiting => "[.]",
|
|
NodeStatus::Failed => "[!]",
|
|
NodeStatus::Blocked => "[-]",
|
|
};
|
|
buf.push_str(&format!(" {glyph} {} — {}", node.id, node.title));
|
|
if node.status == NodeStatus::Waiting && !node.deps.is_empty() {
|
|
let deps: Vec<&str> = node.deps.iter().map(|d| d.on.as_str()).collect();
|
|
buf.push_str(&format!(" (waiting on {})", deps.join(", ")));
|
|
}
|
|
if node.status == NodeStatus::Achieved {
|
|
buf.push_str(&format!(
|
|
" ({} tokens, {} rounds)",
|
|
node.tokens_used, node.rounds
|
|
));
|
|
}
|
|
if let Some(failure) = &node.failure {
|
|
buf.push_str(&format!(" — {failure}"));
|
|
}
|
|
buf.push('\n');
|
|
}
|
|
let mut tokens = s.tokens_spent_nodes;
|
|
if s.current_node.is_some() {
|
|
tokens += node_goal_tokens;
|
|
}
|
|
buf.push_str(&format!("Tokens: {tokens}"));
|
|
if let Some(budget) = s.token_budget {
|
|
buf.push_str(&format!(" | Budget: {budget}"));
|
|
}
|
|
if let Some(node_id) = &s.current_node {
|
|
buf.push_str(&format!("\nCurrent node: {node_id}"));
|
|
}
|
|
if let Some(msg) = &s.pause_message {
|
|
buf.push_str(&format!("\nPaused: {msg}"));
|
|
}
|
|
buf
|
|
}
|
|
}
|