Add /graph G0: serial graph engineering mode over the goal engine
A deterministic DAG scheduler layered on the existing goal engine: /graph <objective> decomposes the objective via a graph-planner subagent, gates the result through Agentproof-style static validation (cycles, unknown deps, duplicate slugs, caps), appends a structural final-verification node, then executes each node as one ordinary goal — planner, worker loop, adversarial verifier, budget and pause machinery all reused verbatim. The in-turn loop advances nodes within the same turn (multi-loop closed loop); goal-side auto-pauses cascade to the graph at a single chokepoint; node goals are armed with the remaining graph budget so mid-node overruns trip graph-wide. Gated by KIGI_GRAPH=1 (default off) + the goal harness. State persists to <session_dir>/graph/state.json with a clear-tombstone; per-version immutable baselines and per-node artifact archives live under graph/<graph_id>/. Restore demotes Active->UserPaused and Running->Ready (verifier-gated re-run). Review hardening (adversarial multi-agent pass, 24 confirmed findings fixed): the goal-inactive loop break now consults the graph seam (mid-turn classifier-disabled completions can no longer strand an Active graph), the pause cascade fires even when no node goal is in flight (cancel during planning), node bookkeeping precedes the long setup await, /graph clear only resets the engine it owns, budget trips terminally fail the in-flight node, and the pause transitions in /goal pause + /graph pause no longer hide inside debug_assert! (a release-build no-op inherited from upstream). Tests: 4908 kigi-shell lib tests green, including a serial 4-node closed-loop e2e, restore/resume, cascade, mutual-exclusion, planning-retry, persistence round-trip + tombstone, and status rendering.
This commit is contained in:
1 parent
2c29d3ecd5
commit
4361b03259
31 files changed
+3760
-50
No files matched your search
@@ -0,0 +1,874 @@
|
||||
//! Graph mode state machine.
|
||||
//!
|
||||
//! This module contains [`GraphTracker`], a pure state machine (no async
|
||||
//! I/O) modeled after [`GoalTracker`](super::goal_tracker::GoalTracker).
|
||||
//! The `SessionActor` owns one `GraphTracker` behind a `Mutex` and calls
|
||||
//! its methods at the graph orchestration points.
|
||||
//!
|
||||
//! Architecture: a graph is a deterministic scheduler over goals. Each
|
||||
//! node executes as one ordinary goal on the existing goal engine
|
||||
//! (planner, worker loop, adversarial verifier, budget, pauses), so the
|
||||
//! agentic loop lives INSIDE the node and the edges between nodes stay
|
||||
//! deterministic Rust. The graph layer therefore reuses the goal
|
||||
//! vocabulary — [`GoalStatus`], [`GoalPhase`], [`GoalPauseReason`] — and
|
||||
//! adds only the DAG bookkeeping.
|
||||
//!
|
||||
//! Restore semantics: a snapshot restored from disk can never resurrect
|
||||
//! as a self-driving graph. `from_snapshot` demotes `Active` to
|
||||
//! `UserPaused` and any `Running`/`Verifying` node back to `Ready`; the
|
||||
//! user re-arms with `/graph resume`, which re-launches the node as a
|
||||
//! fresh goal (the per-node verifier gates completion, so a re-run is
|
||||
//! always safe).
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::time::Instant;
|
||||
|
||||
use super::goal_tracker::{GoalPauseReason, GoalPhase, GoalStatus};
|
||||
|
||||
/// Max retained graph-history entries; oldest dropped past the cap so a
|
||||
/// long graph's snapshot stays bounded. Mirrors `GOAL_HISTORY_MAX`.
|
||||
const GRAPH_HISTORY_MAX: usize = 64;
|
||||
|
||||
/// Canonical id of the harness-appended terminal verification node. It
|
||||
/// depends on every planner node and its goal re-verifies the OVERALL
|
||||
/// objective, closing the composition gap ("every node passed but the
|
||||
/// whole didn't").
|
||||
pub const FINAL_NODE_ID: &str = "gn-final";
|
||||
|
||||
// Node status / dependency kinds
|
||||
|
||||
/// Lifecycle status of one graph node. Single-direction machine:
|
||||
/// `Waiting -> Ready -> Running -> Achieved`, with `Failed` (node judged
|
||||
/// unachievable / budget-dead) and `Blocked` (a dependency failed) as
|
||||
/// terminal side exits. Retries of a node stay in `Running` across
|
||||
/// rounds — the goal engine owns intra-node iteration.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum NodeStatus {
|
||||
Waiting,
|
||||
Ready,
|
||||
Running,
|
||||
/// Reserved for the live "verifier in flight" badge (G2); the G0
|
||||
/// scheduler never stores it and `from_snapshot` demotes it to
|
||||
/// `Ready`.
|
||||
Verifying,
|
||||
Achieved,
|
||||
Failed,
|
||||
Blocked,
|
||||
}
|
||||
|
||||
impl<'de> serde::Deserialize<'de> for NodeStatus {
|
||||
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
|
||||
where
|
||||
D: serde::Deserializer<'de>,
|
||||
{
|
||||
let s = String::deserialize(deserializer)?;
|
||||
Ok(Self::from_wire_str(&s))
|
||||
}
|
||||
}
|
||||
|
||||
impl NodeStatus {
|
||||
/// Parse a persisted status string. Unknown values map to `Ready`:
|
||||
/// a node state this shell cannot interpret must restore as
|
||||
/// re-runnable work, never as silently-done or stuck.
|
||||
pub fn from_wire_str(s: &str) -> Self {
|
||||
match s {
|
||||
"waiting" => Self::Waiting,
|
||||
"ready" => Self::Ready,
|
||||
"running" => Self::Running,
|
||||
"verifying" => Self::Verifying,
|
||||
"achieved" => Self::Achieved,
|
||||
"failed" => Self::Failed,
|
||||
"blocked" => Self::Blocked,
|
||||
_ => Self::Ready,
|
||||
}
|
||||
}
|
||||
|
||||
/// Terminal states the scheduler never leaves.
|
||||
pub fn is_terminal(&self) -> bool {
|
||||
matches!(self, Self::Achieved | Self::Failed | Self::Blocked)
|
||||
}
|
||||
}
|
||||
|
||||
/// Dependency edge kind. `Blocks` is the planner-authored ordering
|
||||
/// dependency; `DiscoveredFrom` marks a node appended by a replan (G3)
|
||||
/// pointing back at the node whose execution surfaced it. Both gate
|
||||
/// scheduling identically; the kind is audit/render metadata.
|
||||
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum DepKind {
|
||||
#[default]
|
||||
Blocks,
|
||||
DiscoveredFrom,
|
||||
}
|
||||
|
||||
/// One dependency edge: this node cannot start until `on` is `Achieved`.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
|
||||
pub struct NodeDep {
|
||||
pub on: String,
|
||||
#[serde(default)]
|
||||
pub kind: DepKind,
|
||||
}
|
||||
|
||||
// GraphNode
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
pub struct GraphNode {
|
||||
/// Short content-derived id (`gn-<fnv hex>`); stable across replans
|
||||
/// of the same node title so cross-machine graph merges (G4) stay
|
||||
/// line-mergeable.
|
||||
pub id: String,
|
||||
pub title: String,
|
||||
/// Node-level objective — the core of the node goal's objective
|
||||
/// string. Written by the graph planner.
|
||||
pub spec: String,
|
||||
#[serde(default)]
|
||||
pub deps: Vec<NodeDep>,
|
||||
pub status: NodeStatus,
|
||||
/// Goal id of the node's most recent goal instance (`None` until
|
||||
/// first launch). Links the node to the goal engine's own artifacts.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub goal_id: Option<String>,
|
||||
/// Worker rounds the node's goal consumed, recorded at node
|
||||
/// completion.
|
||||
#[serde(default)]
|
||||
pub rounds: u32,
|
||||
/// Goal-scoped tokens the node consumed, recorded at node
|
||||
/// completion.
|
||||
#[serde(default)]
|
||||
pub tokens_used: i64,
|
||||
/// Short failure detail when `status` is `Failed`/`Blocked`.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub failure: Option<String>,
|
||||
}
|
||||
|
||||
// History
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum GraphEvent {
|
||||
GraphCreated,
|
||||
PlanningStarted,
|
||||
PlanningCompleted,
|
||||
PlanningFailed,
|
||||
NodeStarted,
|
||||
NodeAchieved,
|
||||
NodeFailed,
|
||||
GraphPaused,
|
||||
GraphResumed,
|
||||
GraphCompleted,
|
||||
GraphCleared,
|
||||
BudgetExceeded,
|
||||
/// Forward-compat sink for history written by a newer shell.
|
||||
#[serde(other)]
|
||||
Unknown,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
pub struct GraphHistoryEntry {
|
||||
pub timestamp: String,
|
||||
pub event: GraphEvent,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub node_id: Option<String>,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub detail: Option<String>,
|
||||
}
|
||||
|
||||
impl GraphHistoryEntry {
|
||||
pub(crate) fn now(event: GraphEvent, node_id: Option<String>, detail: Option<String>) -> Self {
|
||||
Self {
|
||||
timestamp: chrono::Utc::now().to_rfc3339(),
|
||||
event,
|
||||
node_id,
|
||||
detail,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// GraphOrchestration (full persisted state)
|
||||
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
pub struct GraphOrchestration {
|
||||
pub graph_id: String,
|
||||
pub objective: String,
|
||||
pub status: GoalStatus,
|
||||
pub phase: GoalPhase,
|
||||
/// Monotonic plan version; replans/optimizer passes (G3/G6) bump it.
|
||||
/// Version 1 is the initial planner output.
|
||||
#[serde(default = "default_plan_version")]
|
||||
pub plan_version: u32,
|
||||
/// Topological-friendly storage order (planner order + harness
|
||||
/// appendix). The scheduler picks the first `Ready` node in this
|
||||
/// order, so execution is deterministic.
|
||||
pub nodes: Vec<GraphNode>,
|
||||
/// Node currently running as the active goal, if any.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub current_node: Option<String>,
|
||||
pub created_at: String,
|
||||
#[serde(default)]
|
||||
pub elapsed_ms: u64,
|
||||
/// Graph-level token budget; each node goal is armed with the
|
||||
/// remaining share so mid-node overruns trip the goal engine's own
|
||||
/// enforcement.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub token_budget: Option<i64>,
|
||||
/// Tokens consumed by completed node goals (boundary-accumulated).
|
||||
#[serde(default)]
|
||||
pub tokens_spent_nodes: i64,
|
||||
pub history: Vec<GraphHistoryEntry>,
|
||||
/// Human-readable reason set on paused/blocked transitions; cleared
|
||||
/// on resume/complete (mirrors `GoalOrchestration::pause_message`).
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub pause_message: Option<String>,
|
||||
}
|
||||
|
||||
fn default_plan_version() -> u32 {
|
||||
1
|
||||
}
|
||||
|
||||
// GraphTracker
|
||||
|
||||
/// Pure graph-mode state machine owned by the `SessionActor`.
|
||||
pub struct GraphTracker {
|
||||
session_dir: PathBuf,
|
||||
state: Option<GraphOrchestration>,
|
||||
/// Wall-clock anchor for `account_elapsed`; `Some` only while the
|
||||
/// graph is `Active`. Never persisted.
|
||||
last_probe: Option<Instant>,
|
||||
}
|
||||
|
||||
impl GraphTracker {
|
||||
pub fn new(session_dir: PathBuf) -> Self {
|
||||
Self {
|
||||
session_dir,
|
||||
state: None,
|
||||
last_probe: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Restore from a persisted snapshot, sanitized so it can never
|
||||
/// resurrect self-driving: `Active` demotes to `UserPaused` (the
|
||||
/// in-turn loop that drove it is gone) and `Running`/`Verifying`
|
||||
/// nodes demote to `Ready` for a fresh, verifier-gated re-run.
|
||||
pub fn from_snapshot(session_dir: PathBuf, mut snapshot: GraphOrchestration) -> Self {
|
||||
if snapshot.status == GoalStatus::Active {
|
||||
snapshot.status = GoalStatus::UserPaused;
|
||||
snapshot.pause_message =
|
||||
Some("Restored after a restart. Use /graph resume to continue.".to_owned());
|
||||
}
|
||||
for node in &mut snapshot.nodes {
|
||||
if matches!(node.status, NodeStatus::Running | NodeStatus::Verifying) {
|
||||
node.status = NodeStatus::Ready;
|
||||
}
|
||||
}
|
||||
snapshot.current_node = None;
|
||||
let mut tracker = Self {
|
||||
session_dir,
|
||||
state: Some(snapshot),
|
||||
last_probe: None,
|
||||
};
|
||||
tracker.recompute_ready();
|
||||
tracker
|
||||
}
|
||||
|
||||
// Paths
|
||||
|
||||
/// `<session_dir>/graph` — root for graph-owned state (`state.json`
|
||||
/// lives here, session-scoped: one current graph per session).
|
||||
pub fn graph_dir(&self) -> PathBuf {
|
||||
self.session_dir.join("graph")
|
||||
}
|
||||
|
||||
/// Per-graph artifact root (`<session_dir>/graph/<graph_id>`), so a
|
||||
/// later `/graph` in the same session can never overwrite a prior
|
||||
/// graph's frozen baselines or node archives. Falls back to
|
||||
/// `graph_dir` when no graph is set (callers always have one).
|
||||
pub fn artifacts_dir(&self) -> PathBuf {
|
||||
match self.state.as_ref() {
|
||||
Some(s) => self.graph_dir().join(&s.graph_id),
|
||||
None => self.graph_dir(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Immutable baseline snapshot for `version`
|
||||
/// (`<artifacts_dir>/graph.baseline.v{N}.json`).
|
||||
pub fn baseline_path(&self, version: u32) -> PathBuf {
|
||||
self.artifacts_dir()
|
||||
.join(format!("graph.baseline.v{version}.json"))
|
||||
}
|
||||
|
||||
/// Per-node artifact archive dir (`<artifacts_dir>/<node_id>`).
|
||||
/// Node-goal artifacts (plan.md, …) are copied here when the node
|
||||
/// completes, before the goal engine is cleared for the next node.
|
||||
pub fn node_archive_dir(&self, node_id: &str) -> PathBuf {
|
||||
self.artifacts_dir().join(node_id)
|
||||
}
|
||||
|
||||
// Accessors
|
||||
|
||||
pub fn snapshot(&self) -> Option<&GraphOrchestration> {
|
||||
self.state.as_ref()
|
||||
}
|
||||
|
||||
pub fn snapshot_mut(&mut self) -> Option<&mut GraphOrchestration> {
|
||||
self.state.as_mut()
|
||||
}
|
||||
|
||||
pub fn status(&self) -> Option<GoalStatus> {
|
||||
self.state.as_ref().map(|s| s.status)
|
||||
}
|
||||
|
||||
pub fn is_active(&self) -> bool {
|
||||
self.status() == Some(GoalStatus::Active)
|
||||
}
|
||||
|
||||
pub fn objective(&self) -> Option<&str> {
|
||||
self.state.as_ref().map(|s| s.objective.as_str())
|
||||
}
|
||||
|
||||
pub fn current_node_id(&self) -> Option<&str> {
|
||||
self.state.as_ref()?.current_node.as_deref()
|
||||
}
|
||||
|
||||
pub fn node(&self, id: &str) -> Option<&GraphNode> {
|
||||
self.state.as_ref()?.nodes.iter().find(|n| n.id == id)
|
||||
}
|
||||
|
||||
/// Remaining graph token budget (`None` when no budget is set).
|
||||
/// Saturates at zero.
|
||||
pub fn remaining_budget(&self) -> Option<i64> {
|
||||
let s = self.state.as_ref()?;
|
||||
let budget = s.token_budget?;
|
||||
Some((budget - s.tokens_spent_nodes).max(0))
|
||||
}
|
||||
|
||||
// Transitions
|
||||
|
||||
/// Create a fresh graph in `Planning` phase with no nodes yet.
|
||||
pub fn create_graph(
|
||||
&mut self,
|
||||
graph_id: String,
|
||||
objective: String,
|
||||
token_budget: Option<i64>,
|
||||
created_at: String,
|
||||
) {
|
||||
let mut state = GraphOrchestration {
|
||||
graph_id,
|
||||
objective,
|
||||
status: GoalStatus::Active,
|
||||
phase: GoalPhase::Planning,
|
||||
plan_version: 1,
|
||||
nodes: Vec::new(),
|
||||
current_node: None,
|
||||
created_at,
|
||||
elapsed_ms: 0,
|
||||
token_budget,
|
||||
tokens_spent_nodes: 0,
|
||||
history: Vec::new(),
|
||||
pause_message: None,
|
||||
};
|
||||
state
|
||||
.history
|
||||
.push(GraphHistoryEntry::now(GraphEvent::GraphCreated, None, None));
|
||||
state.history.push(GraphHistoryEntry::now(
|
||||
GraphEvent::PlanningStarted,
|
||||
None,
|
||||
None,
|
||||
));
|
||||
self.state = Some(state);
|
||||
self.last_probe = Some(Instant::now());
|
||||
}
|
||||
|
||||
/// Install the validated node set (planner output + harness-appended
|
||||
/// final node) and move to `Executing`. Roots become `Ready`.
|
||||
pub fn install_nodes(&mut self, nodes: Vec<GraphNode>) {
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return;
|
||||
};
|
||||
state.nodes = nodes;
|
||||
state.phase = GoalPhase::Executing;
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(GraphEvent::PlanningCompleted, None, None),
|
||||
);
|
||||
self.recompute_ready();
|
||||
}
|
||||
|
||||
/// Record a planning failure in history (the caller pauses the graph
|
||||
/// with the canonical message).
|
||||
pub fn record_planning_failed(&mut self, detail: String) {
|
||||
if let Some(state) = self.state.as_mut() {
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(GraphEvent::PlanningFailed, None, Some(detail)),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// First `Ready` node in storage order — the deterministic serial
|
||||
/// scheduling rule.
|
||||
pub fn next_ready_node(&self) -> Option<&GraphNode> {
|
||||
self.state
|
||||
.as_ref()?
|
||||
.nodes
|
||||
.iter()
|
||||
.find(|n| n.status == NodeStatus::Ready)
|
||||
}
|
||||
|
||||
/// Mark `id` as launched under goal `goal_id`.
|
||||
pub fn mark_node_running(&mut self, id: &str, goal_id: String) {
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return;
|
||||
};
|
||||
if let Some(node) = state.nodes.iter_mut().find(|n| n.id == id) {
|
||||
node.status = NodeStatus::Running;
|
||||
node.goal_id = Some(goal_id);
|
||||
}
|
||||
state.current_node = Some(id.to_owned());
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(GraphEvent::NodeStarted, Some(id.to_owned()), None),
|
||||
);
|
||||
}
|
||||
|
||||
/// Mark `id` achieved with its consumed rounds/tokens, clear the
|
||||
/// current-node pointer, and unlock dependents.
|
||||
pub fn mark_node_achieved(&mut self, id: &str, rounds: u32, tokens_used: i64) {
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return;
|
||||
};
|
||||
if let Some(node) = state.nodes.iter_mut().find(|n| n.id == id) {
|
||||
node.status = NodeStatus::Achieved;
|
||||
node.rounds = rounds;
|
||||
node.tokens_used = tokens_used;
|
||||
}
|
||||
state.tokens_spent_nodes = state.tokens_spent_nodes.saturating_add(tokens_used);
|
||||
state.current_node = None;
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(GraphEvent::NodeAchieved, Some(id.to_owned()), None),
|
||||
);
|
||||
self.recompute_ready();
|
||||
}
|
||||
|
||||
/// Mark `id` failed with `detail`, clear the current-node pointer,
|
||||
/// and block every transitive dependent. The caller decides the
|
||||
/// graph-level consequence (pause/block).
|
||||
pub fn mark_node_failed(&mut self, id: &str, detail: String) {
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return;
|
||||
};
|
||||
if let Some(node) = state.nodes.iter_mut().find(|n| n.id == id) {
|
||||
node.status = NodeStatus::Failed;
|
||||
node.failure = Some(detail.clone());
|
||||
}
|
||||
state.current_node = None;
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(GraphEvent::NodeFailed, Some(id.to_owned()), Some(detail)),
|
||||
);
|
||||
block_dependents(state, id);
|
||||
}
|
||||
|
||||
/// All nodes `Achieved` — the graph's success condition.
|
||||
pub fn all_achieved(&self) -> bool {
|
||||
self.state.as_ref().is_some_and(|s| {
|
||||
!s.nodes.is_empty() && s.nodes.iter().all(|n| n.status == NodeStatus::Achieved)
|
||||
})
|
||||
}
|
||||
|
||||
/// True when no node can make progress: nothing `Ready`/`Running`
|
||||
/// and at least one node is not `Achieved`.
|
||||
pub fn is_wedged(&self) -> bool {
|
||||
self.state.as_ref().is_some_and(|s| {
|
||||
!s.nodes.is_empty()
|
||||
&& !s.nodes.iter().all(|n| n.status == NodeStatus::Achieved)
|
||||
&& !s
|
||||
.nodes
|
||||
.iter()
|
||||
.any(|n| matches!(n.status, NodeStatus::Ready | NodeStatus::Running))
|
||||
})
|
||||
}
|
||||
|
||||
/// `Active -> paused-family`; `true` if the transition happened.
|
||||
pub fn pause(&mut self, reason: GoalPauseReason) -> bool {
|
||||
self.pause_inner(reason, None)
|
||||
}
|
||||
|
||||
/// Like [`Self::pause`] but records a human-readable reason.
|
||||
pub fn pause_with_message(&mut self, reason: GoalPauseReason, message: String) -> bool {
|
||||
self.pause_inner(reason, Some(message))
|
||||
}
|
||||
|
||||
fn pause_inner(&mut self, reason: GoalPauseReason, message: Option<String>) -> bool {
|
||||
self.account_elapsed();
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return false;
|
||||
};
|
||||
if state.status != GoalStatus::Active {
|
||||
return false;
|
||||
}
|
||||
state.status = reason.to_status();
|
||||
state.pause_message = message;
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(
|
||||
GraphEvent::GraphPaused,
|
||||
state.current_node.clone(),
|
||||
Some(reason.history_detail().to_owned()),
|
||||
),
|
||||
);
|
||||
self.last_probe = None;
|
||||
true
|
||||
}
|
||||
|
||||
/// `paused-family -> Active`; `true` if the transition happened.
|
||||
/// Recomputes the ready set so a resume after restore re-arms roots.
|
||||
pub fn resume(&mut self) -> bool {
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return false;
|
||||
};
|
||||
if !state.status.is_paused() {
|
||||
return false;
|
||||
}
|
||||
state.status = GoalStatus::Active;
|
||||
state.pause_message = None;
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(GraphEvent::GraphResumed, None, None),
|
||||
);
|
||||
self.last_probe = Some(Instant::now());
|
||||
self.recompute_ready();
|
||||
true
|
||||
}
|
||||
|
||||
/// `Active -> Complete`; `true` if the transition happened.
|
||||
pub fn complete(&mut self) -> bool {
|
||||
self.account_elapsed();
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return false;
|
||||
};
|
||||
if state.status != GoalStatus::Active {
|
||||
return false;
|
||||
}
|
||||
state.status = GoalStatus::Complete;
|
||||
state.pause_message = None;
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(GraphEvent::GraphCompleted, None, None),
|
||||
);
|
||||
self.last_probe = None;
|
||||
true
|
||||
}
|
||||
|
||||
/// `Active -> BudgetLimited`; `true` if the transition happened.
|
||||
/// The in-flight node (if any) is terminally resolved to `Failed` so
|
||||
/// a budget-dead graph never persists a forever-`Running` node.
|
||||
pub fn budget_limit(&mut self) -> bool {
|
||||
self.account_elapsed();
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return false;
|
||||
};
|
||||
if state.status != GoalStatus::Active {
|
||||
return false;
|
||||
}
|
||||
let in_flight = state.current_node.clone();
|
||||
if let Some(node_id) = &in_flight
|
||||
&& let Some(node) = state.nodes.iter_mut().find(|n| n.id == *node_id)
|
||||
&& !node.status.is_terminal()
|
||||
{
|
||||
node.status = NodeStatus::Failed;
|
||||
node.failure = Some("graph token budget exhausted".to_owned());
|
||||
}
|
||||
state.current_node = None;
|
||||
state.status = GoalStatus::BudgetLimited;
|
||||
state.pause_message = None;
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(GraphEvent::BudgetExceeded, in_flight, None),
|
||||
);
|
||||
self.last_probe = None;
|
||||
true
|
||||
}
|
||||
|
||||
/// Drop all graph state (history records the clear first so a
|
||||
/// final persisted snapshot, if any, carries it).
|
||||
pub fn clear(&mut self) {
|
||||
if let Some(state) = self.state.as_mut() {
|
||||
push_history(
|
||||
state,
|
||||
GraphHistoryEntry::now(GraphEvent::GraphCleared, None, None),
|
||||
);
|
||||
}
|
||||
self.state = None;
|
||||
self.last_probe = None;
|
||||
}
|
||||
|
||||
/// Fold the wall-clock delta since the last probe into
|
||||
/// `elapsed_ms`. No-op unless `Active`.
|
||||
pub fn account_elapsed(&mut self) {
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return;
|
||||
};
|
||||
if state.status != GoalStatus::Active {
|
||||
return;
|
||||
}
|
||||
let now = Instant::now();
|
||||
if let Some(probe) = self.last_probe {
|
||||
state.elapsed_ms = state
|
||||
.elapsed_ms
|
||||
.saturating_add(now.duration_since(probe).as_millis() as u64);
|
||||
}
|
||||
self.last_probe = Some(now);
|
||||
}
|
||||
|
||||
pub fn append_history(&mut self, entry: GraphHistoryEntry) {
|
||||
if let Some(state) = self.state.as_mut() {
|
||||
push_history(state, entry);
|
||||
}
|
||||
}
|
||||
|
||||
/// Promote every `Waiting` node whose deps are all `Achieved` to
|
||||
/// `Ready`.
|
||||
pub fn recompute_ready(&mut self) {
|
||||
let Some(state) = self.state.as_mut() else {
|
||||
return;
|
||||
};
|
||||
let achieved: std::collections::HashSet<String> = state
|
||||
.nodes
|
||||
.iter()
|
||||
.filter(|n| n.status == NodeStatus::Achieved)
|
||||
.map(|n| n.id.clone())
|
||||
.collect();
|
||||
for node in &mut state.nodes {
|
||||
if node.status == NodeStatus::Waiting
|
||||
&& node.deps.iter().all(|d| achieved.contains(&d.on))
|
||||
{
|
||||
node.status = NodeStatus::Ready;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Append with the history cap (oldest dropped).
|
||||
fn push_history(state: &mut GraphOrchestration, entry: GraphHistoryEntry) {
|
||||
state.history.push(entry);
|
||||
if state.history.len() > GRAPH_HISTORY_MAX {
|
||||
let overflow = state.history.len() - GRAPH_HISTORY_MAX;
|
||||
state.history.drain(..overflow);
|
||||
}
|
||||
}
|
||||
|
||||
/// Mark every transitive dependent of `failed_id` as `Blocked` (only
|
||||
/// non-terminal nodes; an already-achieved dependent stays achieved).
|
||||
fn block_dependents(state: &mut GraphOrchestration, failed_id: &str) {
|
||||
let mut blocked: std::collections::HashSet<String> = std::collections::HashSet::new();
|
||||
blocked.insert(failed_id.to_owned());
|
||||
// Fixed-point pass; node count is small (planner-capped), so the
|
||||
// quadratic sweep is simpler than building an adjacency index.
|
||||
loop {
|
||||
let mut changed = false;
|
||||
for node in &mut state.nodes {
|
||||
if node.status.is_terminal() || blocked.contains(&node.id) {
|
||||
continue;
|
||||
}
|
||||
if node.deps.iter().any(|d| blocked.contains(&d.on)) {
|
||||
node.status = NodeStatus::Blocked;
|
||||
node.failure = Some(format!("blocked: dependency chain failed at {failed_id}"));
|
||||
blocked.insert(node.id.clone());
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
if !changed {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn node(id: &str, deps: &[&str]) -> GraphNode {
|
||||
GraphNode {
|
||||
id: id.to_owned(),
|
||||
title: id.to_owned(),
|
||||
spec: format!("do {id}"),
|
||||
deps: deps
|
||||
.iter()
|
||||
.map(|d| NodeDep {
|
||||
on: (*d).to_owned(),
|
||||
kind: DepKind::Blocks,
|
||||
})
|
||||
.collect(),
|
||||
status: NodeStatus::Waiting,
|
||||
goal_id: None,
|
||||
rounds: 0,
|
||||
tokens_used: 0,
|
||||
failure: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn tracker_with(nodes: Vec<GraphNode>) -> GraphTracker {
|
||||
let mut t = GraphTracker::new(std::env::temp_dir());
|
||||
t.create_graph(
|
||||
"g1".into(),
|
||||
"objective".into(),
|
||||
None,
|
||||
"2026-07-20T00:00:00Z".into(),
|
||||
);
|
||||
t.install_nodes(nodes);
|
||||
t
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn install_promotes_roots_to_ready_in_storage_order() {
|
||||
let t = tracker_with(vec![node("a", &[]), node("b", &["a"]), node("c", &[])]);
|
||||
assert_eq!(t.node("a").unwrap().status, NodeStatus::Ready);
|
||||
assert_eq!(t.node("b").unwrap().status, NodeStatus::Waiting);
|
||||
assert_eq!(t.node("c").unwrap().status, NodeStatus::Ready);
|
||||
// Deterministic serial rule: first Ready in storage order.
|
||||
assert_eq!(t.next_ready_node().unwrap().id, "a");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn achieved_unlocks_dependents_and_accumulates_tokens() {
|
||||
let mut t = tracker_with(vec![node("a", &[]), node("b", &["a"])]);
|
||||
t.mark_node_running("a", "goal-1".into());
|
||||
assert_eq!(t.current_node_id(), Some("a"));
|
||||
t.mark_node_achieved("a", 3, 1_000);
|
||||
assert_eq!(t.current_node_id(), None);
|
||||
assert_eq!(t.node("b").unwrap().status, NodeStatus::Ready);
|
||||
assert_eq!(t.snapshot().unwrap().tokens_spent_nodes, 1_000);
|
||||
t.mark_node_running("b", "goal-2".into());
|
||||
t.mark_node_achieved("b", 1, 500);
|
||||
assert!(t.all_achieved());
|
||||
assert_eq!(t.snapshot().unwrap().tokens_spent_nodes, 1_500);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn failed_node_blocks_transitive_dependents_only() {
|
||||
let mut t = tracker_with(vec![
|
||||
node("a", &[]),
|
||||
node("b", &["a"]),
|
||||
node("c", &["b"]),
|
||||
node("d", &[]),
|
||||
]);
|
||||
t.mark_node_running("a", "goal-1".into());
|
||||
t.mark_node_failed("a", "unachievable".into());
|
||||
assert_eq!(t.node("a").unwrap().status, NodeStatus::Failed);
|
||||
assert_eq!(t.node("b").unwrap().status, NodeStatus::Blocked);
|
||||
assert_eq!(t.node("c").unwrap().status, NodeStatus::Blocked);
|
||||
// Independent chain keeps going.
|
||||
assert_eq!(t.node("d").unwrap().status, NodeStatus::Ready);
|
||||
assert!(!t.is_wedged());
|
||||
t.mark_node_running("d", "goal-2".into());
|
||||
t.mark_node_achieved("d", 1, 10);
|
||||
assert!(t.is_wedged(), "no runnable node left, one chain dead");
|
||||
assert!(!t.all_achieved());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remaining_budget_saturates_and_tracks_node_spend() {
|
||||
let mut t = GraphTracker::new(std::env::temp_dir());
|
||||
t.create_graph("g".into(), "o".into(), Some(1_000), "t".into());
|
||||
t.install_nodes(vec![node("a", &[])]);
|
||||
assert_eq!(t.remaining_budget(), Some(1_000));
|
||||
t.mark_node_running("a", "goal-1".into());
|
||||
t.mark_node_achieved("a", 1, 1_500);
|
||||
assert_eq!(t.remaining_budget(), Some(0), "saturates at zero");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn restore_demotes_active_and_running_for_safe_resume() {
|
||||
let mut t = tracker_with(vec![node("a", &[]), node("b", &["a"])]);
|
||||
t.mark_node_running("a", "goal-1".into());
|
||||
let snapshot = t.snapshot().unwrap().clone();
|
||||
let restored = GraphTracker::from_snapshot(std::env::temp_dir(), snapshot);
|
||||
let s = restored.snapshot().unwrap();
|
||||
assert_eq!(s.status, GoalStatus::UserPaused);
|
||||
assert!(
|
||||
s.pause_message
|
||||
.as_deref()
|
||||
.unwrap()
|
||||
.contains("/graph resume")
|
||||
);
|
||||
assert_eq!(s.current_node, None);
|
||||
assert_eq!(
|
||||
restored.node("a").unwrap().status,
|
||||
NodeStatus::Ready,
|
||||
"running node re-runs, verifier gates completion"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pause_resume_round_trip_recomputes_ready() {
|
||||
let mut t = tracker_with(vec![node("a", &[]), node("b", &["a"])]);
|
||||
assert!(t.pause(GoalPauseReason::User));
|
||||
assert_eq!(t.status(), Some(GoalStatus::UserPaused));
|
||||
assert!(!t.pause(GoalPauseReason::User), "pause is Active-only");
|
||||
// Simulate an externally-restored snapshot where `a` is already
|
||||
// Achieved but `b` was never promoted: resume() must recompute.
|
||||
if let Some(s) = t.snapshot_mut() {
|
||||
s.nodes[0].status = NodeStatus::Achieved;
|
||||
}
|
||||
assert_eq!(t.node("b").unwrap().status, NodeStatus::Waiting);
|
||||
assert!(t.resume());
|
||||
assert_eq!(t.status(), Some(GoalStatus::Active));
|
||||
assert_eq!(
|
||||
t.node("b").unwrap().status,
|
||||
NodeStatus::Ready,
|
||||
"resume must promote unlocked Waiting nodes"
|
||||
);
|
||||
assert_eq!(t.next_ready_node().unwrap().id, "b");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn budget_limit_terminally_fails_the_running_node() {
|
||||
let mut t = GraphTracker::new(std::env::temp_dir());
|
||||
t.create_graph("g".into(), "o".into(), Some(10), "t".into());
|
||||
t.install_nodes(vec![node("a", &[]), node("b", &["a"])]);
|
||||
t.mark_node_running("a", "goal-1".into());
|
||||
assert!(t.budget_limit());
|
||||
assert_eq!(t.status(), Some(GoalStatus::BudgetLimited));
|
||||
let s = t.snapshot().unwrap();
|
||||
assert_eq!(
|
||||
s.nodes[0].status,
|
||||
NodeStatus::Failed,
|
||||
"a budget-dead graph must not persist a forever-Running node"
|
||||
);
|
||||
assert!(
|
||||
s.nodes[0]
|
||||
.failure
|
||||
.as_deref()
|
||||
.unwrap()
|
||||
.contains("budget exhausted")
|
||||
);
|
||||
assert_eq!(s.current_node, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_node_status_restores_as_ready() {
|
||||
assert_eq!(NodeStatus::from_wire_str("half_done"), NodeStatus::Ready);
|
||||
assert_eq!(NodeStatus::from_wire_str("achieved"), NodeStatus::Achieved);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn history_is_capped_dropping_the_oldest() {
|
||||
let mut t = tracker_with(vec![node("a", &[])]);
|
||||
for i in 0..(GRAPH_HISTORY_MAX + 10) {
|
||||
t.append_history(GraphHistoryEntry::now(
|
||||
GraphEvent::Unknown,
|
||||
None,
|
||||
Some(format!("e{i}")),
|
||||
));
|
||||
}
|
||||
let history = &t.snapshot().unwrap().history;
|
||||
assert_eq!(history.len(), GRAPH_HISTORY_MAX);
|
||||
// Setup pushed 3 entries (created/planning-started/completed) and
|
||||
// the loop 74 more; overflow 13 drops the setup entries + e0..e9,
|
||||
// so the oldest survivor is e10 and the newest is e73.
|
||||
assert_eq!(history.first().unwrap().detail.as_deref(), Some("e10"));
|
||||
assert_eq!(history.last().unwrap().detail.as_deref(), Some("e73"));
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user