//! Goal-verification stage (harness-owned). //! //! The adversarial skeptic panel is the whole verification: it //! spawns N independent skeptic subagents in parallel, //! parses each one's JSON verdict (with terminal-token fallback), and //! aggregates via majority-refute to drive `update_goal(completed: //! true)`. Each spawn sends a `SubagentEvent::Spawn` directly over //! `tool_context.subagent_event_tx` — no `task` tool call, so the //! parent model's transcript stays clean. The spawn is hidden behind //! the [`GoalClassifierSpawner`] trait so tests can inject deterministic //! responses; production uses [`ChannelSpawner`]. The struct / trait / //! constant names retain the `classifier` prefix to keep the env / //! remote / config wire contract stable across the rewire. pub(crate) mod evidence; use crate::session::events::{Event, GoalClassifierFailOpenReason}; use crate::session::goal_planner::{ GOAL_ROLE_SUBAGENT_TYPE, RoleRenderedPrompt, RoleSpawnOverride, spawn_with_fail_open_retry, }; use crate::session::goal_role_tools::RoleToolNames; use crate::session::goal_tracker::GoalClassifierVerdict; use kigi_file_utils::events::EventWriter; use std::borrow::Cow; use std::path::{Path, PathBuf}; use std::sync::Arc; use std::time::Duration; // Constants /// Default per-goal classifier run cap. A sane local default; the stall /// early-exit ([`crate::session::goal_tracker::GOAL_CLASSIFIER_STALL_THRESHOLD`]) /// is the primary, cheaper stop for stuck loops, so this cap is a /// runaway-cost backstop. There is no upper ceiling — override via /// `KIGI_GOAL_CLASSIFIER_MAX` or remote `goal_classifier_max_runs` to /// raise it arbitrarily (only the `GOAL_CLASSIFIER_MAX_RUNS_MIN` floor /// is enforced). pub(crate) const GOAL_CLASSIFIER_MAX_RUNS_DEFAULT: u32 = 10; /// Floor for `KIGI_GOAL_CLASSIFIER_MAX` / remote `goal_classifier_max_runs`. /// Floor 1 keeps the gate live (0 would disable rejection entirely). /// There is deliberately no upper ceiling so the cap can be raised /// arbitrarily via remote/env. pub(crate) const GOAL_CLASSIFIER_MAX_RUNS_MIN: u32 = 1; /// Maximum size of the embedded diff in bytes. Past this the diff is /// truncated with an explicit marker — the verifier prompt's /// diff-based rules can still operate on the head of the diff plus the /// truncation marker (and rule 5 if even the head is unavailable). pub(crate) const GOAL_CLASSIFIER_DIFF_MAX_BYTES: usize = 256 * 1024; /// Overall byte cap for the aggregated panel details file. A 3-skeptic /// panel of rich reports runs ~30-40 KB; this ceiling leaves wide /// headroom (≈5 large reports) while bounding a pathological skeptic. /// Overall cap only — never per-line. pub(crate) const GOAL_VERIFIER_PANEL_MAX_BYTES: usize = 512 * 1024; /// Template for the per-attempt details FILE NAME, rooted under the /// owner-only (0700) per-goal scratch root by `format_details_path`. /// Classifier artifacts never live in bare `/tmp`: their names are /// predictable from the prompt/log-visible `verifier_id`, so a /// world-writable directory would let a local attacker pre-plant a /// symlink and redirect the harness's writes (see /// [`super::goal_tracker::ensure_goal_scratch_root`]). pub(crate) const GOAL_CLASSIFIER_DETAILS_PATH_TEMPLATE: &str = "goal-classifier-{verifier_id}-{attempt}.md"; /// Template for the per-attempt patch FILE NAME (rooted like /// [`GOAL_CLASSIFIER_DETAILS_PATH_TEMPLATE`]). The captured diff is /// written here and each skeptic reads it via its `read_file` tool /// instead of receiving the body inline in its prompt. pub(crate) const GOAL_CLASSIFIER_CHANGES_PATH_TEMPLATE: &str = "goal-classifier-{verifier_id}-{attempt}.patch"; /// Wall-clock budget for the best-effort `git rev-parse HEAD` capture /// during goal creation. The call must NEVER block goal creation; if /// the workspace isn't a git repo or HEAD takes longer than this /// (network filesystem, etc.) we drop the baseline and surface /// `(unavailable)` to each skeptic — matching the verifier prompt's /// rule 5. const GIT_BASELINE_CAPTURE_TIMEOUT: Duration = Duration::from_secs(1); /// Subagent type used for each verifier-skeptic spawn. `general-purpose` /// gives the subagent the full read/grep/file tool inventory needed /// to corroborate diff hunks against the workspace — the verifier /// prompt explicitly forbids workspace mutation. The configured `agent_type` /// selects the HARNESS, not this subagent type. const GOAL_CLASSIFIER_SUBAGENT_TYPE: &str = GOAL_ROLE_SUBAGENT_TYPE; /// Description shown in the pager subagent strip. Kept short — the /// stage may spawn up to `GOAL_VERIFIER_SKEPTIC_MAX` skeptics per /// attempt, but a stable label reads more cleanly in the strip than /// a per-spawn suffix. const GOAL_CLASSIFIER_SUBAGENT_DESCRIPTION: &str = "goal achievement skeptic"; const GOAL_VERIFIER_PROMPT_TEMPLATE: &str = include_str!("templates/goal_verifier_prompt.md"); /// Default number of adversarial skeptics spawned per verification /// attempt. Override via `KIGI_GOAL_VERIFIER_N` (clamped 1..=5) or the /// remote `goal_verifier_count` setting. Default 3 yields a genuine /// majority vote (`⌈3/2⌉ = 2` not-refuted to pass): a lone outlier in /// either direction — one rubber-stamp or one false-refute — cannot /// decide the outcome, unlike N=2 where a 1-1 tie survives and a single /// lenient skeptic passes what a single strict one refutes. pub(crate) const GOAL_VERIFIER_SKEPTIC_COUNT: u32 = 3; /// Lower/upper bounds for `KIGI_GOAL_VERIFIER_N` / remote /// `goal_verifier_count`. Five is the practical ceiling — any more is /// pointless cost and saturates the subagent coordinator. pub(crate) const GOAL_VERIFIER_SKEPTIC_MIN: u32 = 1; pub(crate) const GOAL_VERIFIER_SKEPTIC_MAX: u32 = 5; /// Expand a skeptic `pool` to a per-index assignment of length `n` via /// round-robin (index `i` → `pool[i % pool.len()]`), reusing the frozen /// `existing` prefix verbatim. /// /// Resume stability + monotonic growth: committed indices are never /// rewritten, so skeptic-0 always keeps `pool[0]` across resume AND /// cold-fallback, and a later `n` bump only appends new indices (continuing /// the round-robin, clamped by the caller). An empty `pool` keeps `existing` /// unchanged (a frozen assignment survives a remote-cleared pool); empty /// `existing` + empty `pool` ⇒ empty (all skeptics inherit the current /// model). `n` is the CLAMPED skeptic count — identical to the value used at /// the fan-out site — so the assignment never desyncs from the spawned /// indices. pub(crate) fn expand_skeptic_assignment( existing: &[crate::util::config::GoalRoleModel], pool: &[crate::util::config::GoalRoleModel], n: usize, ) -> Vec { let mut out = existing.to_vec(); if pool.is_empty() || out.len() >= n { return out; } for i in out.len()..n { out.push(pool[i % pool.len()].clone()); } out } /// Per-skeptic JSON verdict FILE NAME template (rooted under the /// per-goal scratch root like [`GOAL_CLASSIFIER_DETAILS_PATH_TEMPLATE`]). /// The harness reads each skeptic's JSON to drive the aggregation; the /// terminal token is the fast-path signal but the JSON is authoritative. pub(crate) const GOAL_VERIFIER_VERDICT_PATH_TEMPLATE: &str = "goal-verdict-{verifier_id}-{attempt}-{skeptic_idx}.json"; /// Per-skeptic Markdown details FILE NAME template (rooted like /// [`GOAL_CLASSIFIER_DETAILS_PATH_TEMPLATE`]). Each skeptic writes its /// own analysis here; the harness concatenates them into the canonical /// `GOAL_CLASSIFIER_DETAILS_PATH_TEMPLATE` path the existing ack /// contract surfaces. pub(crate) const GOAL_VERIFIER_DETAILS_PATH_TEMPLATE: &str = "goal-classifier-{verifier_id}-{attempt}-skeptic-{skeptic_idx}.md"; // Outcome + spawner abstraction /// Result of one classifier attempt. `Achieved` / `NotAchieved` are /// PARSE-class outcomes: the subagent produced a usable verdict. /// `FailOpenAchieved` is INFRA-class: the harness could not extract /// a verdict and treats the goal as achieved so an internal failure /// never blocks user progress. PARSE-class fail-closed outcomes /// (malformed terminal token, missing details file) map onto /// `NotAchieved`; telemetry distinguishes them via /// `Event::GoalClassifierFailClosed`. #[derive(Debug, Clone)] pub(crate) enum GoalClassifierOutcome { Achieved { details_path: String, }, NotAchieved { details_path: String, /// One-line-per-refuter gist inlined into the rejection nudge so /// a weak model sees the actionable gaps without a file read (see /// [`build_gaps_summary`]). Never empty for a real rejection /// (≥1 refuter). gaps_summary: String, /// Blocker bullets grouped by [`SkepticBlocking`] class for the /// user-facing auto-pause message (see [`build_pause_summary`]). pause_summary: String, /// Stall fingerprint computed at the SOURCE from the raw /// (undecorated, log-path-free) gap evidence via /// [`gap_fingerprint`]; the drain compares it across attempts. gap_fingerprint: String, }, /// Every refuter classified its gap as a contradiction or /// environment-unverifiable blocker — no model-fixable gap remains, /// so iterating cannot help. The goal pauses for a user decision /// rather than receiving another retry nudge. No stall fingerprint /// is carried — the drain resets the streak when routing here. Blocked { details_path: String, /// Grouped blocker bullets (all non-model-fixable) used as the /// user-facing pause message. pause_summary: String, }, FailOpenAchieved { reason: GoalClassifierFailOpenReason, /// Empty when the failure happened before path resolution /// (e.g. an unsafe path was rejected by the validator). details_path: String, }, } /// Subagent spawn abstraction. Production uses [`ChannelSpawner`]; /// tests use [`MockSpawner`]. #[async_trait::async_trait] pub(crate) trait GoalClassifierSpawner: Send + Sync { /// Spawn under `id` and return the terminal response when the subagent /// finishes. `resume_from`, when `Some`, names a previously-completed /// subagent session whose transcript / tool-state / model the new /// child inherits (used to resume skeptic 0 across attempts). async fn spawn_classifier( &self, id: &str, skeptic_idx: u32, prompt: RoleRenderedPrompt, details_path: &Path, resume_from: Option<&str>, ) -> Result; } /// Spawn-time error. Distinguishes between transport errors (channel /// closed, coordinator unreachable) and runtime errors (subagent /// reported failure, was cancelled, etc.) so the runner can map them /// to the correct fail-open reason. #[derive(Debug)] pub(crate) enum SpawnError { /// Subagent coordinator was unreachable (channel closed, no /// `subagent_event_tx` plumbed). Maps to `SamplerError`. Transport(String), /// Subagent ran but reported failure. `cancelled: true` maps to /// [`GoalClassifierFailOpenReason::Aborted`]; `cancelled: false` /// maps to [`GoalClassifierFailOpenReason::SamplerError`]. Runtime { message: String, cancelled: bool }, } impl std::fmt::Display for SpawnError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { Self::Transport(d) => write!(f, "subagent transport error: {d}"), Self::Runtime { message, cancelled } => { write!( f, "subagent runtime error (cancelled={cancelled}): {message}" ) } } } } impl crate::session::goal_planner::RetryableSpawnError for SpawnError { fn is_cancelled(&self) -> bool { matches!( self, SpawnError::Runtime { cancelled: true, .. } ) } } // Path resolution + validation /// Root a substituted classifier file name under the goal's private /// scratch root. Single seam for every classifier artifact path so the /// owner-only-directory invariant cannot drift per call site. fn scratch_rooted(verifier_id: &str, file_name: String) -> String { super::goal_tracker::goal_scratch_root(verifier_id) .join(file_name) .to_string_lossy() .into_owned() } /// Substitute the `{verifier_id}` / `{attempt}` placeholders in /// `GOAL_CLASSIFIER_DETAILS_PATH_TEMPLATE` and root the result under /// the goal's scratch root. Pure string ops; no I/O. pub(crate) fn format_details_path(verifier_id: &str, attempt: u32) -> String { scratch_rooted( verifier_id, GOAL_CLASSIFIER_DETAILS_PATH_TEMPLATE .replace("{verifier_id}", verifier_id) .replace("{attempt}", &attempt.to_string()), ) } /// Substitute placeholders in `GOAL_CLASSIFIER_CHANGES_PATH_TEMPLATE` /// and root the result under the goal's scratch root. pub(crate) fn format_changes_path(verifier_id: &str, attempt: u32) -> String { scratch_rooted( verifier_id, GOAL_CLASSIFIER_CHANGES_PATH_TEMPLATE .replace("{verifier_id}", verifier_id) .replace("{attempt}", &attempt.to_string()), ) } /// Errors classifying a candidate details-file path. #[derive(Debug, PartialEq, Eq)] pub(crate) enum PathValidationError { /// Path contains `..`, a NUL byte, or starts in a forbidden /// system prefix (`/etc`, `/proc`, `/sys`, `/dev`, `~`). UnsafeComponent, /// Path contains an unresolved `${...}` / `{...}` substitution /// marker other than the known classifier placeholders. UnresolvedSubstitution, /// Resolved path is outside the platform temp dir the classifier /// roots its artifacts under. OutsideAllowedPrefix, } impl std::fmt::Display for PathValidationError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { Self::UnsafeComponent => f.write_str("path contains an unsafe component"), Self::UnresolvedSubstitution => f.write_str("path contains unresolved substitution"), Self::OutsideAllowedPrefix => f.write_str("path is outside the allowed temp root"), } } } /// Validate the resolved classifier details-file path against the /// platform temp dir (the goal scratch root's parent, where /// `format_*_path` roots every artifact). No bare-`/tmp` allowance: /// every production caller validates a freshly `format_*_path`-built /// path. See [`validate_details_path_in_root`] for the rules. pub(crate) fn validate_details_path(path: &Path) -> Result<(), PathValidationError> { validate_details_path_in_root(path, &std::env::temp_dir()) } /// Root-injectable core of [`validate_details_path`], so the /// allowed-prefix rule is unit-testable on every platform (on Linux /// `temp_dir()` IS `/tmp`). String-structural only; symlink resistance /// comes from the owner-only (0700) scratch root. pub(crate) fn validate_details_path_in_root( path: &Path, temp_root: &Path, ) -> Result<(), PathValidationError> { let s = path.to_string_lossy(); // Cheap structural checks first — these don't require any I/O. if s.contains("..") || s.contains('\0') { return Err(PathValidationError::UnsafeComponent); } for prefix in &["/etc", "/proc", "/sys", "/dev"] { if s.starts_with(prefix) { return Err(PathValidationError::UnsafeComponent); } } if s.starts_with('~') { return Err(PathValidationError::UnsafeComponent); } // Substitution markers other than the known classifier placeholders. // The runner substitutes `{verifier_id}` / `{attempt}` BEFORE // validation, so any remaining `{...}` is an error. if s.contains("${") || s.contains('{') || s.contains('}') { return Err(PathValidationError::UnresolvedSubstitution); } // Allowed prefix — the platform temp dir (on macOS this is // /var/folders/..., not /tmp). Extend this check for future // session-dir overrides without changing the failure-class taxonomy. if !path.starts_with(temp_root) { return Err(PathValidationError::OutsideAllowedPrefix); } Ok(()) } // Terminal-token parse /// Parse an adversarial skeptic's terminal response. `Refuted` /// ⇒ `Some(true)`, `Not Refuted` ⇒ `Some(false)`. The JSON verdict /// file is authoritative when present; the terminal token is the /// fast-path signal for the skeptic's vote when JSON parsing fails. /// /// Tolerates code fences/backticks and a trailing `.`/`!` around the /// token, but the response must contain ONLY the token — any other /// prose stays `None`. pub(crate) fn parse_skeptic_terminal_response(text: &str) -> Option { let lines: Vec<&str> = text .lines() .map(str::trim) // Drop fence lines entirely, including language-tagged ones // ("```text") that backtick-trimming alone would leave behind. .filter(|l| !l.starts_with("```")) .map(|l| l.trim_matches('`').trim_end_matches(['.', '!']).trim()) .filter(|l| !l.is_empty()) .collect(); match lines.as_slice() { ["Refuted"] => Some(true), ["Not Refuted"] => Some(false), _ => None, } } // Git baseline capture (called from `setup_goal`) /// Best-effort `git rev-parse HEAD` capture for goal creation. /// /// Returns the commit SHA on success; `None` for any failure /// (workspace is not a git repo, `git` is not installed, HEAD has /// no commits, the call timed out). NEVER blocks goal creation — /// the wall-clock budget is bounded by `GIT_BASELINE_CAPTURE_TIMEOUT` /// and the caller treats `None` as the documented "no baseline" /// signal (each skeptic renders `CHANGES_FILE: (unavailable)` and the /// verifier prompt's rule 5 takes over). pub(crate) async fn capture_git_baseline(workspace_root: &Path) -> Option { let mut cmd = tokio::process::Command::new(evidence::git_bin()); cmd.arg("rev-parse").arg("HEAD").current_dir(workspace_root); let output = match tokio::time::timeout(GIT_BASELINE_CAPTURE_TIMEOUT, cmd.output()).await { Ok(Ok(output)) => output, Ok(Err(err)) => { tracing::debug!( error = %err, "goal baseline capture: failed to spawn git rev-parse", ); return None; } Err(_) => { tracing::debug!("goal baseline capture: git rev-parse exceeded budget"); return None; } }; if !output.status.success() { tracing::debug!( exit = ?output.status.code(), "goal baseline capture: git rev-parse non-zero exit", ); return None; } let sha = String::from_utf8_lossy(&output.stdout).trim().to_string(); if sha.is_empty() { return None; } Some(sha) } // Trace-only recording for harness-spawned subagents // Production spawner — wraps the subagent coordinator channel /// Production spawner. Sends a `SubagentEvent::Spawn` to the session's /// coordinator and awaits the result on a fresh oneshot. The parent model /// never sees the spawn live — it is direct (no `task` tool call). pub(crate) struct ChannelSpawner { pub(crate) event_tx: tokio::sync::mpsc::UnboundedSender< kigi_tools::implementations::kigi::task::types::SubagentEvent, >, pub(crate) parent_session_id: String, pub(crate) parent_prompt_id: Option, pub(crate) cwd: Option, /// Per-skeptic-index resolved model+toolset override, indexed by /// `skeptic_idx`. An out-of-range index (or `Default`) inherits the /// current model — round-robin expansion + auth/capability fail-open is /// resolved parent-side before the spawner is built. pub(crate) skeptic_overrides: Vec, /// Event sink for the spawn-and-retry-once fail-open telemetry; `None` /// in tests / when no event log is wired. pub(crate) events: Option, } #[async_trait::async_trait] impl GoalClassifierSpawner for ChannelSpawner { async fn spawn_classifier( &self, id: &str, skeptic_idx: u32, prompt: RoleRenderedPrompt, _details_path: &Path, resume_from: Option<&str>, ) -> Result { // Per-index override; out-of-range ⇒ inherit (defensive). let inherit = RoleSpawnOverride::default(); let override_ = self .skeptic_overrides .get(skeptic_idx as usize) .unwrap_or(&inherit); spawn_with_fail_open_retry( "skeptic", Some(skeptic_idx), override_, self.events.as_ref(), prompt, |model, harness, prompt| self.send_one(id, prompt, model, harness, resume_from), ) .await } } impl ChannelSpawner { /// Send one skeptic spawn (model + harness override resolved by the caller) /// and await its terminal result. The fail-open wrapper calls this once /// or twice (retry on the current model + session harness). The /// subagent_type is always [`GOAL_CLASSIFIER_SUBAGENT_TYPE`]; /// `harness_agent_type` selects the harness flavor (`None` ⇒ session /// harness). async fn send_one( &self, id: &str, prompt: String, model: Option, harness_agent_type: Option, resume_from: Option<&str>, ) -> Result { use kigi_tools::implementations::kigi::task::types::{ SubagentEvent, SubagentRequest, SubagentRuntimeOverrides, }; let (result_tx, result_rx) = tokio::sync::oneshot::channel(); let request = SubagentRequest { id: id.to_string(), prompt, description: GOAL_CLASSIFIER_SUBAGENT_DESCRIPTION.to_string(), subagent_type: GOAL_CLASSIFIER_SUBAGENT_TYPE.to_string(), parent_session_id: self.parent_session_id.clone(), parent_prompt_id: self.parent_prompt_id.clone(), resume_from: resume_from.map(str::to_string), cwd: self.cwd.clone(), runtime_overrides: SubagentRuntimeOverrides { model, harness_agent_type, ..Default::default() }, run_in_background: false, // Harness-internal: never surface to the model's idle reminder. surface_completion: false, fork_context: false, result_tx, }; if self .event_tx .send(SubagentEvent::Spawn(Box::new(request))) .is_err() { return Err(SpawnError::Transport( "subagent coordinator channel closed".to_string(), )); } let result = result_rx .await .map_err(|_| SpawnError::Transport("subagent result channel dropped".to_string()))?; if !result.success { let message = result.error.unwrap_or_else(|| "unknown error".to_string()); return Err(SpawnError::Runtime { message, cancelled: result.cancelled, }); } Ok(result.output.to_string()) } } // Fail-open helper (shared by verification stage) /// Record a fail-open outcome: emit telemetry, write a placeholder /// details file (when the path is resolved), and return the /// `FailOpenAchieved` value. Empty `details_raw` skips the write. async fn record_fail_open( reason: GoalClassifierFailOpenReason, attempt: u32, started: std::time::Instant, emit_event: &dyn Fn(Event), details_path: Option<&Path>, details_raw: String, ) -> GoalClassifierOutcome { let latency_ms = started.elapsed().as_millis() as u64; emit_event(Event::GoalClassifierFailOpen { reason: reason.as_const_str(), attempt, latency_ms, }); let resolved_path = match details_path { // Surface the path only when the placeholder is on disk — a failed // write would point the user at a missing file (empty = no details). Some(p) if maybe_write_fail_open_placeholder(p, reason).await => details_raw, _ => String::new(), }; GoalClassifierOutcome::FailOpenAchieved { reason, details_path: resolved_path, } } /// Write `body` to `path` atomically via tempfile + rename. The /// tempfile sits next to the target so `rename` stays on one FS. async fn write_patch_file_atomic(path: &Path, body: &str) -> std::io::Result<()> { // Scratch-rooted paths always have a parent; a rootless path is a bug. let Some(dir) = path.parent() else { return Err(std::io::Error::other("patch path has no parent directory")); }; let file_name = path .file_name() .and_then(|s| s.to_str()) .unwrap_or("goal-classifier.patch"); let tmp = dir.join(format!(".{file_name}.{}.tmp", uuid::Uuid::now_v7())); tokio::fs::write(&tmp, body).await?; let dest = path.to_path_buf(); tokio::task::spawn_blocking(move || crate::util::fs::replace_file(&tmp, &dest)) .await .map_err(std::io::Error::other)??; Ok(()) } /// Write a placeholder file at `path` unless a non-empty file is /// already there. `headline` becomes the Markdown `# ` /// header; `body` is appended verbatim. Best-effort. /// /// Returns `true` when a non-empty details file exists at `path` /// afterward (it already did, or the write succeeded) and `false` when /// the write was attempted and failed — so the caller never surfaces a /// path to a file that isn't there. async fn maybe_write_classifier_placeholder(path: &Path, headline: &str, body: &str) -> bool { if let Ok(meta) = tokio::fs::metadata(path).await && meta.is_file() && meta.len() > 0 { return true; } let content = format!("# {headline}\n\n{body}\n"); match tokio::fs::write(path, content).await { Ok(()) => true, Err(err) => { tracing::warn!( path = %path.display(), error = %err, "goal classifier: failed to write placeholder", ); false } } } /// Returns `true` iff the placeholder is on disk afterward (see /// [`maybe_write_classifier_placeholder`]). async fn maybe_write_fail_open_placeholder( path: &Path, reason: GoalClassifierFailOpenReason, ) -> bool { let reason_str = reason.as_const_str(); let body = format!( "The verification stage did not produce a verdict (infra-class \ failure). The harness treated the goal as Achieved as a \ fail-open fallback. No skeptic analysis was captured.\n\n\ ## Reason\n\n{reason_str}" ); maybe_write_classifier_placeholder( path, &format!("Verification fail-open: {reason_str}"), &body, ) .await } // Verifier — the adversarial skeptic panel /// Confidence label on a skeptic verdict. The JSON wire vocabulary is /// `high|medium|low`; any other (or missing) value normalises to /// `Unknown` so a verifier with a botched JSON field still produces an /// aggregable vote. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum SkepticConfidence { High, Medium, Low, Unknown, } impl SkepticConfidence { pub(crate) fn parse(s: &str) -> Self { match s.trim().to_ascii_lowercase().as_str() { "high" => Self::High, "medium" => Self::Medium, "low" => Self::Low, _ => Self::Unknown, } } pub(crate) fn as_const_str(self) -> &'static str { match self { Self::High => "high", Self::Medium => "medium", Self::Low => "low", Self::Unknown => "unknown", } } /// Sort key for the inlined gaps summary: high-confidence refuters /// surface first (`High` → 0 … `Unknown` → 3). fn rank(self) -> u8 { match self { Self::High => 0, Self::Medium => 1, Self::Low => 2, Self::Unknown => 3, } } } /// Classification of a refutation's blocker. `None` is an ordinary /// model-fixable gap (the default — absent or unrecognised wire values /// normalise here, keeping the JSON contract back-compatible). /// `Contradiction` flags an objective/plan internal conflict; /// `Unverifiable` flags evidence that is infeasible to capture in the /// current environment. A rejection whose refuters are *all* non-`None` /// cannot progress by iterating and routes to the blocked outcome. #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] pub(crate) enum SkepticBlocking { #[default] None, Contradiction, Unverifiable, } impl SkepticBlocking { pub(crate) fn parse(s: &str) -> Self { match s.trim().to_ascii_lowercase().as_str() { "contradiction" => Self::Contradiction, "unverifiable" => Self::Unverifiable, _ => Self::None, } } fn is_blocking(self) -> bool { !matches!(self, Self::None) } } /// Parsed skeptic verdict — JSON shape mirrors the verifier prompt's /// contract. `evidence` and `details_md` are kept for the aggregated /// details file; the harness operates on `refuted` + `confidence` + /// `blocking`. /// One concise verifier finding (the implementer-facing gap list). Fields /// default to empty for weak-model robustness; an all-empty finding is /// dropped at parse time. #[derive(Debug, Clone, Default, serde::Deserialize)] pub(crate) struct Finding { /// `bug` | `gap` | `todo` (rendered verbatim after trim). #[serde(default)] pub kind: String, /// `path:line` when code-related, else a short place; may be empty. #[serde(default)] pub location: String, /// One-line description. #[serde(default)] pub detail: String, } impl Finding { fn is_empty(&self) -> bool { self.kind.trim().is_empty() && self.location.trim().is_empty() && self.detail.trim().is_empty() } } #[derive(Debug, Clone)] pub(crate) struct SkepticVerdict { pub refuted: bool, pub evidence: String, pub confidence: SkepticConfidence, pub blocking: SkepticBlocking, pub details_md: String, /// Structured findings (the implementer-facing gap list); empty when /// the verifier emitted none (then the `evidence` fallback is used). pub findings: Vec, } #[derive(Debug, Clone, serde::Deserialize)] struct SkepticVerdictRaw { #[serde(default)] refuted: Option, #[serde(default)] evidence: Option, #[serde(default)] confidence: Option, #[serde(default)] blocking: Option, #[serde(default)] details_md: Option, #[serde(default)] findings: Option>, } /// Parse the JSON body the skeptic wrote to its `{VERDICT_FILE}`. /// /// Matches the verdict schema `required: ["refuted", "evidence", /// "confidence"]`: all three are mandatory. /// A missing or empty `evidence` field rejects (`None`) — without /// evidence the rubber-stamp failure mode this contract explicitly /// closes is back open. `details_md` is optional (it's a harness-side /// extension to the schema; the aggregator prefers the on-disk /// per-skeptic report and uses this JSON field only as a fallback when /// that file is missing/empty). Extra fields are /// tolerated. The skeptic-level fallback (`run_one_skeptic`) maps any /// `None` here to a synthetic `refuted: true` vote. pub(crate) fn parse_verdict_json(body: &str) -> Option { let raw: SkepticVerdictRaw = serde_json::from_str(body.trim()).ok()?; let refuted = raw.refuted?; let evidence = raw.evidence?; if evidence.trim().is_empty() { return None; } let confidence = SkepticConfidence::parse(&raw.confidence?); let blocking = raw .blocking .as_deref() .map(SkepticBlocking::parse) .unwrap_or_default(); let findings = raw .findings .unwrap_or_default() .into_iter() .filter(|f| !f.is_empty()) .collect(); Some(SkepticVerdict { refuted, evidence, confidence, blocking, details_md: raw.details_md.unwrap_or_default(), findings, }) } /// Result of one skeptic in the panel. The `refuted` flag is the /// aggregator's input; the rest is for the details-file render. A /// malformed / missing JSON file maps to `refuted: true` (fail-closed /// at the skeptic level) per the verifier prompt's bias. #[derive(Debug, Clone)] pub(crate) struct SkepticResult { pub skeptic_idx: u32, pub refuted: bool, pub confidence: SkepticConfidence, /// Blocker classification carried over from the verdict JSON; /// `None` (default) for a model-fixable gap, a synthetic refute, or /// a terminal-token-only fallback. pub blocking: SkepticBlocking, /// Single-line `path:line` citation from the verdict JSON. Drives /// the stall fingerprint and the gaps-summary fallback when no /// structured `findings` were emitted. pub evidence: String, /// Structured findings for the implementer (preferred over `evidence` /// when non-empty). Empty on fallback / failure paths. pub findings: Vec, /// `None` on a clean parse; populated when the JSON file was /// missing/malformed or the spawn failed. pub fallback_note: Option, /// Per-skeptic spawn-to-verdict wall clock in ms. Plumbed up so /// the panel-level event can surface slow outliers even though /// emissions are batched after `join_all`. pub latency_ms: u64, } /// Substitute the per-skeptic JSON-verdict path placeholders and root /// the result under the goal's scratch root. pub(crate) fn format_verdict_path(verifier_id: &str, attempt: u32, skeptic_idx: u32) -> String { scratch_rooted( verifier_id, GOAL_VERIFIER_VERDICT_PATH_TEMPLATE .replace("{verifier_id}", verifier_id) .replace("{attempt}", &attempt.to_string()) .replace("{skeptic_idx}", &skeptic_idx.to_string()), ) } /// Substitute the per-skeptic Markdown-details path placeholders and /// root the result under the goal's scratch root. pub(crate) fn format_verifier_details_path( verifier_id: &str, attempt: u32, skeptic_idx: u32, ) -> String { scratch_rooted( verifier_id, GOAL_VERIFIER_DETAILS_PATH_TEMPLATE .replace("{verifier_id}", verifier_id) .replace("{attempt}", &attempt.to_string()) .replace("{skeptic_idx}", &skeptic_idx.to_string()), ) } /// Aggregate the panel into a quorum result. /// /// **Variant-C** — for a fan-out panel (`total > 1`), skeptic 0's /// not-refuted vote does NOT count: approval needs a STRICT MAJORITY of /// the COLD panel (`skeptic_idx >= 1`), `needed = cold_count / 2 + 1`. /// /// The required cold-approval COUNT is monotone non-decreasing in N /// (1, 2, 2, 3 for cold sizes 1..4), so more skeptics never let fewer /// independent cold judges carry approval. The tolerated-dissenter /// FRACTION still loosens with N (N=3 needs 2/2, N=4 needs 2/3) — that is /// majority voting's intended resilience to one flaky/biased skeptic, not /// a defect. A strict majority of the FULL panel (incl. skeptic 0) is /// rejected: it would force cold UNANIMITY on even N (N=4 → 3/3), making /// the panel brittle to a single bad skeptic. /// /// The bar derives from the cold-panel SIZE, not `total`: for a /// contiguous panel `cold_count = total - 1` and `cold_count/2 + 1 ≡ /// ⌈total/2⌉`, but the cold-size form stays a true majority if skeptic 0 /// is ever absent from `results` (where `⌈total/2⌉` would slip to a /// plurality). /// /// Skeptic 0 is the resumed reject-gatekeeper, so letting its not-refuted /// vote tip a borderline panel toward approval is the bias we explicitly /// avoid. Its REFUTE still counts (in `refuted_count`, the pause/gaps /// summaries, and the upstream high-confidence decisive-refute /// short-circuit). `total <= 1` — the N==1 sole judge, and the /// short-circuit case where `results` holds only skeptic 0 — keeps the /// simple all-votes rule (`needed = 1`). /// /// The adversarial bias-to-FAIL is deliberately enforced at the /// per-skeptic level — transport / cancelled / runtime / malformed /// outputs all degrade to a synthetic `refuted: true` vote in /// [`run_one_skeptic`], NOT at the aggregator. The aggregator counts /// votes; the bias lives upstream where the missing evidence is. /// /// Returns `(refuted_count, total, quorum_achieved)`. `quorum_achieved` /// is the quorum result only; the caller (`run_verification_stage`) /// AND-tightens it with `!decisive_refute` for the final outcome. pub(crate) fn aggregate_skeptic_verdicts(results: &[SkepticResult]) -> (u32, u32, bool) { let total = results.len() as u32; // Defensive empty-case: `run_verification_stage` clamps N >= 1 // before fan-out, but the function is `pub(crate)` and tests // call it directly with `&[]`. Returning `(0, 0, false)` (not // achieved) matches the "default to refuted=true if uncertain" // bias if the clamp ever regresses. if total == 0 { return (0, 0, false); } let refuted_count = results.iter().filter(|r| r.refuted).count() as u32; let (needed, not_refuted) = if total <= 1 { // Sole judge / single-result short-circuit: the lone vote decides. (1, total - refuted_count) } else { // Variant-C: strict majority of the COLD panel; skeptic 0 excluded. let cold_count = results.iter().filter(|r| r.skeptic_idx >= 1).count() as u32; let cold_not_refuted = results .iter() .filter(|r| r.skeptic_idx >= 1 && !r.refuted) .count() as u32; (cold_count / 2 + 1, cold_not_refuted) }; (refuted_count, total, not_refuted >= needed) } /// Per-evidence-line char cap for the inlined gaps summary — bounds a /// runaway verdict yet holds a full multi-point gap without cutting the /// primary finding mid-sentence. The model's reminder inlines only this /// bounded summary; the untruncated per-skeptic writeup is persisted to /// `last_classifier_details_path` for the user. Counted in `char`s, never /// bytes, so truncation can't split a codepoint. const GAPS_EVIDENCE_MAX_CHARS: usize = 800; /// Neutralize and cap a model-written evidence string before it is /// inlined into the `` rejection nudge. The skeptic's /// `evidence` is the only model-controlled text on the gaps path, so a /// verifier emitting `` or the `` tags /// could otherwise close/reopen the reminder frame; a zero-width space /// after the leading `<` breaks each literal tag while staying visually /// identical. Capped on a `char` boundary (placeholder inertness is the /// renderer's last-substitution concern, not this function's). fn sanitize_evidence(evidence: &str) -> String { neutralize_reminder_tags(cap_chars(evidence.trim(), GAPS_EVIDENCE_MAX_CHARS)) } /// Char cap for the whole multi-skeptic `{PRIOR_GAPS}` block, sized for /// 2-3 skeptics × [`GAPS_MAX_FINDINGS`] findings — the per-line /// [`GAPS_EVIDENCE_MAX_CHARS`] cap would chop later skeptics' gaps. const PRIOR_GAPS_MAX_CHARS: usize = 4_000; /// [`sanitize_evidence`]'s neutralization with the block-sized /// [`PRIOR_GAPS_MAX_CHARS`] cap, for the `{PRIOR_GAPS}` prompt slot. fn sanitize_prior_gaps(gaps: &str) -> String { neutralize_reminder_tags(cap_chars(gaps.trim(), PRIOR_GAPS_MAX_CHARS)) } /// Truncate to `max_chars` `char`s (never bytes, so a codepoint can't /// split) with an `…` suffix when capped; single pass via `char_indices`. pub(crate) fn cap_chars(text: &str, max_chars: usize) -> String { match text.char_indices().nth(max_chars) { Some((cut, _)) => { let mut s = String::with_capacity(cut + '…'.len_utf8()); s.push_str(&text[..cut]); s.push('…'); s } None => text.to_string(), } } /// Break the literal reminder-frame tags with a zero-width space so /// model-written text cannot close/reopen the `` / /// `` frames it is embedded in. pub(crate) fn neutralize_reminder_tags(text: String) -> String { text.replace("", "<\u{200b}/system-reminder>") .replace("", "<\u{200b}system-reminder>") .replace("", "<\u{200b}/goal-state>") .replace("", "<\u{200b}goal-state>") } /// Cap on findings rendered per refuter — bounds a runaway verdict while /// holding a full multi-point gap list. const GAPS_MAX_FINDINGS: usize = 12; /// Render one structured finding as `kind · location — detail`, dropping /// empty segments. Sanitized like evidence (tag-inert, char-capped). fn render_finding(f: &Finding) -> String { let kind = f.kind.trim(); let loc = f.location.trim(); let detail = f.detail.trim(); let head = if kind.is_empty() { "finding" } else { kind }; let body = match (loc.is_empty(), detail.is_empty()) { (false, false) => format!("{head} · {loc} — {detail}"), (false, true) => format!("{head} · {loc}"), (true, false) => format!("{head} — {detail}"), (true, true) => head.to_string(), }; sanitize_evidence(&body) } /// Render one refuter as a sanitized bullet. Prefers structured `findings` /// (one sub-bullet each), else `evidence`, else the synthetic `fallback_note`, /// else a bare no-evidence note. All model text is sanitized. fn render_refuter_bullet(r: &SkepticResult) -> String { let header = format!( "- [skeptic {}, {}]", r.skeptic_idx, r.confidence.as_const_str() ); if !r.findings.is_empty() { let lines: Vec = r .findings .iter() .take(GAPS_MAX_FINDINGS) .map(|f| format!(" - {}", render_finding(f))) .collect(); return format!("{header}\n{}", lines.join("\n")); } let evidence = r.evidence.trim(); if !evidence.is_empty() { format!("{header} {}", sanitize_evidence(evidence)) } else if let Some(note) = &r.fallback_note { format!( "- [skeptic {}] no verdict produced: {}", r.skeptic_idx, sanitize_evidence(note), ) } else { format!("- [skeptic {}] refuted (no evidence)", r.skeptic_idx) } } /// Refuters ordered high→low confidence (stable within a tier, so /// skeptic index breaks ties). fn refuters_by_confidence(results: &[SkepticResult]) -> Vec<&SkepticResult> { let mut refuters: Vec<&SkepticResult> = results.iter().filter(|r| r.refuted).collect(); refuters.sort_by_key(|r| r.confidence.rank()); refuters } /// Build the inlined gaps summary for the rejection nudge: one bullet /// per refuting skeptic, ordered high→low confidence. Bounded by the /// panel size. Empty only for a no-refuter panel — unreachable on the /// panel-reject path (`achieved == false` implies a refute majority). fn build_gaps_summary(results: &[SkepticResult]) -> String { refuters_by_confidence(results) .into_iter() .map(render_refuter_bullet) .collect::>() .join("\n") } /// Section headers for the auto-pause blocker summary, one per /// [`SkepticBlocking`] class. `PAUSE_GROUP_FIXABLE` is also reused by /// the synthetic-sampler cap path in `acp_session`. pub(crate) const PAUSE_GROUP_FIXABLE: &str = "Model-fixable gaps"; const PAUSE_GROUP_CONTRADICTION: &str = "Contradictions (objective/plan conflict)"; const PAUSE_GROUP_UNVERIFIABLE: &str = "Unverifiable in this environment"; /// Build the user-facing auto-pause summary: refuter bullets grouped by /// [`SkepticBlocking`] class so a paused goal tells the user which /// blockers are model-fixable versus contradictions versus /// environment-unverifiable. Empty groups are omitted; reuses /// [`render_refuter_bullet`] so sanitization stays single-sourced. fn build_pause_summary(results: &[SkepticResult]) -> String { let refuters = refuters_by_confidence(results); [ (SkepticBlocking::None, PAUSE_GROUP_FIXABLE), (SkepticBlocking::Contradiction, PAUSE_GROUP_CONTRADICTION), (SkepticBlocking::Unverifiable, PAUSE_GROUP_UNVERIFIABLE), ] .into_iter() .filter_map(|(class, header)| { let bullets: Vec = refuters .iter() .copied() .filter(|r| r.blocking == class) .map(render_refuter_bullet) .collect(); (!bullets.is_empty()).then(|| format!("{header}:\n{}", bullets.join("\n"))) }) .collect::>() .join("\n") } /// Normalized fingerprint of a rejection's *raw* gaps, used to detect a /// stuck loop (identical fingerprint across attempts). Operates on the /// undecorated evidence — never the rendered `- [skeptic N, conf]` /// bullets — so identical gaps map to one fingerprint regardless of /// skeptic ordering/confidence. Uses the deduplicated, sorted, /// lowercased `path:line` citations; with none present, falls back to /// the sorted trimmed non-empty lines. Empty input → `""`, which the /// stall guard treats as "no stable fingerprint". pub(crate) fn gap_fingerprint(raw_evidence: &[&str]) -> String { let normalized: Vec> = raw_evidence .iter() .map(|e| normalize_scratch_paths(e)) .collect(); let mut tokens: Vec = normalized .iter() .flat_map(|e| extract_path_line_tokens(e)) .collect(); if tokens.is_empty() { tokens = normalized .iter() .map(|e| e.trim().to_ascii_lowercase()) .filter(|e| !e.is_empty()) .collect(); } tokens.sort(); tokens.dedup(); tokens.join("\n") } /// Replace scratch/temp-path tokens with ``: they embed /// per-attempt ids, so leaving them in makes an identical gap /// fingerprint differently every attempt and the stall guard never /// fires. Borrowed when no scratch token is present (the common case); /// spacing collapses on the owned path — fine for a comparison-only /// fingerprint. fn normalize_scratch_paths(text: &str) -> Cow<'_, str> { const SCRATCH_MARKERS: &[&str] = &["/tmp/", "/var/folders/", "/private/tmp/"]; if !SCRATCH_MARKERS.iter().any(|m| text.contains(m)) { return Cow::Borrowed(text); } Cow::Owned( text.split_whitespace() .map(|tok| { if SCRATCH_MARKERS.iter().any(|m| tok.contains(m)) { "" } else { tok } }) .collect::>() .join(" "), ) } /// Per-refuter fingerprint source: the raw model `evidence`, or the /// `fallback_note` when a synthetic refute carries no evidence. Keeps /// repeated infra-failure rejections stable without the bullet decoration. fn refuter_fingerprint_source(r: &SkepticResult) -> &str { if r.evidence.trim().is_empty() { r.fallback_note.as_deref().unwrap_or("") } else { r.evidence.as_str() } } /// Pull `path:line` citations out of free text, lowercasing the path. A /// token qualifies when the prefix (before the FIRST colon) looks path-ish /// (contains `/` or `.`) and the first colon-segment after it is all /// digits — tolerating the `path:line:col` / trailing-colon forms common /// in compiler / test-runner output (e.g. `src/foo.rs:12:5: error`). fn extract_path_line_tokens(text: &str) -> Vec { text.split(|c: char| c.is_whitespace()) .filter_map(|raw| { let word = raw.trim_matches(|c: char| { !c.is_ascii_alphanumeric() && !matches!(c, '.' | '/' | '_' | '-' | ':') }); let (path, rest) = word.split_once(':')?; let line = rest.split(':').next().unwrap_or_default(); let path_ok = !path.is_empty() && (path.contains('/') || path.contains('.')); let line_ok = !line.is_empty() && line.chars().all(|c| c.is_ascii_digit()); (path_ok && line_ok).then(|| format!("{}:{line}", path.to_ascii_lowercase())) }) .collect() } /// The planner's `## Goal kind` tag (see `goal_planner_prompt.md`). Selects /// the kind-specific verifier review lens; an unrecognised / absent kind maps /// to `None` (no lens — the generic adversarial verifier). #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum GoalKind { CodeChange, Analysis, Research, } /// Parse the `## Goal kind` value from a plan-file body. Reads the first /// non-empty line after the header; trims backticks/whitespace/emphasis /// and normalizes space/underscore separators so a near-miss tag /// (`**code-change**`, `code change`) does not silently drop the lens. pub(crate) fn parse_goal_kind(plan: &str) -> Option { let mut lines = plan.lines(); while let Some(line) = lines.next() { if !line.trim().eq_ignore_ascii_case("## Goal kind") { continue; } for next in lines.by_ref() { let value = next.trim().trim_matches(['`', '*', '_']).trim(); if value.is_empty() { continue; } let normalized: String = value .to_ascii_lowercase() .chars() .map(|c| if c == ' ' || c == '_' { '-' } else { c }) .collect(); return match normalized.as_str() { "code-change" => Some(GoalKind::CodeChange), "analysis" => Some(GoalKind::Analysis), "research" => Some(GoalKind::Research), _ => None, }; } } None } /// `code-change` review lens — adversarial code review layered on the /// acceptance criteria, hunting real defects, test-theater, and cheating. const KIND_LENS_CODE_CHANGE: &str = "\n## Code-change review lens\n\n\ This goal changes code. Satisfying the criteria nominally is NOT enough — do a senior-engineer adversarial review of every file in CHANGED_FILES and the paths they touch. Read the CURRENT contents, run the code, and cite a `path:line` or a command/test transcript for every finding. Bias to `refuted: true`.\n\n\ Your PRIMARY mandate is to actively HUNT for real bugs, issues, and gaps in the shipped behavior — defects you can demonstrate — not to nitpick coverage. Missing coverage alone, when the code is correct and the criteria hold, is NOT a refute.\n\n\ - Correctness — reason over the whole input space (valid, invalid, empty, boundary, large, concurrent, adversarial) for any input that makes the code produce a wrong result; one such input is a decisive refute — state the input and expected-vs-actual. Illustrative, not exhaustive: off-by-one, wrong operator, inverted condition, wrong variable/index, null/empty dereference, unhandled error path, overflow/precision/sign, bad early-return, race.\n\ - Completeness — fully implement the requirement, not just the happy path. Refute when edge/error cases are silently dropped, a value is hardcoded that must be dynamic, a branch returns a placeholder, or only the demo case works.\n\ - Real tests, not theater — judge each test by whether it would catch a deliberately-broken implementation; one that still passes against a wrong implementation (asserts only on mocks/constants, sets internal state instead of using the real entry point, or has no meaningful assertion) is theater — discount it (refute if it is the only evidence for a required behavior). Injecting a fake at an environment boundary (clock, RNG, network/file/output sink) so the unit's REAL logic runs deterministically is honest dependency injection, NOT theater. A green project suite is WEAK evidence, never proof. Refute hard on tests weakened, `#[ignore]`/skipped, commented out, or whose expected values were edited to match buggy output.\n\ - End-to-end reality — build it and exercise each behavioral criterion through the REAL entry point and observed output, judging as the USER would; driving an internal flag or helper proves the mechanism exists, NOT that the wired-up feature works. A criterion whose code is present but whose integrated behavior is wrong, unreachable, or unusable is `refuted: true`, as is anything that fails to compile, fails its tests, or errors at runtime. EXCEPTION — behavior the harness cannot drive headlessly (a UI, a browser, a game loop, a long-running interactive session): the static/structural fallback is the accepted bar (the artifact is present AND the shipped unit-level functions — e.g. physics, collision, input mapping, state transitions — are exercised against the real path); this applies EVEN IF the plan did not spell the fallback out. The fallback still includes the cheap load check: a browser-loaded script must evaluate without error in a browser-like environment (`window` defined, NO Node globals) — an unguarded `module.exports`/`require` in a `