Add on_failure="succeed" as an explicit failure policy

A failed node with an effective `succeed` policy and no explicit recovery
route now finishes as `succeeded` and follows normal success routing. The
original failure stays on the outcome so the stage.completed event and the
checkpoint keep the diagnostic, and the outcome notes record which scope
promoted it.

- OnFailure gains a Succeed variant; Node::on_failure resolves the
  deprecated auto_status=true attribute as an alias, with an explicit
  on_failure winning
- The core executor applies the policy before the lifecycle observes the
  result, so the recorded outcome, context keys, goal gates, events, and
  routing all see the effective outcome; this replaces AutoStatusLifecycle
- Explicit routes take priority: a matching condition, preferred label,
  suggested next node, or handler jump keeps the outcome failed. A failed
  outcome takes an unconditional edge only under route, so under succeed
  any edge selection is an explicit route
- succeed applies only to failed, matching exit; the auto_status alias no
  longer promotes partially_succeeded
- Parallel branches promote after their retry loop, so a failed succeed
  branch counts as succeeded in the parent aggregate
- Validation accepts succeed and adds an auto_status_deprecated warning
  that suggests on_failure="succeed"
- Document the policy table, semantics, and deprecation; add a changelog
  entry

Closes #807

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Bryan Helmkamp 2026-08-26 07:34:24 -04:00
parent c90d195c2f
commit a049f94042
No known key found for this signature in database
20 changed files with 1043 additions and 83 deletions

View file

@ -0,0 +1,34 @@
---
title: "Explicit succeed failure policy"
date: "2026-08-26"
---
`on_failure` now accepts a third policy, `succeed`, alongside `route` and
`exit`. A failed node with an effective `succeed` policy and no explicit
recovery route finishes as `succeeded` and follows normal success routing.
The original failure details stay on the `stage.completed` event and in the
checkpoint, and the outcome's notes record the promotion.
Set it on a node to mark a best-effort step inside a strict graph, or on the
graph to apply it everywhere:
```dot
digraph Review {
graph [on_failure="exit"]
start [shape=Mdiamond]
exit [shape=Msquare]
required_check [script="./required-check"]
optional_scan [script="./optional-scan" on_failure="succeed"]
start -> required_check -> optional_scan -> exit
}
```
An explicit `condition="outcome=failed"` edge still takes priority over the
promotion. A promoted outcome satisfies goal gates, and a failed parallel
branch with a `succeed` policy counts as succeeded in its parent's result.
`auto_status=true` is now a deprecated alias for `on_failure="succeed"`.
Existing workflows keep working, and validation reports a new
`auto_status_deprecated` warning with the replacement. The alias no longer
promotes `partially_succeeded` outcomes; only `failed` outcomes are affected.

View file

@ -309,6 +309,7 @@
"group": "August 2026",
"icon": "clock-rotate-left",
"pages": [
"changelog/2026-08-26",
"changelog/2026-08-25",
"changelog/2026-08-23",
"changelog/2026-08-21",

View file

@ -66,6 +66,27 @@ Set `on_failure` on a node to control that node alone. The node-level attribute
The policy applies only to `failed`. Other outcomes keep their normal routing behavior. For parallel nodes, the policy uses the completed parallel node's final outcome. It does not stop or cancel individual branches early.
## Treat a failed node as succeeded
Set `on_failure="succeed"` on a best-effort node so its failure never blocks the workflow. This pairs well with a strict graph default:
```dot title="best-effort-node.fabro"
digraph Review {
graph [on_failure="exit"]
start [shape=Mdiamond]
exit [shape=Msquare]
required_check [script="./required-check"]
optional_scan [script="./optional-scan" on_failure="succeed"]
start -> required_check -> optional_scan -> exit
}
```
When `optional_scan` fails, Fabro first checks explicit recovery routes with the `failed` outcome. If none match, it rewrites the outcome to `succeeded` and routes the node as a success. Retries still run first; only the final outcome changes. The original failure stays on the `stage.completed` event and in the checkpoint, and the outcome's notes record the promotion. A promoted outcome satisfies a goal gate. Setting `on_failure="succeed"` on the graph applies it to every node.
`succeed` applies only to `failed`. It does not change a `partially_succeeded` outcome. `auto_status=true` is the deprecated spelling of this policy; validation warns and suggests `on_failure="succeed"`.
## Retry layers
Fabro retries failures at three levels: **LLM retries** handle transient API errors inside a single model call, **turn-level retries** recover from dropped streams mid-response, and **node retries** re-execute the entire node handler when the first two levels aren't enough. These layers are independent — a node retry re-runs the full handler, which gets its own fresh set of LLM and turn-level retries.
@ -332,8 +353,8 @@ A node failure does **not** automatically terminate the run. Fabro follows this
4. **Node retries** — re-execute the entire handler (per the retry policy)
5. **Direct jump** — use `jump_to_node` when the outcome supplies one
6. **Explicit edge routing** — look for a matching condition, preferred label, or suggested next node
7. **Failure policy** — with an effective `on_failure="exit"` (node-level `on_failure` first, then graph-level), skip the unconditional edge; with `route` or no attribute, keep normal fallback routing
8. **Unconditional edge** — in `route` mode, use an edge without a condition as the fallback
7. **Failure policy** — with no explicit route, apply the effective `on_failure` (node-level `on_failure` first, then graph-level): `exit` skips the unconditional edge, `succeed` promotes the outcome to `succeeded` and routes it as a success, and `route` (or no attribute) keeps normal fallback routing
8. **Unconditional edge** — in `route` mode, or after a `succeed` promotion, use an edge without a condition as the fallback
9. **Retry target** — if no edge was selected, check `retry_target` and `fallback_retry_target` on the node, then on the graph
10. **Run failure** — if none of the above produces a path forward, the run terminates

View file

@ -83,24 +83,28 @@ In this example, if the agent returns a retryable failure and all 5 standard-pol
See [Retry policies](/execution/failures#retry-policies) for the available presets and backoff settings.
## `auto_status`
## Succeed on failure
When `auto_status=true`, any non-`succeeded` and non-`skipped` outcome is silently overridden to `succeeded` after the handler completes. This is applied after the retry loop, so retries still happen normally — only the final outcome is overridden.
When a node's effective `on_failure` policy is `succeed`, a `failed` outcome with no explicit recovery route is promoted to `succeeded`. This is applied after the retry loop, so retries still happen normally — only the final outcome changes. The original failure details stay on the `stage.completed` event and in the checkpoint, and the outcome's notes record the promotion.
| Attribute | Type | Default |
|---|---|---|
| `auto_status` | Boolean | `false` |
| `on_failure` | String | inherits the graph-level `on_failure` (default `route`) |
```dot
scan [
label="Scan",
shape=parallelogram,
auto_status=true,
on_failure="succeed",
script="find . -name '*.log' | head -20"
]
```
Use `auto_status` for nodes whose failure should never block the workflow — optional scans, best-effort cleanup steps, or informational commands where the output matters more than the exit code.
Use `on_failure="succeed"` for nodes whose failure should never block the workflow — optional scans, best-effort cleanup steps, or informational commands where the output matters more than the exit code. An explicit `condition="outcome=failed"` edge still takes priority; the promotion applies only when no explicit route matches. The policy applies only to `failed` and leaves `partially_succeeded` unchanged. See [Failed-node routing policy](/workflows/transitions#failed-node-routing-policy) for the full set of policies.
<Note>
`auto_status=true` is the deprecated spelling of `on_failure="succeed"`. Fabro still accepts it as an alias, and validation reports an `auto_status_deprecated` warning with the replacement. Unlike the old attribute, the alias no longer promotes `partially_succeeded` outcomes.
</Note>
## Goal gate interaction
@ -111,7 +115,7 @@ make the workflow fail.
Nodes marked with `goal_gate=true` are checked when the workflow reaches the exit node. A goal gate is satisfied if its last outcome was `succeeded` **or** `partially_succeeded`. Any other outcome (`failed`, `skipped`) causes the workflow to fail, even though execution reached the exit.
This means `allow_partial=true` on a goal gate node lets the gate pass even if the node exhausted its retries — the promoted `partially_succeeded` outcome counts as passing.
This means `allow_partial=true` on a goal gate node lets the gate pass even if the node exhausted its retries — the promoted `partially_succeeded` outcome counts as passing. Likewise, a `succeeded` outcome promoted by `on_failure="succeed"` satisfies the gate.
See [Goal gates](/execution/failures#goal-gates) for retry target resolution and failure behavior.

View file

@ -77,7 +77,7 @@ rankdir=LR
| `rankdir` | Identifier | Layout direction: `LR` (left-to-right) or `TB` (top-to-bottom) |
| `model_stylesheet` | String | CSS-like rules for model assignment. The root value supports a MiniJinja template with `inputs` and `vars` (see [Model Stylesheets](/workflows/stylesheets)) |
| `default_max_retries` | Integer | Default retry count for all nodes (default: 0) |
| `on_failure` | String | Failed-node routing policy: `route` (default) or `exit` |
| `on_failure` | String | Failed-node policy when no explicit recovery route matches: `route` (default), `exit`, or `succeed` |
| `retry_target` | String | Default node ID to jump to on retry |
| `fallback_retry_target` | String | Fallback retry target if primary target fails |
| `default_fidelity` | String | Default [fidelity level](/execution/context) for all nodes |
@ -199,13 +199,13 @@ Other node types still need their shape, because their attributes don't identify
| `class` | String | Classes for [stylesheet](/workflows/stylesheets) targeting. Separate multiple classes with spaces. Commas are also accepted for compatibility. |
| `timeout` | Duration | Execution timeout (e.g. `900s`). An agent's wait for human input does not consume this budget. On a human node, this is the response deadline. |
| `max_visits` | Integer | Max times this node can execute in a run. Overrides the graph-level `max_node_visits` for this node. |
| `on_failure` | String | Failed-node routing policy for this node: `route` or `exit`. Overrides the graph-level `on_failure`. |
| `on_failure` | String | Failed-node policy for this node: `route`, `exit`, or `succeed`. Overrides the graph-level `on_failure`. See [Node Outcomes](/execution/outcomes#succeed-on-failure). |
| `max_retries` | Integer | Override default retry count |
| `retry_policy` | String | Named preset: `none`, `standard`, `aggressive`, `linear`, `patient` |
| `retry_target` | String | Node ID to jump to on retry |
| `fallback_retry_target` | String | Fallback node ID if primary `retry_target` is unreachable |
| `goal_gate` | Boolean | When `true`, workflow fails if this node didn't finish with `succeeded` or `partially_succeeded`. See [Node Outcomes](/execution/outcomes#goal-gate-interaction). |
| `auto_status` | Boolean | When `true`, overrides any non-`succeeded`/non-`skipped` outcome to `succeeded` after the handler completes. See [Node Outcomes](/execution/outcomes#auto_status). |
| `auto_status` | Boolean | Deprecated alias for `on_failure="succeed"`. Validation warns when it is present. |
| `allow_partial` | Boolean | When `true` and retries are exhausted on a retry-requesting failure, promotes the outcome to `partially_succeeded` instead of `failed`. Default `false`. See [Node Outcomes](/execution/outcomes#allow_partial). |
| `selection` | String | Edge tiebreaking strategy: `deterministic` (default) or `random` (weighted-random). Cannot be combined with conditional edges. |

View file

@ -13,7 +13,7 @@ When a node completes, it produces an **outcome** with a [stage outcome](/execut
2. **Condition match** — Edges with a `condition` attribute are evaluated first. If one or more conditions match, the edge with the highest `weight` wins (lexical tiebreak on target node ID).
3. **Preferred label** — If the node's outcome includes a preferred label (for example, from a human gate selection), the edge whose `label` matches is chosen.
4. **Suggested next** — If the node suggests a specific next node ID, the edge pointing to that node is chosen.
5. **Failure policy** — For a failed outcome with an effective `on_failure="exit"` policy (node-level `on_failure` first, then graph-level), Fabro skips the unconditional fallback.
5. **Failure policy** — For a failed outcome with no explicit route, the effective `on_failure` policy (node-level `on_failure` first, then graph-level) decides what happens next. `exit` skips the unconditional fallback. `succeed` promotes the outcome to `succeeded` and routes it as a success. `route` continues to the unconditional fallback.
6. **Unconditional fallback** — Edges without conditions are considered last, again using `weight` then lexical tiebreak.
7. **Retry target** — For a failed outcome with no selected edge, Fabro checks node-level and graph-level `retry_target` and `fallback_retry_target` values.
@ -21,10 +21,13 @@ If no edge or retry target supplies a next node, the workflow ends. A failed nod
## Failed-node routing policy
The `on_failure` attribute controls whether a failed node can take an unconditional edge:
The `on_failure` attribute controls what happens to a failed node when no explicit recovery route matches:
- `route` keeps the existing routing behavior. It is the default.
- `exit` stops normal fallback routing after explicit routes fail to match.
| Policy | Effective outcome | Fallback routing |
|---|---|---|
| `route` (default) | stays `failed` | takes the unconditional edge |
| `exit` | stays `failed` | skips the unconditional edge; the run ends unless a retry target applies |
| `succeed` | becomes `succeeded` | uses normal success routing |
Set it at the graph level to apply the policy to every node, or on a node to control that node alone. A node-level `on_failure` overrides the graph level. A node without the attribute inherits the graph policy.
@ -60,17 +63,38 @@ digraph Build {
}
```
The `exit` policy only applies to the `failed` outcome. It does not change routing for `succeeded`, `partially_succeeded`, or `skipped` outcomes.
Use `succeed` for a best-effort node whose failure must not block the workflow. Its failure becomes a `succeeded` outcome, so the node's normal success routing applies:
Conditioned edges, matching preferred labels, and matching suggested node IDs are explicit recovery routes. They still take priority under `exit`. An unmatched preferred label or suggested node ID does not make an unconditional edge explicit.
```dot title="best-effort-node.fabro"
digraph Review {
graph [on_failure="exit"]
Retry targets also remain available. Fabro checks them after it skips the unconditional fallback. Use graph-level `max_node_visits` or node-level `max_visits` to bound workflows whose retry targets return to a failing path.
start [shape=Mdiamond]
exit [shape=Msquare]
required_check [script="./required-check"]
optional_scan [script="./optional-scan" on_failure="succeed"]
A failed human gate never falls through to an unconditional edge, regardless of policy. Node-level `on_failure="route"` does not change that; route an interrupted gate explicitly with `condition="outcome=failed"`.
start -> required_check -> optional_scan -> exit
}
```
Under `succeed`, Fabro first checks explicit routes against the original `failed` outcome. If a `condition="outcome=failed"` edge, a matching preferred label, a matching suggested next node, or a handler jump applies, the outcome stays `failed` and that route is taken. Otherwise Fabro rewrites the outcome to `succeeded` before it records the node, so goal gates, the run context, events, and routing all see the promoted outcome. Edge selection then runs again: `condition="outcome=succeeded"` edges and unconditional edges apply. The original failure details stay on the `stage.completed` event and in the checkpoint, and the outcome's notes record which scope promoted it. A promoted outcome is not `failed`, so retry targets do not apply to it.
Both `exit` and `succeed` apply only to the `failed` outcome. They do not change routing for `succeeded`, `partially_succeeded`, or `skipped` outcomes.
Conditioned edges, matching preferred labels, and matching suggested node IDs are explicit recovery routes. They take priority under every policy. An unmatched preferred label or suggested node ID does not make an unconditional edge explicit.
Retry targets also remain available under `exit`. Fabro checks them after it skips the unconditional fallback. Use graph-level `max_node_visits` or node-level `max_visits` to bound workflows whose retry targets return to a failing path.
A failed human gate never falls through to an unconditional edge as a failure, regardless of policy. Node-level `on_failure="route"` does not change that; route an interrupted gate explicitly with `condition="outcome=failed"`. Under `succeed`, an interrupted gate with no explicit route is promoted like any other node and then follows its success routing.
When `exit` stops routing, Fabro checkpoints the failed node without a next node and ends the run as failed. It does not execute the graph's exit node or emit an edge selection for an edge it did not take. An explicit recovery route can still reach the exit node normally.
For a parallel node, `exit` sees the final outcome returned by the parallel handler. It can stop routing for a failed parallel outcome. It does not add branch-level fail-fast behavior, and a `partially_succeeded` parallel outcome continues normally.
For a parallel node, `exit` and `succeed` see the final outcome returned by the parallel handler. `exit` can stop routing for a failed parallel outcome; `succeed` promotes it. Neither adds branch-level fail-fast behavior, and a `partially_succeeded` parallel outcome continues normally. Inside the fan-out, a branch node whose effective policy is `succeed` counts as `succeeded` in the parent's aggregate when it fails. Branches have no edge routing, so there is no explicit route to prefer.
<Note>
`auto_status=true` is the deprecated spelling of node-level `on_failure="succeed"`. Fabro still accepts it as an alias and validation warns with the replacement. See [Node Outcomes](/execution/outcomes#succeed-on-failure).
</Note>
## Edge attributes
@ -201,7 +225,7 @@ gate -> fast_path [condition="outcome=succeeded"]
gate -> slow_path
```
For a failed outcome, `on_failure="exit"` skips this fallback after explicit routes are checked. The default `on_failure="route"` keeps the behavior shown above.
For a failed outcome, `on_failure="exit"` skips this fallback after explicit routes are checked, and `on_failure="succeed"` promotes the outcome to `succeeded` before taking it. The default `on_failure="route"` keeps the behavior shown above.
## Weight tiebreaking

View file

@ -390,11 +390,29 @@ fn invalid_node_on_failure_is_a_validation_failure() {
Workflow: InvalidNodeOnFailure (3 nodes, 2 edges)
Graph: [FIXTURES]/on_failure_node_invalid.fabro
error [node: work]: Node 'work' has invalid on_failure value 'stop' (on_failure_valid)
fix: Use one of: route, exit
fix: Use one of: route, exit, succeed
× Validation failed
");
}
#[test]
fn deprecated_auto_status_warns_with_succeed_policy_replacement() {
let context = test_context!();
let mut cmd = context.validate();
cmd.arg(fixture("auto_status_deprecated.fabro"));
fabro_snapshot!(context.filters(), cmd, @"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
Workflow: DeprecatedAutoStatus (3 nodes, 2 edges)
Graph: [FIXTURES]/auto_status_deprecated.fabro
warning [node: scan]: Node 'scan' sets deprecated 'auto_status=true' (auto_status_deprecated)
fix: Use on_failure=\"succeed\" instead
Validation: OK
");
}
#[test]
fn invalid_on_failure_is_a_validation_failure() {
let context = test_context!();
@ -408,7 +426,7 @@ fn invalid_on_failure_is_a_validation_failure() {
Workflow: InvalidOnFailure (2 nodes, 1 edges)
Graph: [FIXTURES]/on_failure_invalid.fabro
error: Graph has invalid on_failure value 'stop' (on_failure_valid)
fix: Use one of: route, exit
fix: Use one of: route, exit, succeed
× Validation failed
");
}

View file

@ -0,0 +1,163 @@
use fabro_graphviz::graph::Graph;
use crate::{Diagnostic, LintRule, Severity};
pub(super) fn rule() -> Box<dyn LintRule> {
Box::new(Rule)
}
/// `auto_status=true` is the deprecated spelling of `on_failure="succeed"`.
/// The runtime still honors it as an alias; this rule points workflows at the
/// explicit policy.
struct Rule;
impl LintRule for Rule {
fn name(&self) -> &'static str {
"auto_status_deprecated"
}
fn apply(&self, graph: &Graph) -> Vec<Diagnostic> {
let mut nodes: Vec<_> = graph
.nodes
.values()
.filter(|node| node.attrs.contains_key("auto_status"))
.collect();
nodes.sort_unstable_by(|a, b| a.id.cmp(&b.id));
nodes
.into_iter()
.map(|node| {
let (message, fix) = if node.attrs.contains_key("on_failure") {
(
format!(
"Node '{}' sets deprecated 'auto_status', which is ignored because \
'on_failure' is set",
node.id
),
"Remove 'auto_status'",
)
} else if node.auto_status() {
(
format!("Node '{}' sets deprecated 'auto_status=true'", node.id),
"Use on_failure=\"succeed\" instead",
)
} else {
(
format!(
"Node '{}' sets deprecated 'auto_status', which has no effect \
unless it is true",
node.id
),
"Remove 'auto_status'",
)
};
Diagnostic {
rule: self.name().to_string(),
severity: Severity::Warning,
message,
node_id: Some(node.id.clone()),
fix: Some(fix.to_string()),
..Diagnostic::default()
}
})
.collect()
}
}
#[cfg(test)]
mod tests {
use fabro_graphviz::graph::{AttrValue, Graph};
use super::Rule;
use crate::rules::test_support::{minimal_graph, node_with_attrs};
use crate::{LintRule, Severity};
fn graph_with_auto_status(value: AttrValue, on_failure: Option<&str>) -> Graph {
let mut graph = minimal_graph();
let mut node = match on_failure {
Some(policy) => node_with_attrs("work", &[("on_failure", policy)]),
None => node_with_attrs("work", &[]),
};
node.attrs.insert("auto_status".to_string(), value);
graph.nodes.insert("work".to_string(), node);
graph
}
#[test]
fn accepts_graphs_without_auto_status() {
let mut graph = minimal_graph();
graph.nodes.insert(
"work".to_string(),
node_with_attrs("work", &[("on_failure", "succeed")]),
);
assert!(Rule.apply(&graph).is_empty());
}
#[test]
fn warns_for_auto_status_true_and_suggests_succeed_policy() {
let graph = graph_with_auto_status(AttrValue::Boolean(true), None);
let diagnostics = Rule.apply(&graph);
assert_eq!(diagnostics.len(), 1);
assert_eq!(diagnostics[0].severity, Severity::Warning);
assert_eq!(
diagnostics[0].message,
"Node 'work' sets deprecated 'auto_status=true'"
);
assert_eq!(diagnostics[0].node_id.as_deref(), Some("work"));
assert_eq!(
diagnostics[0].fix.as_deref(),
Some("Use on_failure=\"succeed\" instead")
);
}
#[test]
fn warns_that_auto_status_is_ignored_when_on_failure_is_set() {
let graph = graph_with_auto_status(AttrValue::Boolean(true), Some("exit"));
let diagnostics = Rule.apply(&graph);
assert_eq!(diagnostics.len(), 1);
assert_eq!(diagnostics[0].severity, Severity::Warning);
assert_eq!(
diagnostics[0].message,
"Node 'work' sets deprecated 'auto_status', which is ignored because 'on_failure' is set"
);
assert_eq!(diagnostics[0].fix.as_deref(), Some("Remove 'auto_status'"));
}
#[test]
fn warns_for_auto_status_false() {
let graph = graph_with_auto_status(AttrValue::Boolean(false), None);
let diagnostics = Rule.apply(&graph);
assert_eq!(diagnostics.len(), 1);
assert_eq!(diagnostics[0].severity, Severity::Warning);
assert_eq!(
diagnostics[0].message,
"Node 'work' sets deprecated 'auto_status', which has no effect unless it is true"
);
assert_eq!(diagnostics[0].fix.as_deref(), Some("Remove 'auto_status'"));
}
#[test]
fn reports_nodes_in_id_order() {
let mut graph = minimal_graph();
for id in ["zeta", "alpha"] {
let mut node = node_with_attrs(id, &[]);
node.attrs
.insert("auto_status".to_string(), AttrValue::Boolean(true));
graph.nodes.insert(id.to_string(), node);
}
let ids: Vec<_> = Rule
.apply(&graph)
.into_iter()
.map(|diagnostic| diagnostic.node_id.unwrap())
.collect();
assert_eq!(ids, ["alpha", "zeta"]);
}
}

View file

@ -1,4 +1,5 @@
mod all_conditional_edges;
mod auto_status_deprecated;
mod backend_valid;
mod command_requires_script;
mod condition_syntax;
@ -65,6 +66,7 @@ pub fn built_in_rules() -> Vec<Box<dyn LintRule>> {
all_conditional_edges::rule(),
orphan_custom_outcome::rule(),
on_failure_valid::rule(),
auto_status_deprecated::rule(),
script_absolute_cd::rule(),
command_requires_script::rule(),
import_error::rule(),

View file

@ -97,7 +97,7 @@ mod tests {
let mut graph = minimal_graph();
assert!(Rule.apply(&graph).is_empty());
for value in ["route", "exit"] {
for value in ["route", "exit", "succeed"] {
graph.attrs.insert(
"on_failure".to_string(),
AttrValue::String(value.to_string()),
@ -109,7 +109,7 @@ mod tests {
#[test]
fn accepts_supported_node_values() {
let mut graph = minimal_graph();
for value in ["route", "exit"] {
for value in ["route", "exit", "succeed"] {
graph.nodes.insert(
"work".to_string(),
node_with_attrs("work", &[("on_failure", value)]),
@ -136,7 +136,7 @@ mod tests {
);
assert_eq!(
diagnostics[0].fix.as_deref(),
Some("Use one of: route, exit")
Some("Use one of: route, exit, succeed")
);
}
@ -176,7 +176,7 @@ mod tests {
assert_eq!(diagnostics[0].node_id.as_deref(), Some("work"));
assert_eq!(
diagnostics[0].fix.as_deref(),
Some("Use one of: route, exit")
Some("Use one of: route, exit, succeed")
);
}

View file

@ -75,7 +75,10 @@ pub(crate) fn select_edge<'a>(
}
}
if outcome.status.is_failure() && graph.resolve_on_failure(node).policy() == OnFailure::Exit {
// A failed outcome takes an unconditional edge only under `route`. Under
// `exit` the run stops here; under `succeed` the executor promotes the
// outcome to `succeeded` and routes it again.
if outcome.status.is_failure() && graph.resolve_on_failure(node).policy() != OnFailure::Route {
return None;
}
@ -531,6 +534,81 @@ mod tests {
);
}
#[test]
fn succeed_policy_blocks_unconditional_edge_for_failed_outcome() {
let mut graph = make_graph_with_edges(vec![Edge::new("a", "b")]);
set_on_failure(&mut graph, OnFailure::Succeed);
let node = graph.nodes.get("a").unwrap();
let outcome = Outcome::fail_classify("boom");
assert!(select_edge(node, &outcome, &Context::new(), &graph, "deterministic").is_none());
}
#[test]
fn succeed_policy_allows_matching_failure_condition() {
let mut recovery = Edge::new("a", "recover");
recovery.attrs.insert(
"condition".to_string(),
AttrValue::String("outcome=failed".to_string()),
);
let mut graph = make_graph_with_edges(vec![recovery, Edge::new("a", "fallback")]);
set_on_failure(&mut graph, OnFailure::Succeed);
let node = graph.nodes.get("a").unwrap();
let outcome = Outcome::fail_classify("boom");
let selected =
select_edge(node, &outcome, &Context::new(), &graph, "deterministic").unwrap();
assert_eq!(selected.edge.to, "recover");
assert_eq!(selected.reason, "condition");
}
#[test]
fn succeed_policy_routes_promoted_outcome_as_succeeded() {
let mut on_success = Edge::new("a", "next");
on_success.attrs.insert(
"condition".to_string(),
AttrValue::String("outcome=succeeded".to_string()),
);
let mut graph = make_graph_with_edges(vec![on_success, Edge::new("a", "fallback")]);
set_on_failure(&mut graph, OnFailure::Succeed);
let node = graph.nodes.get("a").unwrap();
let mut outcome = Outcome::fail_classify("boom");
outcome.promote_to_succeeded(graph.resolve_on_failure(node));
let selected =
select_edge(node, &outcome, &Context::new(), &graph, "deterministic").unwrap();
assert_eq!(selected.edge.to, "next");
assert_eq!(selected.reason, "condition");
}
#[test]
fn node_succeed_overrides_graph_route_and_blocks_unconditional_edge() {
let mut graph = make_graph_with_edges(vec![Edge::new("a", "b")]);
set_node_on_failure(&mut graph, "a", "succeed");
let node = graph.nodes.get("a").unwrap();
let outcome = Outcome::fail_classify("boom");
assert!(select_edge(node, &outcome, &Context::new(), &graph, "deterministic").is_none());
}
#[test]
fn auto_status_alias_resolves_to_succeed_policy() {
let mut graph = make_graph_with_edges(vec![Edge::new("a", "b")]);
graph
.nodes
.get_mut("a")
.unwrap()
.attrs
.insert("auto_status".to_string(), AttrValue::Boolean(true));
let node = graph.nodes.get("a").unwrap();
assert_eq!(graph.resolve_on_failure(node).policy(), OnFailure::Succeed);
let outcome = Outcome::fail_classify("boom");
assert!(select_edge(node, &outcome, &Context::new(), &graph, "deterministic").is_none());
}
#[test]
fn node_exit_overrides_graph_route_and_blocks_unconditional_edge() {
for graph_policy in [None, Some(OnFailure::Route)] {

View file

@ -7,7 +7,7 @@ use async_trait::async_trait;
use fabro_core::error::Error as CoreError;
use fabro_graphviz::graph::{AttrValue, Graph, Node, is_llm_handler_type};
use fabro_hooks::{HookContext, HookEvent};
use fabro_types::{ParallelBranchId, ParallelBranchResult, StageId, StageOutcome};
use fabro_types::{OnFailure, ParallelBranchId, ParallelBranchResult, StageId, StageOutcome};
use fabro_util::text;
use futures::FutureExt;
use tokio::sync::{Semaphore, SemaphorePermit};
@ -528,7 +528,7 @@ async fn run_branches(
);
let mut attempt = 0_u32;
let outcome = loop {
let mut outcome = loop {
attempt = attempt.saturating_add(1);
let attempt_result = node_handler::execute_single_attempt(
&target,
@ -569,6 +569,17 @@ async fn run_branches(
backoff_or_cancel(delay, &branch_services).await?;
permit = acquire_branch_permit(&semaphore, &branch_services).await?;
};
// Branches have no edge routing, so `succeed` has no
// explicit recovery route to defer to: a failed branch
// under that policy always counts as succeeded in the
// parent's aggregate, with its failure kept on the
// outcome.
if outcome.status.is_failure() {
let policy = graph.resolve_on_failure(&target);
if policy.policy() == OnFailure::Succeed {
outcome.promote_to_succeeded(policy);
}
}
let context_updates = branch_context_updates(
&parent_snapshot,
@ -1401,6 +1412,122 @@ mod tests {
}
}
/// Fails the named branch node and succeeds everywhere else.
struct FailNamedBranchHandler(&'static str);
#[async_trait]
impl Handler for FailNamedBranchHandler {
async fn execute(
&self,
node: &Node,
_context: &Context,
_graph: &Graph,
_run_dir: &Path,
_services: &EngineServices,
) -> Result<Outcome, Error> {
if node.id == self.0 {
Ok(Outcome::fail_classify("branch boom"))
} else {
Ok(Outcome::success())
}
}
}
/// Mutates a test graph to opt a branch into the `succeed` policy.
type PolicyEdit = fn(&mut Graph);
async fn run_parallel_with_failing_branch_a(graph: &Graph, node: &Node) -> Outcome {
let mut services = make_services();
services.registry = Arc::new(super::super::HandlerRegistry::new(Box::new(
FailNamedBranchHandler("branch_a"),
)));
ParallelHandler
.execute(
node,
&test_context(),
graph,
Path::new("/tmp/test"),
&services,
)
.await
.unwrap()
}
#[tokio::test]
async fn parallel_handler_failed_branch_without_policy_is_partial() {
let (node, graph) = parallel_graph();
let outcome = run_parallel_with_failing_branch_a(&graph, &node).await;
assert_eq!(outcome.status, StageOutcome::PartiallySucceeded);
let results: Vec<ParallelBranchResult> =
serde_json::from_value(outcome.context_updates[keys::PARALLEL_RESULTS].clone())
.unwrap();
assert!(results[0].status.is_failure());
assert_eq!(results[1].status, StageOutcome::Succeeded);
}
#[tokio::test]
async fn parallel_handler_succeed_policy_counts_failed_branch_as_succeeded() {
let cases: [(&str, PolicyEdit); 3] = [
("node", |graph| {
graph.nodes.get_mut("branch_a").unwrap().attrs.insert(
"on_failure".to_string(),
AttrValue::String("succeed".to_string()),
);
}),
("graph", |graph| {
graph.attrs.insert(
"on_failure".to_string(),
AttrValue::String("succeed".to_string()),
);
}),
("alias", |graph| {
graph
.nodes
.get_mut("branch_a")
.unwrap()
.attrs
.insert("auto_status".to_string(), AttrValue::Boolean(true));
}),
];
for (scope, apply) in cases {
let (node, mut graph) = parallel_graph();
apply(&mut graph);
let outcome = run_parallel_with_failing_branch_a(&graph, &node).await;
assert_eq!(outcome.status, StageOutcome::Succeeded, "scope {scope}");
assert_eq!(
outcome.notes.as_deref(),
Some("Parallel node dispatched 2 branches (2 succeeded, 0 failed)"),
"scope {scope}"
);
let results: Vec<ParallelBranchResult> =
serde_json::from_value(outcome.context_updates[keys::PARALLEL_RESULTS].clone())
.unwrap();
assert!(
results
.iter()
.all(|result| result.status == StageOutcome::Succeeded),
"scope {scope}"
);
}
}
#[tokio::test]
async fn parallel_handler_exit_policy_does_not_change_branch_outcomes() {
let (node, mut graph) = parallel_graph();
graph.attrs.insert(
"on_failure".to_string(),
AttrValue::String("exit".to_string()),
);
let outcome = run_parallel_with_failing_branch_a(&graph, &node).await;
assert_eq!(outcome.status, StageOutcome::PartiallySucceeded);
}
#[tokio::test]
async fn parallel_handler_no_branches() {
let outcome = ParallelHandler

View file

@ -1,37 +0,0 @@
use async_trait::async_trait;
use fabro_core::error::Result as CoreResult;
use fabro_core::lifecycle::RunLifecycle;
use fabro_core::outcome::NodeResult;
use fabro_core::state::ExecutionState;
use crate::graph::{WorkflowGraph, WorkflowNode};
use crate::outcome::{BilledModelUsage, StageOutcome};
type WfRunState = ExecutionState<Option<BilledModelUsage>>;
type WfNodeResult = NodeResult<Option<BilledModelUsage>>;
/// Sub-lifecycle responsible for auto-status override on nodes with
/// `auto_status=true`.
pub(crate) struct AutoStatusLifecycle;
#[async_trait]
impl RunLifecycle<WorkflowGraph> for AutoStatusLifecycle {
async fn after_node(
&self,
node: &WorkflowNode,
result: &mut WfNodeResult,
_state: &WfRunState,
) -> CoreResult<()> {
let gv = node.inner();
let outcome = &mut result.outcome;
if gv.auto_status()
&& outcome.status != StageOutcome::Succeeded
&& outcome.status != StageOutcome::Skipped
{
outcome.status = StageOutcome::Succeeded;
outcome.notes =
Some("auto-status: handler completed without writing status".to_string());
}
Ok(())
}
}

View file

@ -1,5 +1,4 @@
pub(crate) mod artifact;
pub(crate) mod auto_status;
pub(crate) mod circuit_breaker;
pub(crate) mod event;
pub(crate) mod fidelity;
@ -26,7 +25,6 @@ use fabro_sandbox::Sandbox;
use fabro_types::RunId;
use self::artifact::ArtifactLifecycle;
use self::auto_status::AutoStatusLifecycle;
use self::circuit_breaker::CircuitBreakerLifecycle;
use self::event::EventLifecycle;
use self::fidelity::FidelityLifecycle;
@ -56,7 +54,6 @@ pub(crate) struct WorkflowLifecycle {
event: EventLifecycle,
hook: HookLifecycle,
fidelity: FidelityLifecycle,
auto_status: AutoStatusLifecycle,
circuit_breaker: Arc<CircuitBreakerLifecycle>,
git: GitLifecycle,
artifact: ArtifactLifecycle,
@ -187,7 +184,6 @@ impl WorkflowLifecycle {
event,
hook,
fidelity,
auto_status: AutoStatusLifecycle,
circuit_breaker,
git,
artifact,
@ -361,7 +357,6 @@ impl RunLifecycle<WorkflowGraph> for WorkflowLifecycle {
result: &mut WfNodeResult,
state: &WfRunState,
) -> CoreResult<()> {
self.auto_status.after_node(node, result, state).await?;
self.circuit_breaker.after_node(node, result, state).await?;
self.artifact.after_node(node, result, state).await?;
self.event.after_node(node, result, state).await?;

View file

@ -1319,6 +1319,108 @@ async fn node_on_failure_route_overrides_graph_exit_policy() {
assert_eq!(*run.visits.lock().unwrap(), vec!["work", "downstream"]);
}
#[tokio::test]
async fn node_on_failure_succeed_promotes_failed_node_and_keeps_failure_in_events() {
let graph = on_failure_graph(
r#"graph [on_failure="exit"]
work [on_failure="succeed"]"#,
);
let emitter = Emitter::default();
let events = collect_events(&emitter);
let run = run_on_failure(&graph, emitter).await;
assert_eq!(run.outcome.status, StageOutcome::Succeeded);
assert_eq!(*run.visits.lock().unwrap(), vec!["work", "downstream"]);
let checkpoint = run
.state
.current_checkpoint()
.expect("downstream should be checkpointed");
let work = &checkpoint.node_outcomes["work"];
assert_eq!(work.status, StageOutcome::Succeeded);
assert_eq!(work.failure_reason(), Some("forced work failure"));
assert_eq!(
work.notes.as_deref(),
Some("node on_failure=succeed promoted a failed outcome to succeeded")
);
let events = events.lock().unwrap();
let completed = events
.iter()
.find_map(|event| match &event.body {
EventBody::StageCompleted(props) if event.node_id.as_deref() == Some("work") => {
Some(props.clone())
}
_ => None,
})
.expect("promoted work stage should emit stage.completed");
assert_eq!(completed.status, StageOutcome::Succeeded);
assert_eq!(
completed
.failure
.as_ref()
.map(|failure| failure.message.as_str()),
Some("forced work failure")
);
assert!(!events.iter().any(|event| {
matches!(&event.body, EventBody::StageFailed(_)) && event.node_id.as_deref() == Some("work")
}));
assert!(
events
.iter()
.any(|event| matches!(&event.body, EventBody::RunCompleted(_)))
);
}
#[tokio::test]
async fn auto_status_true_is_an_alias_for_on_failure_succeed() {
let graph = on_failure_graph(
r#"graph [on_failure="exit"]
work [auto_status=true]"#,
);
let run = run_on_failure(&graph, Emitter::default()).await;
assert_eq!(run.outcome.status, StageOutcome::Succeeded);
assert_eq!(*run.visits.lock().unwrap(), vec!["work", "downstream"]);
let checkpoint = run
.state
.current_checkpoint()
.expect("downstream should be checkpointed");
assert_eq!(
checkpoint.node_outcomes["work"].status,
StageOutcome::Succeeded
);
}
#[tokio::test]
async fn node_on_failure_succeed_prefers_explicit_failure_edge() {
let graph = on_failure_graph(
r#"work [on_failure="succeed"]
recovery
work -> recovery [condition="outcome=failed"]
recovery -> exit"#,
);
let run = run_on_failure(&graph, Emitter::default()).await;
assert_eq!(run.outcome.status, StageOutcome::Succeeded);
assert_eq!(*run.visits.lock().unwrap(), vec!["work", "recovery"]);
let checkpoint = run
.state
.current_checkpoint()
.expect("recovery should be checkpointed");
assert!(checkpoint.node_outcomes["work"].status.is_failure());
}
#[tokio::test]
async fn node_on_failure_succeed_satisfies_goal_gate_without_retry_target() {
let graph =
on_failure_graph(r#"work [on_failure="succeed" goal_gate=true retry_target="start"]"#);
let run = run_on_failure(&graph, Emitter::default()).await;
assert_eq!(run.outcome.status, StageOutcome::Succeeded);
assert_eq!(*run.visits.lock().unwrap(), vec!["work", "downstream"]);
}
#[tokio::test]
async fn goal_gate_routes_to_retry_target_on_failure() {
// Pipeline:

View file

@ -243,6 +243,8 @@ impl<G: Graph + 'static> Executor<G> {
}
Err(err) => return Err(err),
};
self.apply_succeed_policy(&node, &mut result, &state, graph)
.await?;
self.lifecycle
.after_node(&node, &mut result, &state)
.await?;
@ -285,7 +287,10 @@ impl<G: Graph + 'static> Executor<G> {
if outcome.status.is_failure() {
let resolved = graph.resolve_on_failure(&node);
let message = match resolved.policy() {
OnFailure::Route => {
// A failed outcome under `succeed` only reaches
// the end when an explicit route matched but
// produced no next node, which mirrors `route`.
OnFailure::Route | OnFailure::Succeed => {
format!("stage {} failed with no outgoing fail edge", node.id())
}
OnFailure::Exit => format!(
@ -423,6 +428,46 @@ impl<G: Graph + 'static> Executor<G> {
unreachable!("loop always returns or continues")
}
/// Applies the `on_failure="succeed"` policy to a failed node result.
///
/// This runs before the lifecycle observes the result, so the recorded
/// outcome, context keys, goal gates, events, and routing all see the
/// effective outcome. Explicit recovery routes take priority: a failed
/// outcome that carries a jump, or that an explicit edge would route,
/// stays `failed`. Under `succeed`, `select_edge` never returns an
/// unconditional edge for a failed outcome, so any selection here is an
/// explicit route.
async fn apply_succeed_policy(
&self,
node: &G::Node,
result: &mut NodeResult<G::Meta>,
state: &ExecutionState<G::Meta>,
graph: &G,
) -> Result<()> {
let outcome = &result.outcome;
if !outcome.status.is_failure() || outcome.jump_to_node.is_some() {
return Ok(());
}
let resolved = graph.resolve_on_failure(node);
if resolved.policy() != OnFailure::Succeed {
return Ok(());
}
let routing_context = self
.handler
.context_for_edge_selection(&state.context, graph)
.await?;
if graph.select_edge(node, outcome, &routing_context).is_some() {
return Ok(());
}
tracing::debug!(
node = %node.id(),
scope = %resolved.scope(),
"on_failure=succeed promoted failed outcome"
);
result.outcome.promote_to_succeeded(resolved);
Ok(())
}
async fn resolve_next_step(
&self,
node: &G::Node,
@ -2324,6 +2369,277 @@ mod tests {
assert!(state.node_outcomes.contains_key("downstream"));
}
/// Records the outcome status each lifecycle callback observed, so tests
/// can prove the `succeed` policy is applied before `after_node`.
struct StatusCaptureLifecycle(Arc<Mutex<Vec<(String, StageOutcome)>>>);
#[async_trait]
impl RunLifecycle<TestGraph> for StatusCaptureLifecycle {
async fn after_node(
&self,
node: &TestNode,
result: &mut NodeResult,
_state: &ExecutionState,
) -> Result<()> {
self.0
.lock()
.unwrap()
.push((node.id().to_string(), result.outcome.status));
Ok(())
}
}
fn succeed_policy_graph() -> TestGraph {
TestGraph::new(
vec![
TestNode::new("work").with_on_failure(OnFailure::Succeed),
TestNode::new("downstream"),
TestNode::terminal("end"),
],
vec![
TestEdge::new("work", "downstream"),
TestEdge::new("downstream", "end"),
],
"work",
)
}
fn fail_work_handler() -> Arc<dyn NodeHandler<TestGraph>> {
Arc::new(
DispatchHandler::new(Arc::new(AlwaysSucceedHandler))
.with_handler("work", Arc::new(AlwaysFailHandler::new("boom"))),
)
}
#[tokio::test]
async fn executor_succeed_policy_promotes_failed_node_before_lifecycle_and_continues() {
let graph = succeed_policy_graph();
let state = ExecutionState::new(&graph).unwrap();
let seen = Arc::new(Mutex::new(Vec::new()));
let executor = ExecutorBuilder::new(fail_work_handler())
.lifecycle(Box::new(StatusCaptureLifecycle(Arc::clone(&seen))))
.build();
let (outcome, state) = executor.run(&graph, state).await.unwrap();
assert_eq!(outcome.status, StageOutcome::Succeeded);
let work = &state.node_outcomes["work"];
assert_eq!(work.status, StageOutcome::Succeeded);
assert_eq!(
work.failure
.as_ref()
.map(|failure| failure.message.as_str()),
Some("boom"),
"the original failure stays on the recorded outcome"
);
assert_eq!(
work.notes.as_deref(),
Some("node on_failure=succeed promoted a failed outcome to succeeded")
);
assert!(state.node_outcomes.contains_key("downstream"));
assert_eq!(
seen.lock().unwrap().clone(),
vec![
("work".to_string(), StageOutcome::Succeeded),
("downstream".to_string(), StageOutcome::Succeeded),
],
"after_node observes the effective outcome"
);
}
#[tokio::test]
async fn executor_succeed_policy_keeps_failed_outcome_when_explicit_route_matches() {
let graph = TestGraph::new(
vec![
TestNode::new("work").with_on_failure(OnFailure::Succeed),
TestNode::new("recovery"),
TestNode::new("downstream"),
TestNode::terminal("end"),
],
vec![
TestEdge::new("work", "recovery").with_label("failed"),
TestEdge::new("work", "downstream"),
TestEdge::new("recovery", "end"),
TestEdge::new("downstream", "end"),
],
"work",
);
let state = ExecutionState::new(&graph).unwrap();
let executor = ExecutorBuilder::new(fail_work_handler()).build();
let (outcome, state) = executor.run(&graph, state).await.unwrap();
assert_eq!(outcome.status, StageOutcome::Succeeded);
assert_eq!(state.node_outcomes["work"].status, StageOutcome::Failed {
retry_requested: false,
});
assert!(state.node_outcomes.contains_key("recovery"));
assert!(!state.node_outcomes.contains_key("downstream"));
}
#[tokio::test]
async fn executor_succeed_policy_keeps_failed_outcome_with_jump() {
struct FailWithJump;
#[async_trait]
impl NodeHandler<TestGraph> for FailWithJump {
async fn execute(
&self,
_node: &TestNode,
_context: &Context,
_graph: &TestGraph,
) -> Result<Outcome> {
let mut outcome = Outcome::fail("boom");
outcome.jump_to_node = Some("recovery".to_string());
Ok(outcome)
}
}
let graph = TestGraph::new(
vec![
TestNode::new("work").with_on_failure(OnFailure::Succeed),
TestNode::new("recovery"),
TestNode::new("downstream"),
TestNode::terminal("end"),
],
vec![
TestEdge::new("work", "downstream"),
TestEdge::new("recovery", "end"),
TestEdge::new("downstream", "end"),
],
"work",
);
let state = ExecutionState::new(&graph).unwrap();
let handler = DispatchHandler::new(Arc::new(AlwaysSucceedHandler))
.with_handler("work", Arc::new(FailWithJump));
let executor =
ExecutorBuilder::new(Arc::new(handler) as Arc<dyn NodeHandler<TestGraph>>).build();
let (outcome, state) = executor.run(&graph, state).await.unwrap();
assert_eq!(outcome.status, StageOutcome::Succeeded);
assert!(state.node_outcomes["work"].status.is_failure());
assert!(state.node_outcomes.contains_key("recovery"));
assert!(!state.node_outcomes.contains_key("downstream"));
}
#[tokio::test]
async fn executor_succeed_policy_satisfies_goal_gate() {
let graph = TestGraph::new(
vec![
TestNode::new("work").with_on_failure(OnFailure::Succeed),
TestNode::terminal("end").with_goal_gate("work", StageOutcome::Succeeded),
],
vec![TestEdge::new("work", "end")],
"work",
);
let state = ExecutionState::new(&graph).unwrap();
let executor = ExecutorBuilder::new(fail_work_handler()).build();
let (outcome, _state) = executor.run(&graph, state).await.unwrap();
assert_eq!(outcome.status, StageOutcome::Succeeded);
}
#[tokio::test]
async fn executor_succeed_policy_skips_retry_target() {
let graph = TestGraph::new(
vec![
TestNode::new("work").with_on_failure(OnFailure::Succeed),
TestNode::new("downstream"),
TestNode::new("retry_only"),
TestNode::terminal("end"),
],
vec![
TestEdge::new("work", "downstream"),
TestEdge::new("downstream", "end"),
TestEdge::new("retry_only", "end"),
],
"work",
)
.with_retry_target("work", "retry_only");
let state = ExecutionState::new(&graph).unwrap();
let executor = ExecutorBuilder::new(fail_work_handler()).build();
let (outcome, state) = executor.run(&graph, state).await.unwrap();
assert_eq!(outcome.status, StageOutcome::Succeeded);
assert!(state.node_outcomes.contains_key("downstream"));
assert!(
!state.node_outcomes.contains_key("retry_only"),
"a promoted outcome is not failed, so retry targets do not apply"
);
}
#[tokio::test]
async fn executor_graph_succeed_policy_promotes_every_failed_node() {
let graph = TestGraph::new(
vec![
TestNode::new("work"),
TestNode::new("downstream"),
TestNode::terminal("end"),
],
vec![
TestEdge::new("work", "downstream"),
TestEdge::new("downstream", "end"),
],
"work",
)
.with_on_failure(OnFailure::Succeed);
let state = ExecutionState::new(&graph).unwrap();
let executor = ExecutorBuilder::new(
Arc::new(AlwaysFailHandler::new("boom")) as Arc<dyn NodeHandler<TestGraph>>
)
.build();
let (outcome, state) = executor.run(&graph, state).await.unwrap();
assert_eq!(outcome.status, StageOutcome::Succeeded);
for node_id in ["work", "downstream"] {
let recorded = &state.node_outcomes[node_id];
assert_eq!(recorded.status, StageOutcome::Succeeded);
assert_eq!(
recorded.notes.as_deref(),
Some("graph on_failure=succeed promoted a failed outcome to succeeded")
);
}
}
#[tokio::test]
async fn executor_succeed_policy_leaves_partial_outcome_unchanged() {
struct PartialHandler;
#[async_trait]
impl NodeHandler<TestGraph> for PartialHandler {
async fn execute(
&self,
_node: &TestNode,
_context: &Context,
_graph: &TestGraph,
) -> Result<Outcome> {
let mut outcome = Outcome::success();
outcome.status = StageOutcome::PartiallySucceeded;
Ok(outcome)
}
}
let graph = succeed_policy_graph();
let state = ExecutionState::new(&graph).unwrap();
let handler = DispatchHandler::new(Arc::new(AlwaysSucceedHandler))
.with_handler("work", Arc::new(PartialHandler));
let executor =
ExecutorBuilder::new(Arc::new(handler) as Arc<dyn NodeHandler<TestGraph>>).build();
let (outcome, state) = executor.run(&graph, state).await.unwrap();
assert_eq!(outcome.status, StageOutcome::Succeeded);
assert_eq!(
state.node_outcomes["work"].status,
StageOutcome::PartiallySucceeded
);
assert_eq!(state.node_outcomes["work"].notes, None);
}
#[tokio::test]
async fn executor_goal_gate_retry_target_to_terminal_fails_without_looping() {
let terminal_visits = Arc::new(AtomicU32::new(0));

View file

@ -226,7 +226,8 @@ impl Graph for TestGraph {
}
}
if outcome.status.is_failure() && self.resolve_on_failure(node).policy() == OnFailure::Exit
// A failed outcome takes an unconditional edge only under `route`.
if outcome.status.is_failure() && self.resolve_on_failure(node).policy() != OnFailure::Route
{
return None;
}

View file

@ -6,7 +6,11 @@ use strum::VariantNames;
use crate::AgentBackend;
/// Policy for routing a failed node when no explicit recovery edge matches.
/// Policy for a failed node when no explicit recovery route matches.
///
/// Explicit routes (a jump, a matching edge condition, a matching preferred
/// label, or a matching suggested next node) take priority under every
/// policy. The policy decides what happens when none of them match.
#[derive(
Debug,
Clone,
@ -24,9 +28,15 @@ use crate::AgentBackend;
#[serde(rename_all = "snake_case")]
#[strum(serialize_all = "snake_case")]
pub enum OnFailure {
/// The outcome stays `failed` and may take an unconditional edge.
#[default]
Route,
/// The outcome stays `failed` and skips the unconditional edge, so the
/// run ends unless a retry target applies.
Exit,
/// The outcome becomes `succeeded` and follows normal success routing.
/// The original failure details stay on the outcome for observability.
Succeed,
}
impl OnFailure {
@ -349,13 +359,17 @@ impl Node {
self.str_attr("fallback_retry_target")
}
/// Node-level failure routing policy override. `None` means the node
/// inherits the graph-level policy. Invalid values are rejected during
/// workflow validation, so runtime resolution treats them as absent.
/// Node-level failure policy override. `None` means the node inherits
/// the graph-level policy. Invalid values are rejected during workflow
/// validation, so runtime resolution treats them as absent.
///
/// The deprecated `auto_status=true` attribute is a compatibility alias
/// for `on_failure="succeed"`. An explicit `on_failure` attribute wins.
#[must_use]
pub fn on_failure(&self) -> Option<OnFailure> {
self.str_attr("on_failure")
.and_then(|value| value.parse().ok())
.or_else(|| self.auto_status().then_some(OnFailure::Succeed))
}
#[must_use]
@ -392,6 +406,8 @@ impl Node {
self.str_attr("speed")
}
/// Deprecated spelling of `on_failure="succeed"`. Validation warns when
/// it is present; [`Node::on_failure`] resolves the alias at runtime.
#[must_use]
pub fn auto_status(&self) -> bool {
self.bool_attr("auto_status").unwrap_or(false)
@ -614,7 +630,7 @@ impl Graph {
.and_then(AttrValue::as_str)
}
/// Graph-level failure routing policy. Invalid values are rejected during
/// Graph-level failure policy. Invalid values are rejected during
/// workflow validation, so runtime resolution can use the compatibility
/// default.
#[must_use]
@ -626,7 +642,7 @@ impl Graph {
.unwrap_or_default()
}
/// Effective failure routing policy for a node. A node-level `on_failure`
/// Effective failure policy for a node. A node-level `on_failure`
/// attribute overrides the graph level; an absent (or invalid, hence
/// validation-rejected) node attribute inherits the graph policy.
#[must_use]
@ -770,9 +786,11 @@ mod tests {
fn on_failure_parses_and_displays_supported_values() {
assert_eq!("route".parse::<OnFailure>().unwrap(), OnFailure::Route);
assert_eq!("exit".parse::<OnFailure>().unwrap(), OnFailure::Exit);
assert_eq!("succeed".parse::<OnFailure>().unwrap(), OnFailure::Succeed);
assert_eq!(OnFailure::Route.to_string(), "route");
assert_eq!(OnFailure::Exit.to_string(), "exit");
assert_eq!(OnFailure::expected_values(), "route, exit");
assert_eq!(OnFailure::Succeed.to_string(), "succeed");
assert_eq!(OnFailure::expected_values(), "route, exit, succeed");
}
#[test]
@ -815,6 +833,28 @@ mod tests {
assert_eq!(node.on_failure(), None);
}
#[test]
fn node_auto_status_is_an_alias_for_on_failure_succeed() {
let mut node = Node::new("work");
node.attrs
.insert("auto_status".to_string(), AttrValue::Boolean(true));
assert!(node.auto_status());
assert_eq!(node.on_failure(), Some(OnFailure::Succeed));
// An explicit on_failure attribute wins over the alias.
node.attrs.insert(
"on_failure".to_string(),
AttrValue::String("exit".to_string()),
);
assert_eq!(node.on_failure(), Some(OnFailure::Exit));
// auto_status=false does not set a policy.
let mut node = Node::new("work");
node.attrs
.insert("auto_status".to_string(), AttrValue::Boolean(false));
assert_eq!(node.on_failure(), None);
}
#[test]
fn resolve_on_failure_prefers_node_policy_over_graph_policy() {
let mut graph = Graph::new("test");

View file

@ -8,7 +8,7 @@ use serde::{Deserialize, Deserializer, Serialize, Serializer};
use serde_json::Value;
use strum::{Display, EnumString, IntoStaticStr};
use crate::{ExecOutputTail, FailureSignature, StageTiming, SystemActorKind};
use crate::{ExecOutputTail, FailureSignature, ResolvedOnFailure, StageTiming, SystemActorKind};
pub trait OutcomeMeta:
Default + Clone + Send + Sync + fmt::Debug + Serialize + DeserializeOwned + 'static
@ -329,13 +329,77 @@ impl<M: OutcomeMeta> Outcome<M> {
..Self::default()
}
}
/// Applies the `on_failure="succeed"` policy: a `failed` outcome becomes
/// `succeeded`. The original `failure` stays on the outcome so durable
/// events and collected results keep the diagnostic, and `notes` records
/// which scope promoted it. Other statuses are left unchanged.
pub fn promote_to_succeeded(&mut self, policy: ResolvedOnFailure) {
if !self.status.is_failure() {
return;
}
self.status = StageOutcome::Succeeded;
let note = format!(
"{} on_failure=succeed promoted a failed outcome to succeeded",
policy.scope()
);
self.notes = Some(match self.notes.take() {
Some(existing) => format!("{existing}\n{note}"),
None => note,
});
}
}
#[cfg(test)]
mod tests {
use serde_json::json;
use super::{FailureCategory, FailureDetail, StageOutcome, StageState};
use super::{FailureCategory, FailureDetail, Outcome, StageOutcome, StageState};
use crate::{OnFailure, ResolvedOnFailure};
#[test]
fn promote_to_succeeded_keeps_failure_and_records_scope() {
let mut outcome: Outcome = Outcome::fail("boom");
outcome.promote_to_succeeded(ResolvedOnFailure::node(OnFailure::Succeed));
assert_eq!(outcome.status, StageOutcome::Succeeded);
assert_eq!(
outcome
.failure
.as_ref()
.map(|failure| failure.message.as_str()),
Some("boom")
);
assert_eq!(
outcome.notes.as_deref(),
Some("node on_failure=succeed promoted a failed outcome to succeeded")
);
}
#[test]
fn promote_to_succeeded_appends_to_existing_notes() {
let mut outcome: Outcome = Outcome::fail("boom");
outcome.notes = Some("handler note".to_string());
outcome.promote_to_succeeded(ResolvedOnFailure::graph(OnFailure::Succeed));
assert_eq!(
outcome.notes.as_deref(),
Some("handler note\ngraph on_failure=succeed promoted a failed outcome to succeeded")
);
}
#[test]
fn promote_to_succeeded_ignores_non_failed_outcomes() {
let mut partial: Outcome = Outcome::success();
partial.status = StageOutcome::PartiallySucceeded;
let mut skipped: Outcome = Outcome::skipped("not needed");
for outcome in [&mut partial, &mut skipped] {
let before = outcome.clone();
outcome.promote_to_succeeded(ResolvedOnFailure::node(OnFailure::Succeed));
assert_eq!(*outcome, before);
}
}
#[test]
fn stage_outcome_failed_serde_is_lossy_for_retry_intent() {

View file

@ -0,0 +1,7 @@
digraph DeprecatedAutoStatus {
start [shape=Mdiamond]
exit [shape=Msquare]
scan [prompt="Scan the tree" auto_status=true]
start -> scan -> exit
}