fabro/docs/public/execution/failures.mdx

---
title: "Failures"
description: "How Fabro classifies, retries, and recovers from failures during workflow execution"
---

Failures are inevitable when orchestrating LLM-powered workflows — models hit rate limits, agents produce bad output, commands fail, and providers go down. Fabro handles this with multiple layers of defense: automatic retries with backoff, provider failover, failure classification for intelligent routing, circuit breakers to prevent infinite loops, and goal gates to catch quality problems before a run completes.

## Failure classification

When a node fails, Fabro classifies the failure into one of six categories. These classes drive retry decisions, circuit breaker logic, and edge routing.

| Class | Description | Examples |
|---|---|---|
| `transient_infra` | Temporary infrastructure problem — likely to resolve on retry | Rate limits, timeouts, network errors, 5xx responses |
| `deterministic` | Permanent failure — retrying won't help | Authentication errors, bad configuration, invalid requests |
| `budget_exhausted` | Resource limit reached | Context length exceeded, token/turn limits, quota exhausted |
| `compilation_loop` | Reserved for loop detection | — |
| `canceled` | User or system cancellation | Cancel signal, abort |
| `structural` | Reserved for scope enforcement | Write scope violations |

Classification happens automatically. Fabro inspects SDK error types, HTTP status codes, and error message patterns to assign the right class. The `failure_class` is written to [context](/execution/context) after each stage, so you can route on it in edge conditions:

```dot
implement -> fix       [condition="failure_class=transient_infra"]
implement -> escalate  [condition="failure_class=deterministic"]
```

## Human gates fail closed when unanswered

Human approval gates are special: lack of an answer is treated as a failure, not as an implicit approval. If a prompt ends because stdin is closed, the user cancels it, a web session disconnects, or a test/replay interviewer has no answer available, the human stage returns a failure outcome instead of selecting an unconditional branch.

That means an unanswered gate will only continue if you model that path explicitly, for example:

```dot
approve [shape=hexagon, label="Approve release?"]

approve -> ship          [label="[A] Approve"]
approve -> manual_review [condition="outcome=failed"]
```

If no `outcome=failed` edge or `retry_target` exists, the run stops rather than advancing past the approval gate.

## Stop linear workflows on failure

By default, Fabro uses `on_failure="route"`. A failed node can take an unconditional edge when no explicit route matches. This compatibility default lets existing workflows decide how later nodes handle the failure.

Set graph-level `on_failure="exit"` to stop a linear workflow at a failed node:

```dot title="stop-on-failure.fabro"
digraph Build {
    graph [on_failure="exit"]

    start [shape=Mdiamond]
    exit [shape=Msquare]
    plan [prompt="Plan the work"]
    implement [prompt="Implement the plan"]
    verify [prompt="Verify the implementation"]

    start -> plan -> implement -> verify -> exit
}
```

Fabro still uses an explicit recovery edge, such as `condition="outcome=failed"`, before it applies this policy. Matching preferred labels and suggested next node IDs also remain explicit routes. If no explicit edge matches, `exit` skips the unconditional edge and checks retry targets. The run ends as failed only when no retry target exists.

Set `on_failure` on a node to control that node alone. The node-level attribute overrides the graph level, in both directions: a node can opt out of a graph-level `exit` with `on_failure="route"`, or stop the run on its own failure with `on_failure="exit"` while the rest of the graph keeps the default. A node without the attribute inherits the graph policy. See [Failed-node routing policy](/workflows/transitions#failed-node-routing-policy).

The policy applies only to `failed`. Other outcomes keep their normal routing behavior. For parallel nodes, the policy uses the completed parallel node's final outcome. It does not stop or cancel individual branches early.

## Treat a failed node as succeeded

Set `on_failure="succeed"` on a best-effort node so its failure never blocks the workflow. This pairs well with a strict graph default:

```dot title="best-effort-node.fabro"
digraph Review {
    graph [on_failure="exit"]

    start [shape=Mdiamond]
    exit [shape=Msquare]
    required_check [script="./required-check"]
    optional_scan [script="./optional-scan" on_failure="succeed"]

    start -> required_check -> optional_scan -> exit
}
```

When `optional_scan` fails, Fabro first checks explicit recovery routes with the `failed` outcome. If none match, it rewrites the outcome to `succeeded` and routes the node as a success. Retries still run first; only the final outcome changes. The original failure stays on the `stage.completed` event and in the checkpoint, and the outcome's notes record the promotion. A promoted outcome satisfies a goal gate. Setting `on_failure="succeed"` on the graph applies it to every node.

`succeed` applies only to `failed`. It does not change a `partially_succeeded` outcome. `auto_status=true` is the deprecated spelling of this policy; validation warns and suggests `on_failure="succeed"`.

## Retry layers

Fabro retries failures at three levels: **LLM retries** handle transient API errors inside a single model call, **turn-level retries** recover from dropped streams mid-response, and **node retries** re-execute the entire node handler when the first two levels aren't enough. These layers are independent — a node retry re-runs the full handler, which gets its own fresh set of LLM and turn-level retries.

### LLM retries

Every LLM call (within an agent session or a one-shot prompt node) has a built-in retry loop for transient API errors. This is invisible to the workflow — it happens inside the model call itself.

| Setting | Default |
|---|---|
| Max retries | 3 |
| Initial delay | 1 second |
| Backoff multiplier | 2x |
| Max delay | 60 seconds |
| Jitter | 0.5x–1.5x random factor |

Only transient errors are retried: rate limits, server errors (5xx), timeouts, network failures, and stream interruptions. Permanent errors like authentication failures or invalid requests fail immediately.

If the provider returns a `Retry-After` header, Fabro respects it — unless the delay exceeds 60 seconds, in which case the call fails rather than blocking the run.

### Turn-level retries

When an LLM stream drops mid-response (common under high concurrency), Fabro retries the same agent turn up to 3 times instead of failing the entire session. Conversation history is preserved across retries, and any partial assistant output from the interrupted stream is replayed so the model can continue where it left off. This avoids restarting the full stage from scratch for transient stream failures.

### Node retries

When a node handler fails (after LLM and turn-level retries are exhausted), the engine can retry the entire node. This is controlled by **retry policies**.

#### Retry policies

Set a retry policy on a node with the `retry_policy` attribute:

```dot
implement [retry_policy="standard"]
```

| Policy | Max attempts | Initial delay | Backoff | Typical delays |
|---|---|---|---|---|
| `none` | 1 | — | — | No retries |
| `standard` | 5 | 200ms | 2x exponential | 200ms, 400ms, 800ms, 1.6s |
| `aggressive` | 5 | 500ms | 2x exponential | 500ms, 1s, 2s, 4s |
| `linear` | 3 | 500ms | 1x (constant) | 500ms, 500ms |
| `patient` | 3 | 2s | 3x exponential | 2s, 6s |

All policies apply random jitter (0.5x–1.5x) and cap individual delays at 60 seconds.

#### Setting retries without a policy

You can also set just the retry count using `max_retries`:

```dot
implement [max_retries="5"]
```

This uses the default backoff (5s initial, 2x exponential) with the specified number of retries.

#### Resolution order

The engine resolves retry configuration in this order:

1. Node attribute `retry_policy` — named preset
2. Node attribute `max_retries` — count only, default backoff
3. Graph attribute `default_max_retries` — applies to all nodes without explicit config (default: **3**)

#### What gets retried

Not all errors trigger a node retry. The handler's `should_retry` check must return true — generally, only errors classified as transient are retried. Deterministic errors (auth failures, bad config) fail immediately without consuming retry attempts.

When a handler reports a retryable failure, retries always proceed if attempts remain. If retries are exhausted and the node has `allow_partial=true`, the outcome is promoted to `partially_succeeded` instead of `failed`. See [Node Outcomes — Retry loop](/execution/outcomes#retry-loop) for a detailed flow diagram.

## Model fallbacks

When a model provider fails with a provider-local error, Fabro can automatically switch to another target. Configure one fixed chain for each requested model in your [run configuration](/execution/run-configuration):

```toml title="run.toml"
[run.model]
name = "claude-opus-4-6"
provider = "anthropic"

[run.model.fallbacks]
"claude-opus-4-6" = ["gemini", "openai"]
```

When Anthropic fails, Fabro tries Gemini first, then OpenAI. Fallback resolution is provider-aware:

- A bare provider token such as `"gemini"` selects that provider's closest compatible model.
- A qualified selector such as `"openrouter:gpt-56-sol"` resolves only within that provider. The selector may be a canonical model ID, alias, or provider API ID such as `"openrouter:moonshotai/kimi-k3"`.
- A bare model slug or alias considers ready providers and uses provider priority.

Qualified fallback references always remain provider pins. For example, `"openai:gpt-5.6-sol"` pins the direct OpenAI offering. Legacy `provider/model` fallback references remain accepted for compatibility.

Fabro selects the chain by the original requested model. A target in that chain never activates the target model's own chain. The same chain position is retained across structured-output repairs and cached agent sessions.

The primary provider and model were already resolved and persisted when the run was created; resuming does not re-run primary selection. Fallbacks are only considered after an eligible runtime failure. If the fallback model does not support the requested reasoning level, Fabro uses the nearest supported level and rounds equal-distance choices up.

### What triggers failover

Failover is a superset of LLM retry eligibility:

| Error type | LLM retry | Provider failover |
|---|---|---|
| Rate limit | Yes | Yes |
| Server error (5xx) | Yes | Yes |
| Timeout / network | Yes | Yes |
| Quota exceeded | No | Yes |
| Authentication (401) | No | Yes |
| Access denied (403) | No | Yes |
| Model not found (404) | No | Yes |
| Model refusal | No | Yes |
| Invalid request (400) | No | No |
| Context length (413) | No | No |
| Content filter | No | No |

Quota errors are the key distinction — they aren't retried against the same provider (the quota won't reset) but *are* eligible for failover to a provider with its own quota.

## Loop detection

Fabro has two independent mechanisms for detecting stuck loops: **node visit limits** that catch workflow-level cycles, and **tool call pattern detection** that catches agent-level repetition.

### Node visit limits

The `max_node_visits` graph attribute sets the maximum number of times any single node can execute before the run is terminated:

```dot title="example.fabro"
digraph Example {
    graph [max_node_visits="20"]
    // ...
}
```

| Context | Default |
|---|---|
| Normal runs | Disabled (unlimited) |
| Dry runs (`--dry-run`) | 10 |
| Explicit `max_node_visits` | The configured value |

When a node hits the limit, the run fails immediately:

```
node "verify" visited 20 times (graph limit 20); run is stuck in a cycle
```

#### Per-node overrides

You can set `max_visits` on individual nodes to override the graph-level limit for that node:

```dot title="example.fabro"
digraph Example {
    graph [max_node_visits="20"]
    fix [max_visits=3]
}
```

The per-node `max_visits` takes precedence over `max_node_visits` (and the dry-run default of 10). This is useful when specific nodes — like a fix-and-verify loop — should have a tighter limit than the rest of the workflow:

```
node "fix" visited 3 times (node limit 3); run is stuck in a cycle
```

### Tool call loop detection

Inside an agent session, Fabro monitors the last 10 assistant turns for repeating tool call patterns. It detects patterns of length 1 (same call repeated), 2 (A-B-A-B), or 3 (A-B-C-A-B-C). Every complete group in the window must match for detection to trigger.

When a loop is detected, Fabro injects a steering message into the conversation:

> WARNING: Loop detected. You appear to be repeating the same tool calls. Please try a different approach or ask for clarification.

This gives the agent a chance to break out of the loop without failing the node.

## Failure signatures and circuit breakers

Failure signatures are a deduplication mechanism that prevents the same failure from recurring indefinitely across loop iterations. They are particularly important for workflows with retry loops (implement → verify → fix → verify → ...).

### How signatures work

After each failed node, Fabro constructs a **failure signature** — a normalized fingerprint combining the node ID, failure class, and error message:

```
implement|deterministic|handler panicked: index out of bounds
```

The error message is normalized by lowercasing, replacing hex strings with `<hex>`, replacing digits with `<n>`, and truncating to 240 characters. This groups failures with the same root cause even when details like line numbers or timestamps vary.

### Circuit breaker

Fabro tracks signature counts across the run. When the same signature repeats **3 times** (configurable via `loop_restart_signature_limit`), the run is terminated:

```
deterministic failure cycle detected: signature ... repeated 3 times (limit 3)
```

Only `deterministic` and `structural` failures are tracked — transient failures are excluded because they may genuinely resolve on retry.

<Note>
Failure signature counts are never reset on success. This is intentional — it prevents cycles like "implement succeeds → verify fails → fix → implement succeeds → verify fails" from running indefinitely.
</Note>

### Loop restart edges

Taking an edge marked with `loop_restart=true` restarts the workflow from the edge's target node. A restart is more than a jump: the completed-stage history, per-node outcomes, and retry counts are cleared, and the run context is replaced with a **fresh, empty context** — the target node starts over as if the run had just begun there, with no preamble of prior stages. Node visit counts are the one thing preserved, so `max_visits` and `max_node_visits` still bound how many times a restart loop can run.

A **successful** outcome may take a `loop_restart` edge freely. This is the "start another round from a clean slate" pattern — for example, a self-loop that begins a fresh batch of work and re-derives its remaining work from the repository state rather than from accumulated context.

A **failed** outcome faces an additional guard: only `transient_infra` failures may cross a `loop_restart` edge. If the failure class is anything else, the run is terminated:

```
loop_restart blocked: failure_class=deterministic (requires transient_infra)
```

Loop restart edges also have their own separate circuit breaker (`restart_failure_signatures`) that enforces the same signature limit.

## Goal gates

Goal gates are quality checkpoints that are enforced when the workflow reaches an exit node. A node marked with `goal_gate=true` must have completed with `succeeded` or `partially_succeeded` — otherwise the run cannot finish. See [Node Outcomes — Goal gate interaction](/execution/outcomes#goal-gate-interaction) for how `partially_succeeded` and `allow_partial` interact with goal gates.

```dot
verify [shape=box, goal_gate="true"]
```

When a goal gate is unsatisfied at the exit node, Fabro looks for a **retry target** — a node to jump back to for another attempt:

1. Failed node's `retry_target` attribute
2. Failed node's `fallback_retry_target` attribute
3. Graph-level `retry_target` attribute
4. Graph-level `fallback_retry_target` attribute

```dot title="example.fabro"
digraph Example {
    graph [retry_target="plan"]
    verify [shape=box, goal_gate="true", retry_target="implement"]
    // If verify fails, jump to implement
    // If no node-level target existed, would jump to plan
}
```

If no retry target is found at any level, the run fails:

```
goal gate unsatisfied for node verify and no retry target
```

## Stall watchdog

Fabro runs a background watchdog that monitors event activity. If no events are emitted for longer than the **stall timeout**, the run is canceled. This catches cases where a handler hangs indefinitely without producing errors.

| Setting | Default |
|---|---|
| `stall_timeout` | 1800 seconds (30 minutes) |
| Set to `0` | Disables the watchdog |

```dot title="example.fabro"
digraph Example {
    graph [stall_timeout="300"]  // 5 minutes
}
```

## When failures become fatal

A node failure does **not** automatically terminate the run. Fabro follows this escalation path:

1. **LLM retries** — transient API errors are retried inside the model call (up to 3 retries)
2. **Turn-level retries** — dropped streams retry the same agent turn (up to 3 retries), preserving conversation history
3. **Provider failover** — if configured, switch to a fallback provider
4. **Node retries** — re-execute the entire handler (per the retry policy)
5. **Direct jump** — use `jump_to_node` when the outcome supplies one
6. **Explicit edge routing** — look for a matching condition, preferred label, or suggested next node
7. **Failure policy** — with no explicit route, apply the effective `on_failure` (node-level `on_failure` first, then graph-level): `exit` skips the unconditional edge, `succeed` promotes the outcome to `succeeded` and routes it as a success, and `route` (or no attribute) keeps normal fallback routing
8. **Unconditional edge** — in `route` mode, or after a `succeed` promotion, use an edge without a condition as the fallback
9. **Retry target** — if no edge was selected, check `retry_target` and `fallback_retry_target` on the node, then on the graph
10. **Run failure** — if none of the above produces a path forward, the run terminates

When a retry target sends the run back to a failing path, use graph-level `max_node_visits` or node-level `max_visits` to stop an unbounded cycle.

The run also terminates immediately for:

- **Node visit limit exceeded** — a node has been visited too many times
- **Circuit breaker tripped** — the same failure signature has repeated too many times
- **Loop restart blocked** — a non-transient failure tried to cross a `loop_restart` edge
- **Goal gate failure with no retry target** — a required gate was unsatisfied at the exit node
- **Stall timeout** — no events for too long
- **Cancellation** — user or system cancel signal