## Summary
Run cancellation now reaches in-flight agent work instead of waiting for
an agent stage to finish or recording cancellation as a failed stage.
The workflow cancellation primitive is now
`tokio_util::sync::CancellationToken`, with child tokens passed through
setup, handlers, manager-loop child runs, sandbox streaming commands,
CLI agent invocations, and API agent sessions.
### Plan Summary
- Promote run cancellation to `CancellationToken` while keeping stall
timeout separate.
- Route CLI agents through cancellable sandbox streaming with optional
timeouts.
- Bridge run cancellation into API sessions and preserve
`Error::Cancelled` propagation.
- Add typed events/projections for CLI cancellation and timeout.
## Cancellation flow
```mermaid
flowchart TB
RunToken[Run CancellationToken]
Executor[Core executor]
Services[RunServices]
Manager[Manager-loop child run]
CLI[Agent CLI backend]
API[Agent API backend]
Sandbox[Sandbox streaming exec]
Session[fabro-agent Session]
RunToken --> Executor
RunToken --> Services
Services -- child_token --> Manager
Services -- child_token --> CLI
CLI -- child_token --> Sandbox
Services --> API
API -- bridge guard --> Session
```
## What changed and why
- `RunOptions`, `RunServices`, core `ExecutorOptions`, CLI/server run
state, and detached-run guards now use `CancellationToken` instead of
`Arc<AtomicBool>`. Dropping services or tokens still does not mean
cancellation; only explicit `.cancel()` does.
- Manager-loop child workflows are given child tokens so parent
cancellation propagates down, while stop/max-cycle cancellation remains
scoped to the child workflow.
- Stall timeout remains intentionally separate as a stall token and
still returns `Error::StallTimeout { node_id }`, not `Error::Cancelled`.
- Agent, prompt, human, fan-in, and parallel handler paths now pass
cancellation tokens through and avoid converting `Error::Cancelled` into
normal failed outcomes.
## Agent backend behavior
CLI-mode agents no longer launch detached `setsid` jobs with temp
stdout/stderr/exit-code polling. They run through
`Sandbox::exec_command_streaming` with a child token; a missing node
timeout passes `None` to preserve the existing unbounded agent runtime,
while explicit node timeouts still apply. Cancelled CLI runs emit
`agent.cli.cancelled`, clean temp files, and return `Error::Cancelled`;
timed-out CLI runs emit `agent.cli.timed_out` and return a handler
timeout error; `agent.cli.completed` remains natural-exit only.
API-mode agents install a per-invocation `SessionCancelBridgeGuard`
after acquiring a fresh or cached session. The guard maps the run token
into the session interrupt reason and session cancel token, and aborts
stale bridge tasks before session replacement or cache reinsertion so
reused sessions are not tied to old run tokens. `Session::initialize`
now returns `Result`, and project-doc, skill, MCP, and environment
discovery paths check cancellation and pass child tokens to sandbox
commands.
## Sandbox and event model
`Sandbox::exec_command_streaming` now accepts `Option<u64>` for timeout.
Production streaming implementations use a pending future for `None`
instead of a giant sleep, while the trait fallback maps `None` to
`u64::MAX` only when delegating to non-streaming `exec_command`.
The run event model now includes typed `agent.cli.cancelled` and
`agent.cli.timed_out` payloads with stdout, stderr, and duration, plus
conversion and projection support. OpenAPI/client regeneration was
unnecessary because the API schema already models run events with a free
event string and arbitrary properties; only Rust event types changed.
## Reviewer notes
Expect signature churn around `Session::initialize`,
`CodergenBackend::run`, `RunOptions.cancel_token`,
`StartServices.cancel_token`, and `Sandbox::exec_command_streaming`. The
main behavioral checks are that user cancellation reaches in-flight
CLI/API work and that timeout/stall paths remain distinct from user
cancellation.
### Fabro Details
<details>
<summary>Ran 9 stages in 117m 40s for $150.32</summary>
| Stage | Duration | Cost | Retries |
|---|---|---|---|
| start | 0s | – | 0 |
| toolchain | 1s | – | 0 |
| preflight_compile | 2m 8s | – | 0 |
| preflight_lint | 2m 13s | – | 0 |
| implement | 77m 12s | $56.78 | 0 |
| simplify_opus | 18m 5s | $5.83 | 0 |
| simplify_gpt | 15m 33s | $87.71 | 0 |
| verify | 1m 48s | – | 0 |
| fmt | 2s | – | 0 |
| **Total** | **117m 40s** | **$150.32** | **0** |
</details>
<details>
<summary>Ran <code>ImplementPlan.fabro</code> (12 nodes and 15
edges)</summary>
```dot
digraph ImplementPlan {
graph [
goal="Implement and simplify",
model_stylesheet="
* { model: claude-opus-4-7; }
"
]
rankdir=LR
start [shape=Mdiamond, label="Start"]
exit [shape=Msquare, label="Exit"]
toolchain [label="Toolchain", shape=parallelogram, script="command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1", max_retries=0]
preflight_compile [label="Preflight Compile", shape=parallelogram, script="cargo check -q --workspace 2>&1", max_retries=0]
preflight_lint [label="Preflight Lint", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1", max_retries=0]
fix_lints [label="Fix Lints", prompt="The preflight lint step failed. Read the build output from context and fix all clippy lint warnings.", max_visits=3]
implement [label="Implement", prompt="Read the plan file referenced in the goal and implement every step. Make all the code changes described in the plan. Use red/green TDD."]
simplify_opus [label="Simplify (Opus)", prompt="@prompts/simplify.md"]
simplify_gpt [label="Simplify (GPT-55)", prompt="@prompts/simplify.md", model="gpt-55"]
verify [label="Verify", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --cargo-quiet --workspace --status-level fail 2>&1 && cargo dev docs refresh 2>&1 && cargo dev docs check 2>&1", goal_gate=true, retry_target="fixup"]
fixup [label="Fixup", prompt="The verify step failed. Read the build output from context and fix all clippy lint warnings, test failures, and generated docs errors.", max_visits=3]
fmt [label="Format", shape=parallelogram, script="cargo +nightly-2026-04-14 fmt --all 2>&1", max_retries=0]
start -> toolchain
toolchain -> preflight_compile [condition="outcome=succeeded"]
toolchain -> exit
preflight_compile -> preflight_lint [condition="outcome=succeeded"]
preflight_compile -> exit
preflight_lint -> implement [condition="outcome=succeeded"]
preflight_lint -> fix_lints
fix_lints -> preflight_lint
implement -> simplify_opus -> simplify_gpt -> verify
verify -> fmt [condition="outcome=succeeded"]
verify -> fixup
fixup -> verify
fmt -> exit
}
```
</details>
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
Use to_rfc3339_opts with SecondsFormat::Secs so auth status output
shows clean second-precision timestamps instead of nanoseconds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
### Summary
Silent fallback paths now emit stable warnings instead of degrading
without a user-visible signal. The fallback behavior is unchanged; runs
still continue, but worktree, Git, checkpoint, and LLM failover issues
now show up in the run feed and logs.
### Plan Summary
- Emit run notices for workflow Git/worktree fallback paths.
- Reuse the existing failover event for one-shot LLM provider fallback.
- Add tracing for sandbox pipe drain failures.
### What changed
- Added `worktree_skipped_no_git` and gated `sandbox_git_unavailable`
notices during initialization.
- Added `git_push_failed` and `parallel_base_checkpoint_failed` notices,
including redacted output tails where available.
- Logged GitHub token mint failures with a structured `error` field
before the existing notice.
- Plumbed `Emitter` and `StageScope` through `CodergenBackend::one_shot`
so the API backend emits the existing `agent.failover` event instead of
a duplicate tracing-only warning.
- Extracted sandbox pipe draining into a helper that warns on
stdout/stderr read failures, with unit coverage for the error path.
- Updated CLI snapshots for the new worktree warning in stderr and JSON
event output.
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
This change makes bulk `fabro model test` run configured model checks
concurrently instead of serially. A new `--jobs/-j` flag (defaulting to
4, minimum 1) controls the concurrency bound; the single-model path
(`--model <MODEL>`) is unaffected. Under the hood, the serial `for` loop
over configured models is replaced with a
`futures::stream::buffer_unordered(jobs)` pipeline that clones the
shared-state `Client` per request. Completed results carry their
original list index and are sorted before rendering, so final stdout
table rows and JSON output remain in listing order regardless of which
requests finish first.
Three new integration tests verify the concurrency behavior using an
inline Axum harness with a `ConcurrencyGate` barrier. The gate holds all
in-flight requests until the expected number arrive simultaneously, then
releases them, letting tests assert `max_in_flight` exactly rather than
relying on timing. The ordering test goes further by assigning
reverse-listing response delays so the last-listed model always finishes
first; if the index sort were dropped, the JSON result order would
invert and the assertion would fail. A 15-second gate timeout ensures a
regression to serial execution surfaces as a clear `max_in_flight == 1`
failure rather than a hung test.
Existing behavior is fully preserved: unconfigured models are still
skipped without a POST, a configured model returning `skip` after
listing is still a failure, `--deep` uses the same `--jobs` value, and
`--jobs 1` reproduces the previous serial behavior for users hitting
provider rate limits.
### Fabro Details
<details>
<summary>Ran 9 stages in 30m 52s for $19.61</summary>
| Stage | Duration | Cost | Retries |
|---|---|---|---|
| start | 0s | – | 0 |
| toolchain | 1s | – | 0 |
| preflight_compile | 2m 8s | – | 0 |
| preflight_lint | 2m 21s | – | 0 |
| implement | 8m 56s | $3.87 | 0 |
| simplify_opus | 7m 55s | $1.43 | 0 |
| simplify_gpt | 6m 58s | $14.32 | 0 |
| verify | 1m 49s | – | 0 |
| fmt | 2s | – | 0 |
| **Total** | **30m 52s** | **$19.61** | **0** |
</details>
<details>
<summary>Ran <code>ImplementPlan.fabro</code> (12 nodes and 15
edges)</summary>
```dot
digraph ImplementPlan {
graph [
goal="Implement and simplify",
model_stylesheet="
* { model: claude-opus-4-7; }
"
]
rankdir=LR
start [shape=Mdiamond, label="Start"]
exit [shape=Msquare, label="Exit"]
toolchain [label="Toolchain", shape=parallelogram, script="command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1", max_retries=0]
preflight_compile [label="Preflight Compile", shape=parallelogram, script="cargo check -q --workspace 2>&1", max_retries=0]
preflight_lint [label="Preflight Lint", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1", max_retries=0]
fix_lints [label="Fix Lints", prompt="The preflight lint step failed. Read the build output from context and fix all clippy lint warnings.", max_visits=3]
implement [label="Implement", prompt="Read the plan file referenced in the goal and implement every step. Make all the code changes described in the plan. Use red/green TDD."]
simplify_opus [label="Simplify (Opus)", prompt="@prompts/simplify.md"]
simplify_gpt [label="Simplify (GPT-55)", prompt="@prompts/simplify.md", model="gpt-55"]
verify [label="Verify", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --cargo-quiet --workspace --status-level fail 2>&1 && cargo dev docs refresh 2>&1 && cargo dev docs check 2>&1", goal_gate=true, retry_target="fixup"]
fixup [label="Fixup", prompt="The verify step failed. Read the build output from context and fix all clippy lint warnings, test failures, and generated docs errors.", max_visits=3]
fmt [label="Format", shape=parallelogram, script="cargo +nightly-2026-04-14 fmt --all 2>&1", max_retries=0]
start -> toolchain
toolchain -> preflight_compile [condition="outcome=succeeded"]
toolchain -> exit
preflight_compile -> preflight_lint [condition="outcome=succeeded"]
preflight_compile -> exit
preflight_lint -> implement [condition="outcome=succeeded"]
preflight_lint -> fix_lints
fix_lints -> preflight_lint
implement -> simplify_opus -> simplify_gpt -> verify
verify -> fmt [condition="outcome=succeeded"]
verify -> fixup
fixup -> verify
fmt -> exit
}
```
</details>
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
The default OpenAI model moved from gpt-5.5 back to gpt-5.4 in 38b51c4c2,
but this attach test snapshot still asserted gpt-5.5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GPT-5.5 (released 2026-04-23) replaces 5.4 as the default OpenAI model.
Live integration tests confirm both new IDs respond on the OpenAI API;
they require default temperature like other reasoning models.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Keep sandbox exec failures structured until event emission so git push, checkpoint, notice, and retro failures can expose redacted output tails without expanding their terse error strings.
Also add log rendering that appends sanitized tail content for exec-backed errors while preserving the existing safe Display behavior.
Emit snapshot slow-path events only when Docker or Daytona actually performs image or snapshot work, replace retired completion markers with snapshot.ready, and render the lifecycle in attach/log output.
So that CLI and other API consumers can surface a clickable link to the
run's web UI page instead of guessing route shapes or probing settings.
The server populates `web_url` from `server.web.enabled` and
`server.web.url`, returns it on `RunStatusResponse` (create plus all
lifecycle transitions), and persists it on the `run.created` event so
attach replays the same link without re-deriving it.
CLI: prints `Web UI: <url>` as a run-header info line, driven off the
replayed event so fresh runs and `attach` share one code path. Absent
when the UI is disabled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use HTTP health checks with short deadlines for managed server readiness and add finite control-plane request timeouts for CLI/server clients. Keep stream bodies uncapped so SSE attach flows can remain long-lived.
Extract STAGE_RANK_WIDTH and a derived MAX_STAGES_IN_DUMP in fabro-dump so
the path-prefix format and the stage-count cap can't drift, and replace the
two `{rank:03}-...` literals with a shared stage_dir_name helper.
Replace the cli/dump.rs `u32::try_from(artifact.retry)` with the symmetric
inverse of the server's `cast_signed()` emit. The OpenAPI schema declares
`minimum: 0`, so the negative branch is unreachable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Keep typed transport and provider errors intact through API, GitHub, OAuth, install, diagnostics, and artifact paths. Add regression coverage for cloned shared errors and communication error chains.
Restore snapshot form for auth status JSON test so re-introducing an
env_dev_token field would fail the snapshot, and rename the ps test to
reflect that it proves FABRO_DEV_TOKEN is ignored rather than that auth
is generally required.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add bounded redacted exec output tails to failure events while keeping tracing log-safe. Centralize tail projection on ExecResult and thread diagnostics through metadata, setup, devcontainer, and CLI install failures.
Move RunDump into fabro-dump so CLI export and retro uploads share the same hydrated run layout. Drop the legacy artifact file-ref parser, add best-effort run.log retrieval for retro, and update retro prompts/docs to use events.jsonl and checkpoints.
Replace 16 inline copies of the `[cli.target] type = "http"` settings TOML across CLI integration tests with a single `set_http_target(&base_url)` method on `TestContext`. Removes a brittle format string that was maintained in ten files but only meaningfully asserted-against in one.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add configured to the model API contract and server responses so clients can see whether provider credential material exists before testing. Use that signal in bulk model tests to skip unconfigured providers before printing progress and treat post-list skips as race failures.
The Metadata init/checkpoint/finalize lines added noise to `fabro run`
output. The underlying events still flow into progress.jsonl; only the
live rendering is removed. Failures continue to render as warnings.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add typed metadata snapshot events around init, checkpoint, and finalize archive writes so run logs expose durable metadata timing and failures. Include snapshot accounting, CLI rendering with compatibility-notice suppression, and event documentation.
The bail!("Validation failed") was nested inside `if !quiet`, so
`fabro run --detach` and `fabro run create` (both pass quiet=true)
silently created runs from invalid workflows. Move the bail outside the
gate; only the workflow summary print remains gated on !quiet.
Also simplifies the surrounding preflight code:
- Extract `cyan_spinner` helper in fabro-cli's shared utilities;
collapse three copy-pasted 13-line spinner setups in preflight.rs,
doctor.rs, and install.rs.
- Add `SandboxProvider::is_clone_based()`; replace the local
`is_clone_based_provider` helper and two inline
`matches!(_, Docker | Daytona)` sites in run_manifest.rs.
- Promote `fabro_sandbox::redact::redact_auth_url` to pub and reuse it;
delete the duplicate `redact_remote_output` in run_manifest.rs.
- Inline the one-liner `preflight_docker_config` /
`preflight_daytona_config` helpers and drop their dedicated tests.
- Type the `prepared_and_resolved_for_sandbox` test helper with
`SandboxProvider` instead of `&str`.
- Drop git ls-remote preflight timeout from 30s to 10s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`fabro preflight` blocks on a single server-side `run_preflight` call.
Without feedback the terminal sits blank until it returns. Mirror the
existing `fabro doctor` spinner (cyan braille, "Running checks...",
80ms tick), gated on `!ctx.json_output()` so JSON and piped callers
stay clean. The network calls run inside an inline async block so the
spinner is `finish_and_clear`ed before any `?`-propagated error
prints.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce ManifestPath as the canonical in-memory key for run manifests so CLI-produced bundle keys and workflow/server consumers share the same normalization rules. Validate wire keys at the server boundary and add a CLI-to-server round-trip test for user-global @path references.
GitHubAppCredentials now carries the configured app slug, so the "not
installed" error from the installation lookup links to the specific
app's install page (https://github.com/organizations/{owner}/settings/apps/{slug}/installations)
when known, instead of the generic org installations page. Threaded
through the server, workflow pipeline, and CLI runner.
Also treat docker like daytona for GitHub credential gating: both are
clone-based providers that need an installation token to fetch the repo,
so a docker run now requires credentials when daytona would.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop async from validate::run after the preflight refactor removed all
awaits, replace absolute paths and a one-liner helper in
manifest_validation, swap a redundant to_path_buf for clone in a test,
and regenerate cli.mdx so docs check stays green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace remaining expensive CLI lifecycle checks with seeded fixtures or focused unit coverage so the concurrent suite spends less time on duplicate full-process setup.
Seed read-only CLI tests from run-store fixtures, remove duplicate expensive lifecycle coverage, and keep machine-dependent gh tests offline so the suite no longer probes local credentials.
Add a validation-only API response and route while keeping fabro validate local so it does not start or contact the server for structural workflow checks.
