Route command stderr into stdout at execution time and expose a single output log across events, projections, API clients, and the web UI. Keep replay compatibility for older command.completed events that still contain split stdout/stderr fields.
## Summary
Run cancellation now reaches in-flight agent work instead of waiting for
an agent stage to finish or recording cancellation as a failed stage.
The workflow cancellation primitive is now
`tokio_util::sync::CancellationToken`, with child tokens passed through
setup, handlers, manager-loop child runs, sandbox streaming commands,
CLI agent invocations, and API agent sessions.
### Plan Summary
- Promote run cancellation to `CancellationToken` while keeping stall
timeout separate.
- Route CLI agents through cancellable sandbox streaming with optional
timeouts.
- Bridge run cancellation into API sessions and preserve
`Error::Cancelled` propagation.
- Add typed events/projections for CLI cancellation and timeout.
## Cancellation flow
```mermaid
flowchart TB
RunToken[Run CancellationToken]
Executor[Core executor]
Services[RunServices]
Manager[Manager-loop child run]
CLI[Agent CLI backend]
API[Agent API backend]
Sandbox[Sandbox streaming exec]
Session[fabro-agent Session]
RunToken --> Executor
RunToken --> Services
Services -- child_token --> Manager
Services -- child_token --> CLI
CLI -- child_token --> Sandbox
Services --> API
API -- bridge guard --> Session
```
## What changed and why
- `RunOptions`, `RunServices`, core `ExecutorOptions`, CLI/server run
state, and detached-run guards now use `CancellationToken` instead of
`Arc<AtomicBool>`. Dropping services or tokens still does not mean
cancellation; only explicit `.cancel()` does.
- Manager-loop child workflows are given child tokens so parent
cancellation propagates down, while stop/max-cycle cancellation remains
scoped to the child workflow.
- Stall timeout remains intentionally separate as a stall token and
still returns `Error::StallTimeout { node_id }`, not `Error::Cancelled`.
- Agent, prompt, human, fan-in, and parallel handler paths now pass
cancellation tokens through and avoid converting `Error::Cancelled` into
normal failed outcomes.
## Agent backend behavior
CLI-mode agents no longer launch detached `setsid` jobs with temp
stdout/stderr/exit-code polling. They run through
`Sandbox::exec_command_streaming` with a child token; a missing node
timeout passes `None` to preserve the existing unbounded agent runtime,
while explicit node timeouts still apply. Cancelled CLI runs emit
`agent.cli.cancelled`, clean temp files, and return `Error::Cancelled`;
timed-out CLI runs emit `agent.cli.timed_out` and return a handler
timeout error; `agent.cli.completed` remains natural-exit only.
API-mode agents install a per-invocation `SessionCancelBridgeGuard`
after acquiring a fresh or cached session. The guard maps the run token
into the session interrupt reason and session cancel token, and aborts
stale bridge tasks before session replacement or cache reinsertion so
reused sessions are not tied to old run tokens. `Session::initialize`
now returns `Result`, and project-doc, skill, MCP, and environment
discovery paths check cancellation and pass child tokens to sandbox
commands.
## Sandbox and event model
`Sandbox::exec_command_streaming` now accepts `Option<u64>` for timeout.
Production streaming implementations use a pending future for `None`
instead of a giant sleep, while the trait fallback maps `None` to
`u64::MAX` only when delegating to non-streaming `exec_command`.
The run event model now includes typed `agent.cli.cancelled` and
`agent.cli.timed_out` payloads with stdout, stderr, and duration, plus
conversion and projection support. OpenAPI/client regeneration was
unnecessary because the API schema already models run events with a free
event string and arbitrary properties; only Rust event types changed.
## Reviewer notes
Expect signature churn around `Session::initialize`,
`CodergenBackend::run`, `RunOptions.cancel_token`,
`StartServices.cancel_token`, and `Sandbox::exec_command_streaming`. The
main behavioral checks are that user cancellation reaches in-flight
CLI/API work and that timeout/stall paths remain distinct from user
cancellation.
### Fabro Details
<details>
<summary>Ran 9 stages in 117m 40s for $150.32</summary>
| Stage | Duration | Cost | Retries |
|---|---|---|---|
| start | 0s | – | 0 |
| toolchain | 1s | – | 0 |
| preflight_compile | 2m 8s | – | 0 |
| preflight_lint | 2m 13s | – | 0 |
| implement | 77m 12s | $56.78 | 0 |
| simplify_opus | 18m 5s | $5.83 | 0 |
| simplify_gpt | 15m 33s | $87.71 | 0 |
| verify | 1m 48s | – | 0 |
| fmt | 2s | – | 0 |
| **Total** | **117m 40s** | **$150.32** | **0** |
</details>
<details>
<summary>Ran <code>ImplementPlan.fabro</code> (12 nodes and 15
edges)</summary>
```dot
digraph ImplementPlan {
graph [
goal="Implement and simplify",
model_stylesheet="
* { model: claude-opus-4-7; }
"
]
rankdir=LR
start [shape=Mdiamond, label="Start"]
exit [shape=Msquare, label="Exit"]
toolchain [label="Toolchain", shape=parallelogram, script="command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1", max_retries=0]
preflight_compile [label="Preflight Compile", shape=parallelogram, script="cargo check -q --workspace 2>&1", max_retries=0]
preflight_lint [label="Preflight Lint", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1", max_retries=0]
fix_lints [label="Fix Lints", prompt="The preflight lint step failed. Read the build output from context and fix all clippy lint warnings.", max_visits=3]
implement [label="Implement", prompt="Read the plan file referenced in the goal and implement every step. Make all the code changes described in the plan. Use red/green TDD."]
simplify_opus [label="Simplify (Opus)", prompt="@prompts/simplify.md"]
simplify_gpt [label="Simplify (GPT-55)", prompt="@prompts/simplify.md", model="gpt-55"]
verify [label="Verify", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --cargo-quiet --workspace --status-level fail 2>&1 && cargo dev docs refresh 2>&1 && cargo dev docs check 2>&1", goal_gate=true, retry_target="fixup"]
fixup [label="Fixup", prompt="The verify step failed. Read the build output from context and fix all clippy lint warnings, test failures, and generated docs errors.", max_visits=3]
fmt [label="Format", shape=parallelogram, script="cargo +nightly-2026-04-14 fmt --all 2>&1", max_retries=0]
start -> toolchain
toolchain -> preflight_compile [condition="outcome=succeeded"]
toolchain -> exit
preflight_compile -> preflight_lint [condition="outcome=succeeded"]
preflight_compile -> exit
preflight_lint -> implement [condition="outcome=succeeded"]
preflight_lint -> fix_lints
fix_lints -> preflight_lint
implement -> simplify_opus -> simplify_gpt -> verify
verify -> fmt [condition="outcome=succeeded"]
verify -> fixup
fixup -> verify
fmt -> exit
}
```
</details>
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
### Summary
Silent fallback paths now emit stable warnings instead of degrading
without a user-visible signal. The fallback behavior is unchanged; runs
still continue, but worktree, Git, checkpoint, and LLM failover issues
now show up in the run feed and logs.
### Plan Summary
- Emit run notices for workflow Git/worktree fallback paths.
- Reuse the existing failover event for one-shot LLM provider fallback.
- Add tracing for sandbox pipe drain failures.
### What changed
- Added `worktree_skipped_no_git` and gated `sandbox_git_unavailable`
notices during initialization.
- Added `git_push_failed` and `parallel_base_checkpoint_failed` notices,
including redacted output tails where available.
- Logged GitHub token mint failures with a structured `error` field
before the existing notice.
- Plumbed `Emitter` and `StageScope` through `CodergenBackend::one_shot`
so the API backend emits the existing `agent.failover` event instead of
a duplicate tracing-only warning.
- Extracted sandbox pipe draining into a helper that warns on
stdout/stderr read failures, with unit coverage for the error path.
- Updated CLI snapshots for the new worktree warning in stderr and JSON
event output.
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
Return named PR content from the builder and keep title/body fallback logic inside the builder.
Move the PR body prompt to markdown and scale prompt truncation from model context windows.
Keep PR creation resilient when generated bodies are empty by emitting a reviewer-visible skeleton body.
Keep typed transport and provider errors intact through API, GitHub, OAuth, install, diagnostics, and artifact paths. Add regression coverage for cloned shared errors and communication error chains.
Replace the sandbox-side fast-import metadata writer with an in-process git2 writer that builds metadata commits locally and pushes them with worker-side GitHub credentials. Keep sandbox git probing separate from metadata runtime state so checkpoint commits and metadata snapshots have independent lifecycles.
Tighten the live Daytona timeout coverage so it proves timeout is represented as a missing exit code with the timed_out termination state, not just any non-success result.
Represent command termination explicitly across sandbox results, events,
run projections, API types, and the run stage UI. This removes the fake
-1 exit code path for timeout/cancel and lets consumers tell cancelled
commands apart from timed-out commands.
Persist command stdout/stderr through scratch logs and finalized CAS refs, expose byte-offset tailing through the API, and render separate streaming panels in the web run view.
Resolve command output blob refs for execution-time consumers such as edge routing and retros, and make Docker streaming timeout/cancel drain output before returning.
GitHubAppCredentials now carries the configured app slug, so the "not
installed" error from the installation lookup links to the specific
app's install page (https://github.com/organizations/{owner}/settings/apps/{slug}/installations)
when known, instead of the generic org installations page. Threaded
through the server, workflow pipeline, and CLI runner.
Also treat docker like daytona for GitHub credential gating: both are
clone-based providers that need an installation token to fetch the repo,
so a docker run now requires credentials when daytona would.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
WorkdirStrategy was structurally redundant with the existing
LocalSandboxLayer.worktree_mode config — Local sandboxes always picked
LocalWorktree, everything else picked Cloud, and the LocalDirectory arm
was only ever reachable via the parallel checkpoints_disabled bool.
resolve_worktree_plan now reads worktree_mode directly: Cloud sandboxes
return None with a pre_run_git base sha; Local + Never returns None
with no base sha; Local + non-Never builds the WorktreePlan as before.
RunOptions.checkpoints_disabled drops out: the lifecycle gate becomes
has_run_branch (git: None alone is the canonical "no git checkpoints"
signal), and tests/fixtures stop carrying the field.
Switch Docker sandboxes from host bind mounts to per-run clone-based containers with structured run metadata, reconnect validation, archive-based file transfer, and Docker resource defaults.
Extend run config/API surfaces so Docker image and clone settings flow through manifests, server preflight, workflow startup, and generated clients.
Update docs and tests for the new default Docker provider path.
Add fabro-static::EnvVars as the shared registry for fixed environment variable names and migrate env reads, clap env bindings, and subprocess/test allowlists to use it.
Add clippy bans for raw std::env lookup APIs so future dynamic env facades must be documented explicitly.
Public functions now take ctx: &GitHubContext<'_> instead of by-value
GitHubContext<'_>. Matches the surrounding &str / &GitHubCredentials
convention. The type stays Copy so internal call sites that pass `ctx`
through still work without explicit reborrows.
Touched: 8 fabro-github functions + matching _with_client variants,
plus call sites in fabro-server, fabro-workflow, fabro-sandbox, and
fabro-github's integration + unit tests. Pure mechanical change.
Verified: workspace fmt clean, clippy --all-targets -D warnings clean,
cargo nextest run --workspace 4581 passed, 182 skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build_pr_body and maybe_open_pull_request now take the two things they
actually need — run_store: &RunStoreHandle and llm_source: &dyn
CredentialSource — instead of services: &RunServices. The workflow
PULL_REQUEST phase decomposes services at the callsite; the standalone
fabro pr create command passes its own directly.
This removes RunServices::for_cli, a stub constructor that fabricated
an emitter, sandbox, and provider just to satisfy the RunServices type
for two fields it cared about. The "leaky fake" is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two cleanups:
1. Threaded GitHubContext through the remaining fabro-github functions
that pair credentials with the API base URL: branch_exists,
resolve_clone_credentials, resolve_authenticated_url. Each loses its
trailing `base_url: &str` and replaces `creds: &GitHubCredentials`
with `ctx: GitHubContext<'_>`. is_app_public was skipped — it doesn't
take credentials. Updated production callers in fabro-sandbox/daytona
and fabro-workflow/sandbox_git, plus integration and unit tests.
2. Added OpenPullRequestRequest::from_run_state on the workflow struct.
Bundles the validated unpacked-from-RunState pieces into a draft PR
request with the server's defaults (`draft = true`, `auto_merge =
None`). Server's create_run_pull_request handler now calls the
constructor instead of inlining a 12-field struct literal — the
handler reads as a sequence of validations followed by one named
request build, not as plumbing.
Verified: workspace fmt clean, clippy --all-targets -D warnings clean,
cargo nextest run --workspace 4587 passed, 182 skipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reconciles 61 origin commits (settings/config architectural reshape:
sparse layers → dense snapshots via builders, WorkflowSettings rename,
RunLayer/CliLayer moves, workflow builders, drop of public load wrappers)
with our LLM credential + RunServices refactor.
Our architecture preserved where it conflicted with origin's:
- RunServices / EngineServices stay (services.rs does not exist on
origin, which inlined the fields onto Initialized). Origin's new
Initialized fields (inputs, run_store, emitter, sandbox, registry,
env, dry_run, llm_client, provider) are absorbed through RunServices
and EngineServices instead of being inlined.
- llm_source: Arc<dyn CredentialSource> stays on AppState and
RunServices. Origin had a parallel ProviderCredentials struct in
fabro-server; our CredentialSource trait is more general and
complies with docs-internal/llm-client-resolution.md. Point-of-use
Client::from_source(...) rebuild preserves OAuth refresh.
- CommandContext.llm_source() uses self.storage_dir (origin's direct
field) instead of self.machine_settings (our side's field, removed
by origin).
- standalone_llm_source in fabro-agent drops the dead Result wrap and
uses fabro_config::user::default_storage_dir (origin's entrypoint)
instead of the removed load_settings_user/resolve_storage_root.
Absorbed from origin wholesale:
- SettingsLayer → WorkflowSettings rename everywhere
- Dense run settings: RunOptions.settings is WorkflowSettings, inputs
read via settings.run.inputs directly (not Option<RunLayer>)
- AppState.manifest_run_defaults / manifest_run_settings
- fabro_config re-exports of CliLayer/RunLayer/CliOutputLayer/etc.
- Lifecycle terminal-event changes, finalize dedup, list_events
consolidation — already brought in on the previous merge, kept
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Let callers decide whether to wrap in Arc. Also consolidates the two
state() fetches in build_pr_body into one.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename the last stale workflow test helpers and assertions that still used
pre-refactor checkpoint/retro file terminology, and update the retro docs
to describe the exported layout that now exists.
Implements the plan at
docs/plans/2026-04-20-003-refactor-unify-run-vocabulary-metadata-plan.md.
- Rename RunRecord to RunSpec and RunProjection.run to .spec everywhere
in Rust source, tests, helpers, test names, and error messages.
- Introduce SerializableProjection wrapper that trims bulky node text
fields (prompt, response, diff, stdout, stderr) for run.json snapshots.
- Collapse metadata-branch and CLI export to one RunDump::from_projection
builder emitting run.json + graph.fabro + stages/{stage_id}/... and
drop legacy top-level start/status/checkpoint/sandbox/retro/conclusion
split files.
- Replace MetadataStore::write_checkpoint with write_snapshot returning
the commit SHA; add read_run_projection/read_run_spec; demote
read_checkpoint/read_start_record to projection-field extractors.
- Switch fork, rewind, rebuild_meta, CLI rewind recovery, and retro
upload to read the unified projection layout.
- Add additive query methods on RunSpec and RunProjection.
Serde-level `alias = "spec"` shim dropped; `rename = "run"` retained to
keep the server API wire format stable per the plan's scope boundary.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enable clippy::allow_attributes_without_reason at the workspace level.
Add concise, callsite-specific reasons to existing allow attributes, including generated code paths.
Phase 2/3 of the std::fs lint initiative (Phase 1 refactors landed in
commit 9d1c0d98c).
clippy.toml additions (appended to disallowed-methods):
std::fs::read, read_to_string, write, read_dir, copy, canonicalize
std::fs::File::open, File::create, File::create_new
std::fs::OpenOptions::open
File::options was deliberately excluded — it returns an OpenOptions
builder with no syscall. OpenOptions::open is where the block happens.
Non-blocking std::fs items (metadata, exists, create_dir_all, remove_*,
rename, and all std::fs types) remain legal.
Annotation policy (per updated plan):
- Mixed async/sync production source: function- or statement-scoped
#[expect(...)] so future accidental Tokio-path regressions in the
same file still fire.
- Fully-sync production source, test modules, integration tests,
build.rs: file-level #![expect(...)].
- Every #[expect] has a specific reason identifying the sync context.
Annotations added in ~90 files across the workspace. Notable narrow
placements: fabro-server server.rs current_server_target,
build_disk_usage_response, create_test_app_state_with_session_key;
fabro-server install.rs read_to_string rollback snapshot;
fabro-sandbox local.rs list_recursive; fabro-agent cli.rs FOLLOW-UP on
the JSON-stdout writer; fabro-llm providers/common.rs FOLLOW-UP for
load_file_as_base64 (7 translator call sites; revisit if file:// URL
usage grows).
build.rs blanket allows: fabro-api/build.rs, fabro-util/build.rs.
Pre-existing unrelated nightly-clippy warnings fixed under scope:
fabro-sandbox sandbox_spec.rs (unused_imports, unused_async),
reconnect.rs (unused_variables, unused_async).
Verified: cargo +nightly-2026-04-14 clippy --workspace --all-targets
-- -D warnings passes; fmt clean; 4129/4131 tests pass (two known
flakes under parallel nextest load, both pass individually and are
unrelated to this change).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- rust.yml: move clippy to nightly-2026-04-14 (was stable); also pin
fmt to the same nightly date for consistency. Both jobs now use the
dated nightly and the run-step uses `cargo +nightly-2026-04-14 ...`.
- AGENTS.md: update developer commands to match CI.
- Duration constructors: replace `Duration::from_secs(N * 60)` /
`Duration::from_millis(N * 1000)` with `from_mins` / `from_secs` /
`from_hours` across the workspace to satisfy clippy's new
`duration_suboptimal_units` lint. std::time::Duration only — custom
`settings::duration::Duration` sites kept on `from_secs`.
- map/unwrap_or cleanup: `.map(f).unwrap_or(v)` → `.map_or(v, f)`,
`.map(f).unwrap_or(false)` on Result → `.is_ok_and(f)`, per
`clippy::map_unwrap_or`.
- Misc lints: collapse nested `if` into match guard in
handler/llm/api.rs and run_state.rs; replace `columns.len() > 0`
with `!columns.is_empty()`; switch a pair of `sort_by` calls to
`sort_by_key`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Daytona SDK client was created via Client::new() which only reads
DAYTONA_API_KEY from process env vars. When the key is stored in the
fabro vault (via `fabro secret set`), it was never forwarded to the SDK,
causing "api_key or jwt_token must be provided" errors.
Thread the API key from the vault through SandboxSpec, DaytonaSandbox,
and reconnect paths so the SDK receives it via new_with_config().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When `disk_cache = true` in `[server.slatedb]`, Fabro enables SlateDB's
object-store cache at `<storage_root>/cache/slatedb`, caching raw S3
bytes on local disk to reduce read latency. All cache parameters use
SlateDB defaults (16 GB max, 4 MB parts). A warning is emitted if
enabled with `provider = "local"` since the cache adds overhead when
the object store is already on the local filesystem.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move async subprocess paths to Tokio or spawn_blocking, document the
intentional synchronous std::process::Command callsites, and make CI run
Clippy with --all-targets so the guardrail applies to test code too.
Add a Clippy disallowed-methods guardrail for std::thread sleep/spawn
and convert the CLI polling paths to tokio::time::sleep so they no
longer block Tokio workers. Keep the intentional OS-thread sites with
narrow #[expect(...)] annotations that explain why std::thread is
required there.
Add the shared fabro-http transport crate and route hand-written HTTP client construction through it.
Use FABRO_HTTP_PROXY_POLICY for test no-proxy defaults, remove direct reqwest deps from ordinary crates, and add clippy bans for raw reqwest entrypoints.
Make gh_cli the default GitHub integration path across install, server,
workflow, and CLI surfaces while keeping app-based setup available when
explicitly selected.
Also defer GitHub reqwest client initialization until an HTTP request is
actually needed so missing-token and token-only paths do not trip workspace
test slow timeouts.
No production deployments exist, so there's no need for migration shims.
Remove all six backwards-compat type aliases (AgentError, SdkError,
CoreError, GraphvizError, StoreError, FabroError) and migrate ~880
callsites to use the canonical Error name directly within each crate,
or qualified imports (e.g., `use fabro_llm::Error as LlmError`) for
cross-crate references. Also fix a pre-existing absolute-path clippy
lint in fabro-server error.rs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
