Create run-owned sandbox lifecycle operations so terminal runs stop by default, resumes attach and start persisted sandboxes, and run deletion deletes or hands off provider resources according to preserve settings.
Emit agent.interrupt.injected when run interrupts reach active agent sessions, persist the stage/session fields, and refresh/render those rows in the Transcript tab.
Add a progressive human interview workflow and teach human gates to honor explicit question_type values so the workflow can exercise yes/no, confirmation, multiple-choice, multi-select, and freeform prompts before summarizing the answers.
Route command stderr into stdout at execution time and expose a single output log across events, projections, API clients, and the web UI. Keep replay compatibility for older command.completed events that still contain split stdout/stderr fields.
Compute cheap diff stats on checkpoint and terminal events, roll them into run summaries, and use them for the Files Changed tab badge without fetching full file diffs.
The billing rollup filtered out the synthetic exit handler but left
start visible, so the billing tab showed an asymmetric pair. Both are
no-op workflow boundary handlers and shouldn't appear as billable
stages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace mint_github_token's hand-rolled Pat/Installation/App match with
GitHubCredentials::resolve_bearer_token, removing a near-duplicate of
the same logic already in run_metadata::mint_token.
- Parallelize read_many_files via futures::future::join_all so the tool
actually reads concurrently — previously serial despite the name.
- Replace .expect() on the post-refresh GitHubTokenSource cache with a
proper anyhow error so a refresh edge case can't panic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Split PATs from installation access tokens so static configuration cannot accidentally store expiring ghs_* credentials. Workflow command and API agent stages now resolve GITHUB_TOKEN lazily from a refreshable source, while CLI agent stages surface their launch-time refresh limitation.
## Summary
This makes the advertised mid-run steering path real: users can send
append or interrupt steering messages through the API, CLI, and web UI,
and the worker delivers them to live API-mode agent sessions or buffers
them for the next session. The change adds the control protocol, session
interrupt machinery, workflow hub, server route/OpenAPI/client updates,
and UI feedback needed for the whole path.
### Plan Summary
- Add `SteerKind`/`run.steer` wire protocol and `POST /runs/{id}/steer`
- Deliver steers through subprocess JSONL or the in-process
`SteeringHub`
- Support append and interrupt behavior in agent sessions, with bounded
buffering and events
- Expose steering in the CLI/web UI and surface SSE toasts
## Flow
```mermaid
flowchart TB
UI["CLI / Web UI"] --> API["POST /runs/{id}/steer"]
API -->|"subprocess transport"| Control["Worker control JSONL"]
API -->|"in-process transport"| Hub["SteeringHub"]
Control --> Hub
Hub -->|"active API sessions"| Session["SessionControlHandle"]
Hub -->|"no active session"| Pending["Pending buffer"]
Pending -->|"first future API session"| Session
Session --> Agent["Session round loop"]
Agent --> Events["RunEvent stream"]
Events --> UI
```
## What changed and why
- Agent sessions now expose a lightweight `SessionControlHandle`, drain
steering at the top of each round, and use a replaceable round
cancellation token for interrupts. LLM waits are cancelled promptly,
while tool execution observes cancellation cooperatively so every
committed `tool_use` still gets a matching `tool_result`.
- `SteeringHub` owns active API session registration, broadcast
delivery, pending buffering, FIFO queue caps, and steering
lifecycle/drop events. A completion coordinator closes the
final-response race without introducing a workflow dependency into the
agent crate.
- The server route replaces the 501 stub, validates run state and
best-effort CLI-only steerability, and forwards through either
subprocess control JSONL or the in-process hub. OpenAPI and generated
clients now include the request type.
- The CLI and web UI can send append or interrupt steers. Run detail and
board views open the new composer, and shared SSE subscriptions now
support per-subscriber event callbacks so invalidation and steering
toasts can coexist on one EventSource.
## Review notes
- Steering actors stay on top-level `RunEvent.actor`; event props only
carry steering kind/drop metadata.
- Buffered steers replay as append messages to the first API session
that registers after an empty-active period. Per-stage targeting remains
out of scope.
- CLI-mode agent stages are still not steerable; the server returns a
best-effort 409 when all active agent stages are CLI-mode, while the
worker hub remains the authoritative safety net.
- No persistence or schema migration is required; active and pending
steering state is in memory.
- New tests focus on protocol round-trips, hub buffering/bounds, session
steering-loop behavior, SSE fanout, and basic server rejection paths.
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Run cancellation now reaches in-flight agent work instead of waiting for
an agent stage to finish or recording cancellation as a failed stage.
The workflow cancellation primitive is now
`tokio_util::sync::CancellationToken`, with child tokens passed through
setup, handlers, manager-loop child runs, sandbox streaming commands,
CLI agent invocations, and API agent sessions.
### Plan Summary
- Promote run cancellation to `CancellationToken` while keeping stall
timeout separate.
- Route CLI agents through cancellable sandbox streaming with optional
timeouts.
- Bridge run cancellation into API sessions and preserve
`Error::Cancelled` propagation.
- Add typed events/projections for CLI cancellation and timeout.
## Cancellation flow
```mermaid
flowchart TB
RunToken[Run CancellationToken]
Executor[Core executor]
Services[RunServices]
Manager[Manager-loop child run]
CLI[Agent CLI backend]
API[Agent API backend]
Sandbox[Sandbox streaming exec]
Session[fabro-agent Session]
RunToken --> Executor
RunToken --> Services
Services -- child_token --> Manager
Services -- child_token --> CLI
CLI -- child_token --> Sandbox
Services --> API
API -- bridge guard --> Session
```
## What changed and why
- `RunOptions`, `RunServices`, core `ExecutorOptions`, CLI/server run
state, and detached-run guards now use `CancellationToken` instead of
`Arc<AtomicBool>`. Dropping services or tokens still does not mean
cancellation; only explicit `.cancel()` does.
- Manager-loop child workflows are given child tokens so parent
cancellation propagates down, while stop/max-cycle cancellation remains
scoped to the child workflow.
- Stall timeout remains intentionally separate as a stall token and
still returns `Error::StallTimeout { node_id }`, not `Error::Cancelled`.
- Agent, prompt, human, fan-in, and parallel handler paths now pass
cancellation tokens through and avoid converting `Error::Cancelled` into
normal failed outcomes.
## Agent backend behavior
CLI-mode agents no longer launch detached `setsid` jobs with temp
stdout/stderr/exit-code polling. They run through
`Sandbox::exec_command_streaming` with a child token; a missing node
timeout passes `None` to preserve the existing unbounded agent runtime,
while explicit node timeouts still apply. Cancelled CLI runs emit
`agent.cli.cancelled`, clean temp files, and return `Error::Cancelled`;
timed-out CLI runs emit `agent.cli.timed_out` and return a handler
timeout error; `agent.cli.completed` remains natural-exit only.
API-mode agents install a per-invocation `SessionCancelBridgeGuard`
after acquiring a fresh or cached session. The guard maps the run token
into the session interrupt reason and session cancel token, and aborts
stale bridge tasks before session replacement or cache reinsertion so
reused sessions are not tied to old run tokens. `Session::initialize`
now returns `Result`, and project-doc, skill, MCP, and environment
discovery paths check cancellation and pass child tokens to sandbox
commands.
## Sandbox and event model
`Sandbox::exec_command_streaming` now accepts `Option<u64>` for timeout.
Production streaming implementations use a pending future for `None`
instead of a giant sleep, while the trait fallback maps `None` to
`u64::MAX` only when delegating to non-streaming `exec_command`.
The run event model now includes typed `agent.cli.cancelled` and
`agent.cli.timed_out` payloads with stdout, stderr, and duration, plus
conversion and projection support. OpenAPI/client regeneration was
unnecessary because the API schema already models run events with a free
event string and arbitrary properties; only Rust event types changed.
## Reviewer notes
Expect signature churn around `Session::initialize`,
`CodergenBackend::run`, `RunOptions.cancel_token`,
`StartServices.cancel_token`, and `Sandbox::exec_command_streaming`. The
main behavioral checks are that user cancellation reaches in-flight
CLI/API work and that timeout/stall paths remain distinct from user
cancellation.
### Fabro Details
<details>
<summary>Ran 9 stages in 117m 40s for $150.32</summary>
| Stage | Duration | Cost | Retries |
|---|---|---|---|
| start | 0s | – | 0 |
| toolchain | 1s | – | 0 |
| preflight_compile | 2m 8s | – | 0 |
| preflight_lint | 2m 13s | – | 0 |
| implement | 77m 12s | $56.78 | 0 |
| simplify_opus | 18m 5s | $5.83 | 0 |
| simplify_gpt | 15m 33s | $87.71 | 0 |
| verify | 1m 48s | – | 0 |
| fmt | 2s | – | 0 |
| **Total** | **117m 40s** | **$150.32** | **0** |
</details>
<details>
<summary>Ran <code>ImplementPlan.fabro</code> (12 nodes and 15
edges)</summary>
```dot
digraph ImplementPlan {
graph [
goal="Implement and simplify",
model_stylesheet="
* { model: claude-opus-4-7; }
"
]
rankdir=LR
start [shape=Mdiamond, label="Start"]
exit [shape=Msquare, label="Exit"]
toolchain [label="Toolchain", shape=parallelogram, script="command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1", max_retries=0]
preflight_compile [label="Preflight Compile", shape=parallelogram, script="cargo check -q --workspace 2>&1", max_retries=0]
preflight_lint [label="Preflight Lint", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1", max_retries=0]
fix_lints [label="Fix Lints", prompt="The preflight lint step failed. Read the build output from context and fix all clippy lint warnings.", max_visits=3]
implement [label="Implement", prompt="Read the plan file referenced in the goal and implement every step. Make all the code changes described in the plan. Use red/green TDD."]
simplify_opus [label="Simplify (Opus)", prompt="@prompts/simplify.md"]
simplify_gpt [label="Simplify (GPT-55)", prompt="@prompts/simplify.md", model="gpt-55"]
verify [label="Verify", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --cargo-quiet --workspace --status-level fail 2>&1 && cargo dev docs refresh 2>&1 && cargo dev docs check 2>&1", goal_gate=true, retry_target="fixup"]
fixup [label="Fixup", prompt="The verify step failed. Read the build output from context and fix all clippy lint warnings, test failures, and generated docs errors.", max_visits=3]
fmt [label="Format", shape=parallelogram, script="cargo +nightly-2026-04-14 fmt --all 2>&1", max_retries=0]
start -> toolchain
toolchain -> preflight_compile [condition="outcome=succeeded"]
toolchain -> exit
preflight_compile -> preflight_lint [condition="outcome=succeeded"]
preflight_compile -> exit
preflight_lint -> implement [condition="outcome=succeeded"]
preflight_lint -> fix_lints
fix_lints -> preflight_lint
implement -> simplify_opus -> simplify_gpt -> verify
verify -> fmt [condition="outcome=succeeded"]
verify -> fixup
fixup -> verify
fmt -> exit
}
```
</details>
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
### Summary
Silent fallback paths now emit stable warnings instead of degrading
without a user-visible signal. The fallback behavior is unchanged; runs
still continue, but worktree, Git, checkpoint, and LLM failover issues
now show up in the run feed and logs.
### Plan Summary
- Emit run notices for workflow Git/worktree fallback paths.
- Reuse the existing failover event for one-shot LLM provider fallback.
- Add tracing for sandbox pipe drain failures.
### What changed
- Added `worktree_skipped_no_git` and gated `sandbox_git_unavailable`
notices during initialization.
- Added `git_push_failed` and `parallel_base_checkpoint_failed` notices,
including redacted output tails where available.
- Logged GitHub token mint failures with a structured `error` field
before the existing notice.
- Plumbed `Emitter` and `StageScope` through `CodergenBackend::one_shot`
so the API backend emits the existing `agent.failover` event instead of
a duplicate tracing-only warning.
- Extracted sandbox pipe draining into a helper that warns on
stdout/stderr read failures, with unit coverage for the error path.
- Updated CLI snapshots for the new worktree warning in stderr and JSON
event output.
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
Return named PR content from the builder and keep title/body fallback logic inside the builder.
Move the PR body prompt to markdown and scale prompt truncation from model context windows.
Keep PR creation resilient when generated bodies are empty by emitting a reviewer-visible skeleton body.
Keep sandbox exec failures structured until event emission so git push, checkpoint, notice, and retro failures can expose redacted output tails without expanding their terse error strings.
Also add log rendering that appends sanitized tail content for exec-backed errors while preserving the existing safe Display behavior.
Emit snapshot slow-path events only when Docker or Daytona actually performs image or snapshot work, replace retired completion markers with snapshot.ready, and render the lifecycle in attach/log output.
So that CLI and other API consumers can surface a clickable link to the
run's web UI page instead of guessing route shapes or probing settings.
The server populates `web_url` from `server.web.enabled` and
`server.web.url`, returns it on `RunStatusResponse` (create plus all
lifecycle transitions), and persists it on the `run.created` event so
attach replays the same link without re-deriving it.
CLI: prints `Web UI: <url>` as a run-header info line, driven off the
replayed event so fresh runs and `attach` share one code path. Absent
when the UI is disabled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Keep fabro_workflow::event as the public facade while moving event conversion, names, redaction, sink, emitter, stored-field helpers, and StageScope into focused modules. Co-locate the existing event tests with the moved code and update the events strategy docs for the new module layout.
Tighten auth state to remove impossible identity branches and stringly error codes.
Route worker JWTs by header metadata, avoid unnecessary auth context cloning, and reuse shared helpers across tests and Slack payload handling.
Carry typed timeout actor metadata through failures instead of deriving it from display text.
Replace hand-built RequestAuthContext literals in github_webhook with the
existing ::invalid()/::authenticated() constructors, collapse the duplicate
demo/real principal layers into a single cloneable layer, and forward the
_with_anyhow error constructors to their _with_source twins to drop the
duplicated cause-collection bodies. refresh_credential_from_headers now
reuses jwt_auth::bearer_token_from_headers for Authorization parsing.
test_support shares one TEST_DEV_TOKEN-derived bearer header instead of a
hand-pasted literal. Replace for-loops in principal/cli_flow tests with
per-variant cases to honor the no-loops-in-tests rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Defer current_visit_for to the fallback branch in stage_entry_with_current_visit
so events that already carry stage_id avoid an O(N stages) scan per event.
Run state() and list_events() concurrently in build_conclusion_from_store, and
share a single retry- segment prefix between encode and decode.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove unused RunProjection::stage_mut, share decode_retry_and_filename
between artifact_store decoders, reuse stage_visit() in the artifact
lifecycle, and cache the dump.log entry index so RunDump::add_orphan_notice
no longer rescans entries on every call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Expose `fabro_types::first_event_seq` next to `StageProjection`, replacing
six identical `nonzero` test helpers and the private one in `run_state`.
Add `RunProjection::iter_stages_mut` so `SerializableProjection` can
clear bulky fields without the collect-then-lookup dance, and let the
`fabro-dump` loop iterate `(&StageId, &StageProjection)` borrows directly
to drop the per-stage `StageId::clone()` and redundant HashMap lookup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Keep typed transport and provider errors intact through API, GitHub, OAuth, install, diagnostics, and artifact paths. Add regression coverage for cloned shared errors and communication error chains.
write_snapshot_blocking now derives entry_count and bytes from the
entries slice instead of taking them as parameters. The arity drops
from five to three, and the cheap O(n) work moves off the async
runtime into spawn_blocking where the rest of the snapshot already
runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make GitHubCredentials::resolve_bearer_token public and call it from
run_metadata::mint_token instead of re-implementing the JWT-sign +
installation-token branch. Eliminates the unreachable!() that arose from
matching the same enum twice.
Also drop the metadata_ field-name prefix on RunMetadataRuntime fields
(degraded, warning_emitted) — the prefix is redundant inside a struct
already named RunMetadataRuntime. Method names unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Origin advanced 11 commits in parallel, including refactors that
restructured the now-deleted sandbox_metadata fast-import writer
(structured ExecFailure for push errors, redacted_output_tail helper,
RunDump moved to fabro-dump crate, RunDump::from_projection now returns
Result, MetadataSnapshotFailureKind::Write, MetadataSnapshotFailed event
gains exec_output_tail, RunStoreBackend gains read_run_log).
Resolution: take ours for the four metadata-writer files (sandbox_metadata
deleted, lifecycle/git.rs, pipeline/finalize.rs, sandbox_git.rs) since the
git2 writer supersedes that module. Fold origin's API changes into the
ours-side: switch to fabro_dump::RunDump, handle from_projection's Result,
populate exec_output_tail: None in MetadataSnapshotFailed (git2 push
failures have no exec stdout/stderr), implement read_run_log on test
mocks. Drop unused from_raw_entries from fabro-dump.
A follow-up will port the structured push-failure pattern to run_metadata
without widening ExecFailure to non-exec ops.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adapt the from_projection error branches in lifecycle/git.rs (init + checkpoint phases) and pipeline/finalize.rs to the new free-function emit_metadata_snapshot_failed and MetadataSnapshotFailure struct introduced in 543725752. The merge auto-resolved cleanly but left the dump-error sites on the deprecated method/positional-args signature.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the hand-rolled tree builder in run_metadata with
fabro-checkpoint's Store::write_blob/write_tree/write_commit/update_ref,
deleting BuildTreeError, TreeNode, build_tree, insert_tree_node, and
write_tree_node. Also fold three smaller duplications: the identical
metadata_writer_for_repo test helpers in lifecycle/git.rs and
pipeline/finalize.rs become RunMetadataWriterHandle::new_for_test_repo,
sandbox_git_runtime reuses sandbox_git::exec_err, and METADATA_PERMISSIONS
is a LazyLock instead of being rebuilt per snapshot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
push_json_entry and push_json_entry_path silently dropped entries via if let Ok(...) on serde_json::to_value, hiding any future Serialize impl failure as missing files. They now return Result, RunDump::from_projection returns Result<Self>, and the three production callers (pipeline/finalize, lifecycle/git init + checkpoint) report failures via emit_metadata_snapshot_failed with MetadataSnapshotFailureKind::Write.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the synthetic ExecResult fabricated in sandbox_metadata::stdout_output_tail
just to reach private redaction logic. The redact + sanitize + tail pipeline
now lives behind fabro_sandbox::redacted_output_tail(stdout, stderr, max),
which ExecResult::redacted_output_tail also delegates to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Groups the MetadataSnapshotFailed event payload into a MetadataSnapshotFailure
struct and replaces the two near-identical 11-arg emit_metadata_snapshot_failed
helpers in lifecycle/git.rs and pipeline/finalize.rs with one shared helper
in sandbox_metadata.rs. Both #[allow(too_many_arguments)] blocks are removed.
Also deletes two hand-written floor_char_boundary copies (fabro-agent and
fabro-sandbox) in favor of the stable str::floor_char_boundary, matching how
most existing call sites already use it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sandbox::write_file already creates parent dirs in every backend (local, docker, daytona), so the per-file mkdir -p exec_command in upload_data_files was a wasted round-trip. Also remove the run_dir param/field that became unused after RunDump took over hydration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collapses four duplicate tracing field blocks in event.rs behind
ExecOutputTail::trace_summary, drops the parallel MetadataPushError
struct in favor of reusing SandboxMetadataError::Operation, inlines
the single-use Error::exec_result accessor, and gates the test-only
ExecResult::from_process_output to cfg(test).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add bounded redacted exec output tails to failure events while keeping tracing log-safe. Centralize tail projection on ExecResult and thread diagnostics through metadata, setup, devcontainer, and CLI install failures.
Replace the sandbox-side fast-import metadata writer with an in-process git2 writer that builds metadata commits locally and pushes them with worker-side GitHub credentials. Keep sandbox git probing separate from metadata runtime state so checkpoint commits and metadata snapshots have independent lifecycles.
