Commit graph

123 commits

Author SHA1 Message Date
Bryan Helmkamp
eb4891b1b0
refactor(agent): simplify reviewed changes
Use raw sandbox reads for memory and skills, keep line-numbered reads focused on display, and share retry-delay handling across agent and LLM code.

Trim task tool descriptions, bound multi-file read concurrency, restore Docker's text read path, and add the reviewed implementation plan docs.
2026-05-22 21:51:45 -04:00
Bryan Helmkamp
d1cc47324d
fix(agent): use raw sandbox reads for edits
Separate raw file reads from the line-numbered display API so apply_patch and edit_file operate on unformatted UTF-8 content. Keep read_file/read_many_files model-facing output numbered and cover regressions for prefix corruption.
2026-05-22 21:26:36 -04:00
Bryan Helmkamp
12e665c341
fix(sandbox): derive daytona network policies (#341)
Fixes Daytona sandbox network policy rendering so absent allow lists
with `networkBlockAll=false` are reported as open egress instead of
unknown, while Daytona ingress is always reported as blocked.

The mapper still reports blocked egress when Daytona blocks all
networking and CIDR allow-list egress when `networkAllowList` is
present. Tests cover blocked, allow-list, empty allow-list, and default
Daytona network data.

Verified with `cargo nextest run -p fabro-sandbox --features daytona`
and `cargo +nightly-2026-04-14 fmt --check --all`.

---

[![Compound
Engineering](https://img.shields.io/badge/Compound_Engineering-6366f1)](https://github.com/EveryInc/compound-engineering-plugin)
🤖 Generated with GPT-5 via [Codex](https://openai.com/codex)
2026-05-21 14:36:04 -04:00
Bryan Helmkamp
5eb874b55c
feat(sandbox): label Daytona sandboxes as managed (#326)
## Summary

Fabro-created Daytona sandboxes now carry the same managed-resource
labels Docker containers already use: `sh.fabro.managed=true` and
`sh.fabro.run_id=<run-id>` when a run id is available.

This moves the Docker label constants into a shared sandbox helper,
keeps Docker behavior unchanged, and applies the helper when Daytona
create params are built. User-provided Daytona labels are preserved, but
Fabro's reserved keys are authoritative on collisions. Daytona snapshot
behavior is unchanged because the snapshot API does not expose labels.

## Testing

- `cargo test -p fabro-sandbox managed_labels --no-default-features
--features docker,daytona`
- `cargo test -p fabro-sandbox
docker::tests::real_run_container_gets_name_and_labels
--no-default-features --features docker`
- `cargo test -p fabro-sandbox daytona::tests::base_params
--no-default-features --features daytona`
- `cargo test -p fabro-sandbox daytona_managed_labels_live_smoke
--no-default-features --features daytona`
- `cargo test -p fabro-sandbox --no-default-features --features
docker,daytona`
- `cargo +nightly-2026-04-14 fmt --check --all`
- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D
warnings`
- `cargo +nightly-2026-04-14 clippy -p fabro-sandbox --all-targets
--no-default-features --features docker,daytona -- -D warnings`

The live Daytona smoke test remains ignored; it compiles under the
Daytona feature but was not run against live credentials.

## Post-Deploy Monitoring & Validation

- Log queries/search terms: `Failed to create Daytona sandbox`,
`Daytona`, `labels`, `sh.fabro.managed`, `sh.fabro.run_id`, and sandbox
initialization errors for `provider=daytona`.
- Metrics or dashboards: Daytona sandbox creation success/error rate,
Fabro run initialization failures for Daytona runs, and Daytona resource
inventory filtered by `sh.fabro.managed=true`.
- Expected healthy signals: new Fabro-created Daytona sandboxes include
`sh.fabro.managed=true`, run-owned sandboxes include the matching
`sh.fabro.run_id`, user labels remain visible, and Daytona sandbox
creation failure rates stay at baseline.
- Failure signals and rollback trigger: any sustained increase in
Daytona sandbox creation failures, API validation errors around labels,
or missing managed labels on newly created sandboxes. Roll back this PR
or hotfix the label merge to omit Daytona labels if Daytona rejects the
keys in production.
- Validation window and owner: release owner watches the first 24 hours
after deploy, with an immediate manual Daytona dashboard/API spot-check
after the first managed Daytona run.

---

[![Compound
Engineering](https://img.shields.io/badge/Compound_Engineering-6366f1)](https://github.com/EveryInc/compound-engineering-plugin)
🤖 Generated with GPT-5 (context unknown, reasoning enabled) via
[Codex](https://openai.com/codex)
2026-05-20 17:22:12 -04:00
Bryan Helmkamp
ae55bded81
fix(sandbox): clone Daytona repos under /home/daytona/repos (#285)
## Summary

Daytona's default snapshot runs as the `daytona` user (uid 1001), which
lacks write permission on `/`. With `run.clone.enabled = true`, sandbox
init failed at `fs.create_folder("/repos", ...)` with HTTP 400, before
the first workflow stage could run:

```
sandbox.git.failed error="Failed to create Daytona repos root" causes=["HTTP 400"]
run.failed
```

Root cause: the Daytona provider was using Docker's root-level `/repos`
layout. Docker works because its containers run as root; Daytona's
default sandbox user does not.

**Fix:** move `REPOS_ROOT` for Daytona to `/home/daytona/repos`,
alongside the existing `/home/daytona/workspace`. The path is writable
by the default sandbox user, the symlink layout is unchanged
(`/home/daytona/workspace/<repo>` →
`/home/daytona/repos/<owner>/<repo>`),
and Docker keeps its existing `/repos` path.

**Bonus — better error diagnostics.** A new `wrap_fs_error(operation,
path, error)` helper in the Daytona provider:

- includes the attempted path in the message (was just "Failed to create
  Daytona repos root" with no indication of which path);
- classifies HTTP 400 as a likely permission issue and points at
  snapshot configuration;
- classifies HTTP 401/403 as an API key permissions issue;
- preserves the underlying `DaytonaError` in the source chain
  (per `docs/internal/error-handling-strategy.md` — verified by walking
  `Error::source()` in the regression test).

So if this class of failure recurs (custom snapshot, future path
changes, ...) the user gets:

> Failed to create Daytona repos root '/home/daytona/repos' failed
> (HTTP 400). This usually means the sandbox user lacks write permission
> on the parent directory. If you're using a custom Daytona snapshot,
> ensure the sandbox user can write to '/home/daytona/repos', or use a
> path under the user's home directory (e.g. /home/daytona/...).

instead of:

> Failed to create Daytona repos root
> HTTP 400

## Test plan

- [x] `cargo build --workspace`
- [x] `cargo nextest run -p fabro-sandbox --features daytona` — 142/142
pass
- [x] `cargo nextest run -p fabro-types -p fabro-workflow` — 1365/1365
pass
- [x] New unit test `wrap_fs_error_classifies_http_400_and_403` —
asserts
      top-level message contains path + hint AND walks the source chain
      to prove `DaytonaError::Api { status_code: 400, .. }` is preserved
- [x] `cargo +nightly-2026-04-14 fmt --check --all`
- [x] `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D
warnings`
- [x] **Live regression**: `daytona_clone_layout_live_smoke` against the
      default `daytona-medium` snapshot — failed with `Failed to create
      Daytona repos root / HTTP 400` before the change; passes
      end-to-end after (provisions sandbox → clones repo → verifies
      symlink + HEAD match in 2.5s)

## Related

- Closes #284 (thanks @jessmartin for the report, diagnosis, and
proposed fix)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Jess Martin <27258+jessmartin@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 13:47:54 -04:00
Bryan Helmkamp
f790a47da4
feat(sandbox): surface provider links and network policy (#275)
## Summary
- Add provider dashboard URL reporting to `SandboxDetails`, including
Daytona dashboard links and a Sandbox tab provider link.
- Extend `SandboxDetails` with required provider-neutral public network
policy for egress and ingress allow/block rules.
- Populate local, Docker, and Daytona network policies from provider
details when Fabro can assert them, otherwise default to explicit
`unknown` policy.
- Update OpenAPI, Rust API replacements, generated TypeScript client
models, server/API tests, and the Sandbox tab `Network` panel.

## Notes
- This reports policy only; it does not probe live connectivity.
- The network model intentionally excludes ports, previews, IP
addresses, DNS, routes, Docker network IDs, and service discovery.
- Older persisted/API JSON still deserializes through the Rust serde
default for `network`.

## Verification
- `cargo build -p fabro-api`
- `cargo nextest run -p fabro-types -p fabro-api -p fabro-sandbox -p
fabro-server sandbox_details`
- `cargo test -p fabro-sandbox details --features docker,daytona`
- `cargo +nightly-2026-04-14 fmt --check --all`
- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D
warnings`
- `cd apps/fabro-web && bun test app/routes/run-sandbox.test.tsx`
- `cd apps/fabro-web && bun run typecheck`
- `cd apps/fabro-web && bun run build`
- `cd lib/packages/fabro-api-client && bun run typecheck`
- `git diff --check`

## Post-Deploy Monitoring & Validation
- Log queries/search terms: `Failed to inspect Docker container`,
`Daytona sandbox is not initialized`, `missing runtime metadata`, `GET
/api/v1/runs/*/sandbox`, `Sandbox unavailable`.
- Metrics/dashboards to watch: API 5xx/error rate for `GET
/api/v1/runs/{id}/sandbox`, frontend error reporting for the Sandbox
tab, and provider reconnect/inspect failures.
- Expected healthy signals: Sandbox tab loads with Overview, Resources,
Network, Labels, and Timestamps; Daytona sandboxes show provider links;
local/ambiguous network policy shows `Unknown`; Docker `network_mode =
none` shows `Blocked`.
- Failure signals and rollback trigger: sandbox details deserialization
errors, missing `network` crashes, sustained sandbox endpoint 5xx
increase, or blank Sandbox tab after deploy. Roll back this PR or hide
the Network panel if API/client shape issues appear.
- Validation window and owner: first 24 hours after deploy, release
owner/on-call.

## Compound Engineering
- Implemented with OpenAI Codex CLI on GPT-5.
2026-05-16 10:16:18 -04:00
Bryan Helmkamp
e7e4fb5ca1
Add Daytona volume mount passthrough (#263)
## Summary

This adds run-configuration support for mounting existing Daytona
volumes into Fabro-managed Daytona sandboxes.

Concretely, this PR:

- adds `[[run.sandbox.daytona.volumes]]` with `volume_id`, `mount_path`,
and optional `subpath`
- resolves that config through the layer/settings/runtime pipeline
- forwards configured mounts to `daytona_sdk::SandboxBaseParams.volumes`
when creating the sandbox
- documents the configuration surface in the Daytona environment and run
configuration docs

## Motivation

Daytona already supports attaching volumes when a sandbox is created,
but Fabro currently owns that sandbox creation call. That means users
cannot attach a pre-created Daytona volume to a Fabro-managed sandbox
from run config.

The intended use is persistent, provider-owned state such as agent
credentials, caches, datasets, or other files that should survive
ephemeral sandbox lifecycles.

## Scope

This is intentionally a narrow passthrough. Fabro does not create,
delete, list, wait on, or otherwise manage Daytona volume lifecycle.
Users create the volume in Daytona first, then reference its `volume_id`
from Fabro run config.

`volumes` defaults to an empty list in resolved settings for backwards
compatibility with existing serialized settings.

## Testing

- `cargo test -p fabro-server
runtime_daytona_config_preserves_volume_mounts`
- `cargo test -p fabro-config resolves_daytona_volume_mounts`
- `cargo test -p fabro-sandbox --features daytona volume_mounts`
- `cargo test -p fabro-workflow
runtime_daytona_config_preserves_volume_mounts`
- `cargo check -p fabro-server`

---

_Re-opened from #262 (originally by @kimprobably) to land a rustfmt fix
— the original PR came from an org-owned fork, which blocks maintainer
pushes. Branch is now on the base repo. Original commit preserved; one
additional commit fixes rustfmt formatting._

Co-authored-by: Tim Keen <tim@keen.digital>
2026-05-14 10:54:43 -04:00
Bryan Helmkamp
c0fe29390a
feat(sandbox): prepare clone layout for multi-repo runs (#250)
## Summary

- Clone primary GitHub repos into provider-owned `/repos/{owner}/{repo}`
paths for Docker and Daytona sandboxes.
- Keep user/agent execution rooted at the workspace symlink, e.g.
`/workspace/{repo}` or `/home/daytona/workspace/{repo}`.
- Persist optional runtime layout metadata (`workspace_root`,
`repos_root`, `primary_repo_path`, `primary_repo_link`) through events,
projections, OpenAPI, Rust API tests, and the TS client.
- Preserve empty workspace behavior and reconnect from stored
`working_directory` for existing run records.

## Verification

- `cargo nextest run -p fabro-sandbox --features docker,daytona`
- `cargo nextest run -p fabro-workflow`
- `cargo nextest run -p fabro-server`
- `cargo build -p fabro-api`
- `cargo nextest run -p fabro-api run_sandbox_json_matches_openapi_shape
sandbox_details_json_matches_openapi_shape`
- `cd lib/packages/fabro-api-client && bun run typecheck`
- `cargo +nightly-2026-04-14 fmt --check --all`
- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D
warnings`
- `git diff --check`

## Notes

- Added ignored live smoke tests for Docker and Daytona layout
validation; they require real provider credentials/runtime.
2026-05-14 09:38:20 -04:00
Bryan Helmkamp
762ac19649
fix(sandbox): resolve bash via PATH instead of /bin/bash (#241)
## Summary

The local sandbox provider hard-codes `/bin/bash` at three call sites in
`fabro-sandbox/src/local.rs` (`exec_command`, `exec_command_streaming`,
`spawn_stdio_process`). NixOS doesn't ship `/bin/bash` — only `/bin/sh`
and `/usr/bin/env` are managed under `/`, with bash living on `PATH` at
`/run/current-system/sw/bin/bash`. The result: a first run on NixOS dies
on the very first sandbox call (the git probe) with `No such file or
directory (os error 2)`, surfaced as `sandbox git unavailable`.

## Fix

Switch all three sites from `Command::new("/bin/bash")` to
`Command::new("bash")`. `PATH` is already preserved by
`filtered_env_vars` (and explicitly tested at `local.rs:1164`), so
libc's `execvp` lookup resolves bash on every distribution that has it
installed, including NixOS, without forcing users to symlink
`/bin/bash`.

The `/bin/bash` references in `docker.rs` are unaffected — those execute
inside containers where the path always exists.

## Credit

Diagnosis and proposed fix by @allouis in #232 — they ran the
PATH-lookup variant locally on NixOS 26.05 and confirmed workflows ran
cleanly without the symlink workaround.

Closes #232

---

[![Compound Engineering
v2.60.0](https://img.shields.io/badge/Compound_Engineering-v2.60.0-6366f1)](https://github.com/EveryInc/compound-engineering-plugin)
🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via
[Claude Code](https://claude.com/claude-code)

Co-authored-by: Fabien O'Carroll <3218915+allouis@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 10:31:12 -04:00
Bryan Helmkamp
d6d7f18c23
fix(acp): tolerate clean stdio exit after final response 2026-05-12 09:03:53 -04:00
Bryan Helmkamp
234bd5663e
Add ACP backend support (#237)
## Summary
Implemented ACP support as a first-class Fabro backend alongside `api`
and `cli`. This adds a new `fabro-acp` crate using the official ACP Rust
crates, routes `backend=\"acp\"` for agent and prompt nodes, adds
sandbox stdio support for local/Docker/test-support paths, emits ACP
workflow events/projections, updates server steerability handling,
validation, documentation, and black-box CLI coverage.

## Test Plan
Passed strict non-live verification:
- `ulimit -n 4096 && cargo nextest run -p fabro-workflow --run-ignored
all --no-fail-fast` — 1162 passed, 0 skipped.
- `ulimit -n 4096 && cargo nextest run -p fabro-acp -p fabro-sandbox -p
fabro-workflow -p fabro-validate -p fabro-store -p fabro-server -p
fabro-cli --run-ignored all --no-fail-fast -E 'not
test(daytona_streaming_live_smoke)'` — 3125 passed.
- `cargo build --workspace` — passed.
- `ulimit -n 4096 && cargo nextest run --workspace --run-ignored all
--no-fail-fast -E 'not test(daytona_streaming_live_smoke)'` — 5666
passed.
- `cargo +nightly-2026-04-14 fmt --check --all` — passed.
- `cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D
warnings` — passed.

Live-environment tests skipped/excluded under explicit user override:
- `daytona_streaming_live_smoke` was excluded from final nextest runs
because it requires live Daytona infrastructure and `DAYTONA_API_KEY`.
- Confirmed with `env -u DAYTONA_API_KEY cargo test -p fabro-sandbox
--features daytona --test daytona_streaming_live
daytona_streaming_live::daytona_streaming_live_smoke -- --ignored
--exact --nocapture`: failed fast with `DAYTONA_API_KEY must be set to
run this live smoke test`.
2026-05-11 23:39:43 -04:00
Bryan Helmkamp
4fa4716015
refactor(core): simplify reviewed run cleanup
Reuse shared frontend formatting and SSE dedupe helpers, tighten typed sandbox handling, remove obsolete run DTOs, and collapse auth-session revoke into a single store operation.
2026-05-10 22:53:20 -04:00
Bryan Helmkamp
cccb557281
feat(api): unify public run shape
Return canonical Run payloads across run list, board, create, and lifecycle endpoints. Move archive state out of RunStatus and into lifecycle metadata, split sandbox runtime from planned sandbox data, and separate static pull request records from live pull request details.

Regenerate the TypeScript API client and migrate web, CLI, server, store, workflow, and API tests to the new contract.
2026-05-10 20:48:55 -04:00
Bryan Helmkamp
5209d05623
refactor(sandbox): unify run sandbox identity
Replace the separate sandbox record shape with a typed RunSandbox model shared by projections, API responses, and generated clients. The public contract now uses SandboxProvider plus a non-null id and working_directory, and removes sandbox identifier/name leakage.
2026-05-10 13:16:57 -04:00
Bryan Helmkamp
7ee1ef59f9
chore(sandbox): tighten details module imports for nightly clippy 2026-05-09 21:51:48 -04:00
Bryan Helmkamp
02d494c268
feat(sandbox,server,web): add sandbox_details inspection and unify SandboxResources
Adds fabro_sandbox::sandbox_details, a control-plane inspection function
that maps Local, Docker, and Daytona providers into a shared
SandboxDetails record (state, image, resources, labels, timestamps).

To avoid type sprawl, the demo board's SandboxResources is unified with
the new control-plane shape (cpu_cores: f64, memory_bytes: u64,
disk_bytes: u64). The runs board chip in apps/fabro-web converts
memory_bytes back to GB for display.
2026-05-09 21:40:19 -04:00
Bryan Helmkamp
61ab94195e
feat(sandbox): copy Docker exec access command 2026-05-09 17:05:46 -04:00
Bryan Helmkamp
fe77c2a688
fix(sandbox): hide Daytona terminal control frames 2026-05-09 16:18:58 -04:00
Bryan Helmkamp
66b11480d6
fix(sandbox): use Daytona toolbox proxy for terminals 2026-05-09 16:15:38 -04:00
Bryan Helmkamp
db3f348fab
feat(runs): add sandbox terminal
Expose a run-scoped websocket terminal for Docker and Daytona sandboxes, and add the web terminal route so sandbox-backed runs can be inspected interactively from the run detail page.
2026-05-09 16:06:04 -04:00
Bryan Helmkamp
2a8884883a
refactor(workflow): remove local worktree mode
Make local sandbox execution direct by removing the public worktree mode and in-place controls from CLI, config, run state, API surfaces, docs, and UI. Keep worktree support only for internal parallel-node isolation.
2026-05-09 11:04:23 -04:00
Bryan Helmkamp
6719a3cdac
refactor: simplify reviewed run cleanup
Reuse shared PR chip rendering, persist runtime stage handlers from events, and remove duplicated lifecycle/delete helpers found during review.
2026-05-08 23:29:08 -04:00
Bryan Helmkamp
79a095d7c6
fix(runs): restore terminal sandbox regressions
Emit local sandbox stop events so CLI follow and event-history tests can observe terminal cleanup under run-owned sandbox lifecycle. Fall back to stored file diffs when completed runs no longer have an active sandbox.
2026-05-08 22:41:42 -04:00
Bryan Helmkamp
d3e33ce32c
feat(runs): own sandbox lifecycle
Create run-owned sandbox lifecycle operations so terminal runs stop by default, resumes attach and start persisted sandboxes, and run deletion deletes or hands off provider resources according to preserve settings.
2026-05-08 18:38:08 -07:00
fabro-sh-0530[bot]
7cec7825d9
Cancel in-flight agent stages with CancellationToken (#211)
## Summary
Run cancellation now reaches in-flight agent work instead of waiting for
an agent stage to finish or recording cancellation as a failed stage.
The workflow cancellation primitive is now
`tokio_util::sync::CancellationToken`, with child tokens passed through
setup, handlers, manager-loop child runs, sandbox streaming commands,
CLI agent invocations, and API agent sessions.

### Plan Summary
- Promote run cancellation to `CancellationToken` while keeping stall
timeout separate.
- Route CLI agents through cancellable sandbox streaming with optional
timeouts.
- Bridge run cancellation into API sessions and preserve
`Error::Cancelled` propagation.
- Add typed events/projections for CLI cancellation and timeout.

## Cancellation flow
```mermaid
flowchart TB
  RunToken[Run CancellationToken]
  Executor[Core executor]
  Services[RunServices]
  Manager[Manager-loop child run]
  CLI[Agent CLI backend]
  API[Agent API backend]
  Sandbox[Sandbox streaming exec]
  Session[fabro-agent Session]

  RunToken --> Executor
  RunToken --> Services
  Services -- child_token --> Manager
  Services -- child_token --> CLI
  CLI -- child_token --> Sandbox
  Services --> API
  API -- bridge guard --> Session
```

## What changed and why
- `RunOptions`, `RunServices`, core `ExecutorOptions`, CLI/server run
state, and detached-run guards now use `CancellationToken` instead of
`Arc<AtomicBool>`. Dropping services or tokens still does not mean
cancellation; only explicit `.cancel()` does.
- Manager-loop child workflows are given child tokens so parent
cancellation propagates down, while stop/max-cycle cancellation remains
scoped to the child workflow.
- Stall timeout remains intentionally separate as a stall token and
still returns `Error::StallTimeout { node_id }`, not `Error::Cancelled`.
- Agent, prompt, human, fan-in, and parallel handler paths now pass
cancellation tokens through and avoid converting `Error::Cancelled` into
normal failed outcomes.

## Agent backend behavior
CLI-mode agents no longer launch detached `setsid` jobs with temp
stdout/stderr/exit-code polling. They run through
`Sandbox::exec_command_streaming` with a child token; a missing node
timeout passes `None` to preserve the existing unbounded agent runtime,
while explicit node timeouts still apply. Cancelled CLI runs emit
`agent.cli.cancelled`, clean temp files, and return `Error::Cancelled`;
timed-out CLI runs emit `agent.cli.timed_out` and return a handler
timeout error; `agent.cli.completed` remains natural-exit only.

API-mode agents install a per-invocation `SessionCancelBridgeGuard`
after acquiring a fresh or cached session. The guard maps the run token
into the session interrupt reason and session cancel token, and aborts
stale bridge tasks before session replacement or cache reinsertion so
reused sessions are not tied to old run tokens. `Session::initialize`
now returns `Result`, and project-doc, skill, MCP, and environment
discovery paths check cancellation and pass child tokens to sandbox
commands.

## Sandbox and event model
`Sandbox::exec_command_streaming` now accepts `Option<u64>` for timeout.
Production streaming implementations use a pending future for `None`
instead of a giant sleep, while the trait fallback maps `None` to
`u64::MAX` only when delegating to non-streaming `exec_command`.

The run event model now includes typed `agent.cli.cancelled` and
`agent.cli.timed_out` payloads with stdout, stderr, and duration, plus
conversion and projection support. OpenAPI/client regeneration was
unnecessary because the API schema already models run events with a free
event string and arbitrary properties; only Rust event types changed.

## Reviewer notes
Expect signature churn around `Session::initialize`,
`CodergenBackend::run`, `RunOptions.cancel_token`,
`StartServices.cancel_token`, and `Sandbox::exec_command_streaming`. The
main behavioral checks are that user cancellation reaches in-flight
CLI/API work and that timeout/stall paths remain distinct from user
cancellation.

### Fabro Details

<details>
<summary>Ran 9 stages in 117m 40s for $150.32</summary>

| Stage | Duration | Cost | Retries |
|---|---|---|---|
| start | 0s | – | 0 |
| toolchain | 1s | – | 0 |
| preflight_compile | 2m 8s | – | 0 |
| preflight_lint | 2m 13s | – | 0 |
| implement | 77m 12s | $56.78 | 0 |
| simplify_opus | 18m 5s | $5.83 | 0 |
| simplify_gpt | 15m 33s | $87.71 | 0 |
| verify | 1m 48s | – | 0 |
| fmt | 2s | – | 0 |
| **Total** | **117m 40s** | **$150.32** | **0** |

</details>

<details>
<summary>Ran <code>ImplementPlan.fabro</code> (12 nodes and 15
edges)</summary>

```dot
digraph ImplementPlan {
    graph [
        goal="Implement and simplify",
        model_stylesheet="
            * { model: claude-opus-4-7; }
        "
    ]
    rankdir=LR

    start [shape=Mdiamond, label="Start"]
    exit  [shape=Msquare, label="Exit"]

    toolchain         [label="Toolchain", shape=parallelogram, script="command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1", max_retries=0]
    preflight_compile [label="Preflight Compile", shape=parallelogram, script="cargo check -q --workspace 2>&1", max_retries=0]
    preflight_lint    [label="Preflight Lint", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1", max_retries=0]
    fix_lints         [label="Fix Lints", prompt="The preflight lint step failed. Read the build output from context and fix all clippy lint warnings.", max_visits=3]
    implement         [label="Implement", prompt="Read the plan file referenced in the goal and implement every step. Make all the code changes described in the plan. Use red/green TDD."]
    simplify_opus     [label="Simplify (Opus)", prompt="@prompts/simplify.md"]
    simplify_gpt      [label="Simplify (GPT-55)", prompt="@prompts/simplify.md", model="gpt-55"]
    verify            [label="Verify", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --cargo-quiet --workspace --status-level fail 2>&1 && cargo dev docs refresh 2>&1 && cargo dev docs check 2>&1", goal_gate=true, retry_target="fixup"]
    fixup             [label="Fixup", prompt="The verify step failed. Read the build output from context and fix all clippy lint warnings, test failures, and generated docs errors.", max_visits=3]
    fmt               [label="Format", shape=parallelogram, script="cargo +nightly-2026-04-14 fmt --all 2>&1", max_retries=0]

    start -> toolchain
    toolchain -> preflight_compile [condition="outcome=succeeded"]
    toolchain -> exit
    preflight_compile -> preflight_lint [condition="outcome=succeeded"]
    preflight_compile -> exit
    preflight_lint -> implement [condition="outcome=succeeded"]
    preflight_lint -> fix_lints
    fix_lints -> preflight_lint
    implement -> simplify_opus -> simplify_gpt -> verify
    verify -> fmt   [condition="outcome=succeeded"]
    verify -> fixup
    fixup -> verify
    fmt -> exit
}

```

</details>

⚒️ Generated with [Fabro](https://fabro.sh)

---------

Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
2026-05-05 09:54:22 -04:00
fabro-sh-0530[bot]
e901cd3a81
Surface silent fallback warnings in runs and logs (#205)
### Summary
Silent fallback paths now emit stable warnings instead of degrading
without a user-visible signal. The fallback behavior is unchanged; runs
still continue, but worktree, Git, checkpoint, and LLM failover issues
now show up in the run feed and logs.

### Plan Summary
- Emit run notices for workflow Git/worktree fallback paths.
- Reuse the existing failover event for one-shot LLM provider fallback.
- Add tracing for sandbox pipe drain failures.

### What changed
- Added `worktree_skipped_no_git` and gated `sandbox_git_unavailable`
notices during initialization.
- Added `git_push_failed` and `parallel_base_checkpoint_failed` notices,
including redacted output tails where available.
- Logged GitHub token mint failures with a structured `error` field
before the existing notice.
- Plumbed `Emitter` and `StageScope` through `CodergenBackend::one_shot`
so the API backend emits the existing `agent.failover` event instead of
a duplicate tracing-only warning.
- Extracted sandbox pipe draining into a helper that warns on
stdout/stderr read failures, with unit coverage for the error path.
- Updated CLI snapshots for the new worktree warning in stderr and JSON
event output.

⚒️ Generated with [Fabro](https://fabro.sh)

---------

Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
2026-05-05 09:18:05 -04:00
Bryan Helmkamp
88216313bf
perf(sandbox): skip throwaway reqwest::Client in Daytona probe
Two follow-ups that were still costing ~1s per credential probe:

- Bumped the daytona-sdk-rust pin to fa4870f, which deletes a dead
  underscore-prefixed _http_client field on Client. The field was
  unused but new_with_config built a fresh reqwest::Client for it on
  every call, paying the macOS proxy-discovery tax even with our
  injection seam in place.

- build_api_keys_configuration was using Configuration::new() and then
  overwriting cfg.client with our injected client. The Default impl
  generated by openapi-generator builds a reqwest::Client::new() for
  the client field eagerly, which we then threw away — another
  ~470ms hit per probe. Construct the Configuration as a struct
  literal so the injected client is the only one we ever build.

Drops the three credential-probe tests from ~700ms to ~10ms.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 12:48:44 -04:00
Bryan Helmkamp
17f1d1dfeb
perf(sandbox): inject http client into Daytona credential probe
Routes the two reqwest clients in the Daytona credential probe through
fabro_http (system-proxy) in production and fabro_test::test_http_client
(no_proxy) in tests, by threading an http_client parameter through
check_daytona_api_key_with and build_api_keys_configuration. Bumps the
daytona-sdk-rust pin to 314ffd9, which exposes DaytonaConfig::http_client
and ships on reqwest 0.13.

Drops the three credential-probe unit tests from >1s SLOW to ~0.5s by
skipping macOS proxy discovery on the localhost httpmock requests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 12:39:46 -04:00
Bryan Helmkamp
6780dff63f
fix(workflow): retain exec output tails on failures
Keep sandbox exec failures structured until event emission so git push, checkpoint, notice, and retro failures can expose redacted output tails without expanding their terse error strings.

Also add log rendering that appends sanitized tail content for exec-backed errors while preserving the existing safe Display behavior.
2026-05-03 21:30:34 -04:00
Bryan Helmkamp
446c7cc065
fix(sandbox): surface snapshot lifecycle progress
Emit snapshot slow-path events only when Docker or Daytona actually performs image or snapshot work, replace retired completion markers with snapshot.ready, and render the lifecycle in attach/log output.
2026-05-03 20:44:16 -04:00
Bryan Helmkamp
2d7c79de1e
feat(server): validate Daytona API key scopes
Probe the Daytona API at install, `fabro secret set DAYTONA_API_KEY`,
and `fabro doctor` time to confirm the configured key carries the
snapshot/sandbox scopes Fabro needs. Operators now see a precise scope
error against the control plane instead of a generic sandbox-create
failure at first run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 17:09:01 -04:00
Bryan Helmkamp
3f9a861f87
fix(error): preserve source chains across workspace
Keep typed transport and provider errors intact through API, GitHub, OAuth, install, diagnostics, and artifact paths. Add regression coverage for cloned shared errors and communication error chains.
2026-05-01 17:30:20 -04:00
Bryan Helmkamp
79e0783838
fix(sandbox): stabilize Docker exec stop test 2026-05-01 14:06:20 -04:00
Bryan Helmkamp
49fc9e6db7
fix(sandbox): skip Docker exec command after early stop
Short-circuit the controlled shell wrapper when the stop file already exists so a cancelled Docker exec does not launch user code before the pid watcher can terminate it.
2026-05-01 13:54:23 -04:00
Bryan Helmkamp
a4cfc617aa
refactor: expose redacted_output_tail as a public free fn
Drops the synthetic ExecResult fabricated in sandbox_metadata::stdout_output_tail
just to reach private redaction logic. The redact + sanitize + tail pipeline
now lives behind fabro_sandbox::redacted_output_tail(stdout, stderr, max),
which ExecResult::redacted_output_tail also delegates to.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 09:21:59 -04:00
Bryan Helmkamp
5437257521
refactor: unify metadata snapshot failure helpers and drop floor_char_boundary copies
Groups the MetadataSnapshotFailed event payload into a MetadataSnapshotFailure
struct and replaces the two near-identical 11-arg emit_metadata_snapshot_failed
helpers in lifecycle/git.rs and pipeline/finalize.rs with one shared helper
in sandbox_metadata.rs. Both #[allow(too_many_arguments)] blocks are removed.

Also deletes two hand-written floor_char_boundary copies (fabro-agent and
fabro-sandbox) in favor of the stable str::floor_char_boundary, matching how
most existing call sites already use it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 09:17:47 -04:00
Bryan Helmkamp
96d7ba4aa0
refactor: dedup exec failure diagnostic plumbing
Collapses four duplicate tracing field blocks in event.rs behind
ExecOutputTail::trace_summary, drops the parallel MetadataPushError
struct in favor of reusing SandboxMetadataError::Operation, inlines
the single-use Error::exec_result accessor, and gates the test-only
ExecResult::from_process_output to cfg(test).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-01 08:46:08 -04:00
Bryan Helmkamp
2f7aeba417
fix(workflow): preserve exec failure diagnostics
Add bounded redacted exec output tails to failure events while keeping tracing log-safe. Centralize tail projection on ExecResult and thread diagnostics through metadata, setup, devcontainer, and CLI install failures.
2026-05-01 08:45:50 -04:00
Bryan Helmkamp
9f2555e497
test(sandbox): move Docker streaming check out of unit tests
Keep daemon-dependent Docker lifecycle coverage as an ignored integration test so the default unit path remains runnable without Docker.
2026-05-01 08:31:31 -04:00
Bryan Helmkamp
542b56e51c
fix(sandbox): honor early Docker exec stop requests
Preserve timeout stop requests that arrive before the Docker exec wrapper has written its child pid, and cover that path with a fast unit regression test.
2026-05-01 08:31:31 -04:00
Bryan Helmkamp
6da71be2fa
fix(sandbox): forward exec_command_streaming through WorktreeSandbox 2026-04-30 22:45:38 -04:00
Bryan Helmkamp
1383022678
refactor(daytona): wrap session lifecycle in RAII guard 2026-04-30 22:45:38 -04:00
Bryan Helmkamp
de64136a0e
refactor: tighten command-log streaming hot paths 2026-04-30 22:45:38 -04:00
Bryan Helmkamp
3827d3946e
fix(daytona): satisfy streaming smoke lints 2026-04-30 22:45:38 -04:00
Bryan Helmkamp
83919f3569
test(daytona): add live streaming smoke 2026-04-30 22:45:37 -04:00
Bryan Helmkamp
f5f6c76f16
fix(daytona): preserve streamed command semantics 2026-04-30 22:45:37 -04:00
Bryan Helmkamp
d4be6362cb
feat(daytona): stream session command logs 2026-04-30 22:45:37 -04:00
Bryan Helmkamp
e50df2b58b
feat(command): distinguish cancelled commands from timeouts
Represent command termination explicitly across sandbox results, events,
run projections, API types, and the run stage UI. This removes the fake
-1 exit code path for timeout/cancel and lets consumers tell cancelled
commands apart from timed-out commands.
2026-04-30 22:45:37 -04:00
Bryan Helmkamp
8ac400df1c
feat(command): stream command logs from CAS-backed storage
Persist command stdout/stderr through scratch logs and finalized CAS refs, expose byte-offset tailing through the API, and render separate streaming panels in the web run view.

Resolve command output blob refs for execution-time consumers such as edge routing and retros, and make Docker streaming timeout/cancel drain output before returning.
2026-04-30 22:45:37 -04:00
Bryan Helmkamp
f18da11854
chore(logs): promote run lifecycle traces to info level
Promotes per-run observability events (stage start/complete, edge
selection, checkpoint, fidelity resolution, agent session, LLM stream
finish, tool calls, sandbox cleanup, PR build/create) from debug to
info so default-level operators see end-to-end run progress.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 08:25:37 -04:00