`DaytonaSnapshotSettings` carried two independent `Option`s (`image` and
`dockerfile`) that every consumer had to re-validate. Replace them with a
single `source: DaytonaSnapshotSource { Image, Dockerfile }` so the
both-set and neither-set states are unrepresentable at the sandbox layer.
This removes four unreachable error arms in `canonical_manifest` and
`create_snapshot_params`, the `.filter(...)` guard in `initialize`, and
the presence guard in `daytona_config_from_environment`. The
mutual-exclusion rule now lives only in fabro-config, which owns the
`image.docker` / `image.dockerfile` keys the old messages named.
Merge `ImageSnapshotManifest` into `SnapshotManifest` via a flattened
`SourceManifest` enum. The dockerfile case serializes to the same bytes
as before, so existing snapshot names are unchanged; the pinned identity
test still passes. Pin the image-case identity as well so a future
manifest change cannot silently orphan image snapshots.
Fold `validate_daytona_image_settings` into the existing Daytona arm of
`validate_provider_capabilities`; both callers already invoke it right
after `resolve_environment_fields`, so error order is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Introduce a PinnedRevision enum in clone_source so the Docker and Daytona
providers run one fetch/checkout/verify sequence for both an exact commit
and a tag instead of two near-identical arms. Fold the tag-specific
command builders into the generic ones, share the bare-ref grammar check
between branch and tag validation, and derive the workflow clone source
from the validated Git target in a single match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Use /tmp/fabro/runtime for both Docker and Daytona instead of
provider-specific roots. A writable /tmp inside the sandbox is already
a dependency (commit-message files, exec stop-files), it needs no
root-level mkdir for non-root container users, and it makes the two
providers uniform.
The trailing runtime path component stays load-bearing: materialized
blobs at runtime/blobs/{hash}.json are recognized as managed blob
references and normalized back to blob:// in durable context.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Kmn5jyrdpyCdvcfvmEDvA
Remote prompt-value materialization wrote demoted values to
{working_directory}/.fabro/blobs inside the repository checkout, so a
later checkpoint could commit them and leak them into the run pull
request.
Give each sandbox a run-scoped runtime directory outside the source
checkout as part of the Sandbox contract:
- Sandbox::runtime_directory() names the directory; host-local
sandboxes return None because the engine owns a host-side runtime
directory (RunScratch) for those runs.
- Docker creates /fabro/runtime at initialize with umask 077 and
uploads runtime files with mode 0600.
- Daytona creates /home/daytona/fabro/runtime with mode 0700.
- Both remote materialization paths in fabro-workflow share one
materialization-path helper built on the new contract. The paths keep
the runtime/blobs suffix, so durable context still normalizes to
blob://sha256/... references.
- Local materialization now writes owner-private directories and files
on Unix.
Regression coverage: an integration test runs remote-style prompt
demotion against a real git checkout, then a real checkpoint commit,
and asserts the checkout stays clean, the agent-facing file is
readable, and a deleted materialized file is recreated from the
durable blob store. A real-Docker test verifies the runtime directory
and blob file permissions inside a container.
Fixes#798
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Kmn5jyrdpyCdvcfvmEDvA
Apply cleanups from a reuse/simplification/efficiency review of the
bounded-tool-output changes:
- Share one MAX_RUN_EVENT_BODY_BYTES constant in fabro-types; the server
body limit, the agent's serialized-output reservation, and the event
headroom test all derive from it.
- Rework truncation.rs around one split_head_tail helper: drop the
hand-rolled ceil_char_boundary (std's is stable), the duplicate
truncate_plain_output splitter and its dead Tail arm, and the
head_bytes field with its sentinel values.
- Return Cow from preview_tool_output and take retain_tool_output's
input by value, so untruncated output crosses the pipeline without
full copies. Measure serialized JSON size with a counting writer
instead of materializing the payload.
- Reuse fabro-llm's byte-token estimate (now public) instead of a third
copy of the 4-bytes-per-token heuristic.
- Take retain_tool_result's ToolResult by value and mutate content in
place; extract the triplicated error retain-emit-truncate block into
finish_error_result.
- Share the shell retain-and-record sequence between the native and
kimi shell tools as retain_shell_output.
- Move OutputCaptureBuffer::into_parts to reuse the head allocation,
skip the buffer round-trip in replay_exec_result when output fits,
and replace daytona's byte-iterator suffix matching with contiguous
slice comparisons behind one retained_slices accessor.
- Make SessionBoundEmitter's fields private.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TK3QTWQHiXhRbFwTr57LzX
duration_to_minutes_i32 carried stacked docker and daytona cfg attributes,
which combine as AND, so building with the daytona feature alone failed to
find the function. fabro-workflow and fabro-cli enable daytona without
docker in their production dependencies, so that combination is real.
Removing the stray docker gate surfaced items that only docker-gated code
uses: the ResolveError import in from_environment and four exact-checkout
command builders in clone_source. Gate those on the docker feature, keeping
the command builders available to clone_source's own tests under cfg(test).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TK3QTWQHiXhRbFwTr57LzX
Make RunCloneSettings::DEFAULT_DEPTH the single owner of the default
depth, and interpret the "0 = full history" sentinel in one place via
RunCloneSettings::depth_limit(). Docker's clone_depth becomes
Option<usize> to match Daytona's encoding, with a shared
depth_argument() helper for both git command builders. Drop the
unreachable Option on the resolved depth field, the hand-written
DaytonaSettings::Default, and the pure-forwarding
daytona_git_clone_options helper. The blob-import test helper reuses
the pool's own connect options instead of rebuilding a partial copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019bgXj5J218RXfiT72qhbLV
Consolidate the copies that review found across the feature:
- One GITHUB_CREDENTIAL_HELPER / GITHUB_CREDENTIAL_HELPER_KEY pair in
fabro-github, with apply_probe_git_env() for probe commands; the runtime
git bridge, server preflight probe, and live contract test all consume it
so the probes exercise exactly what the bridge configures.
- GitHubRepositoryAccess::resolve_verified_token() owns the
resolve-installations-then-mint choreography shared by server preflight,
workflow initialization, and the live test.
- A shared lookup_installation() helper backs both the shared-installation
resolution and the mint's installation lookup.
- The contents = read|write rule lives once as
RunIntegrationsGithubSettings::contents_permission_allows_repository_access.
- The preflight probe paces retries with fabro-sandbox's exported
replication_backoff() (3s/9s) instead of a contradicting 1s/2s loop, and
shares one run_ls_remote() runner with the existing remote-ref check.
Also: collapse the dead Ok(None) arm and repeated error blocks in the
preflight token check, drop the derivable bridge_entry_count(), privatize
resolve_permissions() behind resolve_integration(), make
GitHubRepositorySlug ordering/hashing allocation-free, and use EnvVars
constants for env names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sandbox exec logs previously required command_len fingerprinting to tell a
push from a credential refresh or a checkpoint commit. The shared git
helpers now instrument their futures with a git_op span, so Daytona's and
Docker's `exec_command: entered` lines inherit the operation label and the
log renders as `git_op{op=push}: exec_command: entered timeout_ms=...`.
Ops: push (git_push_via_exec), refresh-credentials (both providers'
refresh_push_credentials), checkpoint-commit (checked_git_checkpoint),
fetch (fetch_source_run_ref), and metadata-push (the run-metadata snapshot
write). Spans are attached with #[tracing::instrument] — attached to the
future, never an entered() guard held across an await — so they follow the
task across worker threads. No trait or signature changes.
Plan: .ai/plans/git-push-token-resilience.md (PR 3: item 10).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Daytona rejects start/stop with HTTP 400 "State change in progress"
while a lifecycle transition is in flight, and activate() only handled
the Started and Starting states: any other state fell through to
start(), which surfaced the rejection as a hard failure. A run died
exactly this way when an inactivity auto-stop began seconds before the
stage finished — activate() saw the sandbox mid-stop and failed the
whole run 35ms later. The cleanup stop() then failed on the same
rejection.
Transitions finish on their own within seconds, so treat them as
wait-and-retry conditions:
- activate() now waits out a Stopping sandbox and dispatches on
whatever state the transition lands on.
- start() and stop() retry the rejected call within a bounded budget,
re-inspecting state between attempts: a transition that lands on
Started needs no further start, and one that lands on Stopped or
Destroyed needs no further stop.
All call sites go through these three provider methods, so no
lifecycle-layer changes are needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Omitting autoStopInterval from the create-sandbox request inherits
Daytona's server-side default of 15 idle minutes. Daytona counts
inactivity from the last sandbox interaction, and LLM inference never
touches the sandbox, so a single long inference call is enough for the
sandbox to auto-stop mid-run: a workflow failed exactly this way, with
the sandbox entering its stop transition 15 minutes after the last
command while the agent was still thinking.
Send an explicit 120-minute default when lifecycle.auto_stop is unset.
That clears any realistic inference call while still reclaiming
sandboxes leaked by a dead worker. An explicit auto_stop = "0s" still
disables auto-stop entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run the local git steps of the Docker exact checkout under the shared
clone deadline instead of a fixed 10s timeout, so materializing a large
working tree cannot time out and abandon a running checkout in the
container.
Check the admitted commit out onto the admitted branch rather than
detaching. A detached HEAD makes `rev-parse --abbrev-ref HEAD` return
"HEAD", which the git setup helper maps to no base branch, silently
dropping it for callers that rely on it. Daytona does the same after its
native clone and now verifies the resulting HEAD the way Docker does.
Fetch the exact commit at the same depth a branch clone uses, so both
paths can reach the same number of parent commits, and stop suggesting
GitHub App credentials when a purely local git step fails.
Document that reachability of the commit from the branch is an
admission-time invariant that the sandbox layer does not re-verify.
Run 01M0DH033P2XSTHAGVBHG6922F completed 2.8 hours of work, then failed
terminally because four consecutive publish pushes hit GitHub's
token-replication lag (404 "Repository not found") — the push path had no
retry, the failure was misclassified as deterministic, and the same
fresh-mint-then-push pattern silently disabled metadata snapshots. This
generalizes the clone retry machinery to pushes and makes attempt detail
durable.
- clone_retry -> git_retry: the classifier's boolean becomes a
CredentialContext derived from the token snapshot (fresh App tokens retry
404s as replication lag, mature ones as transient infra, static
credentials fail fast), and the attempt/backoff limits become a RetryPlan
with layered optional bounds. Clone behavior is preserved: Docker keeps
its absolute five-minute deadline, Daytona keeps no deadline.
- Pushes take a scoped CredentialLease before the first attempt: it owns
the embed mutex for the whole operation, pins the single successful
resolve, retries only failed resolves, falls back to the last embedded
token when a mint fails, and force-re-embeds the pinned token once after
the first auth-shaped failure (drift repair). The margin invariant
(REFRESH_MARGIN > every push plan's max_elapsed) guarantees the pinned
token outlives the operation; a unit test asserts it.
- Sandbox::git_push_ref now takes a RetryPlan and returns PushReport /
PushError with per-attempt records (classification, redacted output tail,
token generation/provenance/age, credential action, refresh errors).
Checkpoint pushes use a 90-second budget; the terminal publish push gets
5 attempts over at most 4 minutes.
- The single durable git.push event per push gains a nested attempts array
(GitPushAttemptProps, token snapshot flattened to flat fields); stored
events without it still deserialize. Publish push failures now carry an
explicit failure category — exhausted transient retries stay
transient_infra instead of deterministic — plus one bounded cause line
per attempt and the last successful push time in the message.
- Metadata snapshot degradation records why it degraded: push failures with
retryable classifications leave the writer eligible to re-probe at each
later checkpoint, and a successful snapshot clears the degraded state and
re-arms the warning. Permanent failures keep today's latch.
Plan: .ai/plans/git-push-token-resilience.md (PR 2: items 1, 2, 4, 7 and
the metadata re-probe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every push previously re-minted a fresh GitHub App installation token and
embedded it in the origin URL, so pushes routinely landed inside GitHub's
token-replication lag window (run 01M0DH033P2XSTHAGVBHG6922F failed
terminally on four consecutive fresh-token 404s). Reusing mature tokens
removes the failure trigger and saves two GitHub API calls plus one sandbox
exec per push.
- New fabro_github::token_source::InstallationTokenSource: one cached,
single-flight source per origin repo. Static credentials pass through
(generation 0); App credentials mint through the cache and reuse tokens
until REFRESH_MARGIN (10 min) before expiry. Every resolve returns a
non-secret TokenSnapshot (generation + Minted/Reused/Static provenance),
and the source logs mints at INFO and reuses at DEBUG.
- Docker and Daytona share the source through PushCredentialState: an embed
mutex serializes compare -> set-url -> record, a matching generation skips
the set-url exec, and the generation is recorded only after a successful
exec. The clone still mints its own token, but now seeds the source cache
(generation 1) and the last-embedded state, so a refresh mint failure
falls back to the known embedded token instead of believing nothing was
ever embedded.
- RefreshOutcome now reports the remote action (embedded/unchanged/none)
separately from the token snapshot; git_push_via_exec logs token age and
provenance with each push, and refresh failures log the last embedded
generation.
- The run-metadata writer resolves through the sandbox's shared source
instead of minting per snapshot (with its own cached source on resume).
- The ACP refresh-ahead loop reschedules from the embedded token's
expires_at minus the margin instead of a fixed 45-minute interval, which
a cached source would have broken for long turns; static credentials stop
the loop.
Plan: .ai/plans/git-push-token-resilience.md (PR 1: items 3 and 6).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Fold Docker's exact-checkout path into clone_github_repo so the auth,
retry, symlink, and bookkeeping skeleton is shared with branch clones
- Skip the Daytona SDK clone for exact checkouts: init and shallow-fetch
the admitted commit directly instead of cloning the default branch and
discarding it
- Combine the detach checkout and HEAD verification into one shell
command, saving an exec round trip per init
- Drop the spec-level decide_clone pre-checks that duplicated the
constructors' fail-fast validation
- Share a CloneAttemptFailure struct in clone_retry and the GIT command
prefix constant across git command builders
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ExecStreamingRequest: drop #[non_exhaustive] and the six Option-taking
builder setters; call sites use struct literals over ::new(), matching
GrepOptions/WalkOptions, and providers can destructure exhaustively
- Docker: pass ExecStreamingRequest through docker_exec_shell_streaming
instead of seven positional args; revert the no-op StartExecOptions
- Daytona: stdin temp-file cleanup is now best-effort (mirrors
DaytonaSession::close) so a failed delete cannot fail a completed
command or double-delete from Drop; upload overlaps session creation;
one shared DAYTONA_CLEANUP_TIMEOUT
- write_process_stdin tolerates ConnectionReset/ConnectionAborted so a
command that stops reading stdin does not fail on TCP Docker daemons
- Local sandbox aborts the stdin writer after process exit instead of
joining unbounded
- Cap stdin_source payloads at 10 MiB, mirroring the for_each bound
- Add Node::context_key_attr() tri-state so the handler and lint rule
share one definition of a valid context-key attribute
- inert_attribute canonicalizes handler types via StageHandler, fixing
false warnings for command attrs on tool nodes
- Share resolve_flat_context_value between command stdin and for_each;
resolve_json_value takes Value by value, removing a deep clone
- Reuse MockSandbox in command handler stdin tests instead of extending
SpySandbox with a hand-rolled streaming override
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A Daytona-backed run could fail five seconds after start when the sandbox
git clone hit a transient GitHub "Repository not found" error. Clone-based
providers mint an installation access token and clone with it in the same
breath, but GitHub replicates a new token to its edge cache sites
asynchronously. A clone that starts within a second of the mint can be
rejected before the token is visible to the site serving it, and on a
private repo that rejection arrives as "Repository not found" because
GitHub answers unauthorized reads with 404.
Nothing retried the clone, and the failure classified as `deterministic`,
which is the one category `loop_restart` refuses to restart. An identical
run relaunched 46 seconds later succeeded with no changes.
A successful mint is what makes the message safe to retry.
`resolve_clone_credentials` already fails loudly on every deterministic
explanation for a clone 404: the installation lookup 404s when the App is
not installed for the owner, and token creation 422s when the installation
does not cover the repo. Once credentials are in hand, "not found" from the
clone itself cannot mean "no access".
Add `clone_retry` and use it from both clone-based providers: 3 attempts
with 3s then 9s backoff, reusing the same token so replication keeps making
progress instead of restarting the clock. Token-replication signatures
retry only when credentials are present, so a public clone of a wrong URL
still fails fast. Infrastructure failures retry either way.
The Docker provider had the identical single-shot clone and is the default
runtime provider, so it is covered too.
Also fix two nearby issues found while reading the area:
- The GitHub-URL-parse path in the Daytona clone skipped `fail_init`,
unlike every sibling path, so `InitializeFailed` was never emitted.
- The `classify_exec_failure` hint for "repository not found" asserted the
App installation may not cover the repo. After a successful scoped mint
that diagnosis is impossible, and it sent operators hunting a
configuration problem that did not exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The process environment is no longer a configuration source. `{{ vars.NAME }}`
(non-sensitive, server-stored) and `{{ secrets.NAME }}` (vault-backed) cover
both cases, and reading the worker's ambient environment made a run's inputs
depend on how its process happened to be launched.
`Namespace::Env` is kept but wired to nothing, so `{{ env.NAME }}` still
parses and fails with a message naming its replacement rather than reaching
a consumer as literal text. `ResolveCtx::with_env` is gone, so no call site
can opt back in.
Two long-standing warts were env-only and go with it:
- `InterpString::resolve_or_source`, the "fall back to the raw template
source on failure" path, which let an unresolved token reach a sandbox or
the GitHub API as literal `{{ ... }}` text. Its own comment noted it was
slated for hard-error semantics.
- `RunEnvironmentSettings::resolve_env`'s matching source fallback for
env-only values.
Both carried `#[expect(clippy::disallowed_methods)]` escape hatches. Every
run-boundary resolver — sandbox env, prepare steps, MCP transports, GitHub
permissions, Slack channels, run goal files, provider extra_headers — now
fails closed instead.
Hooks lose their `allowed_env_vars` allowlist, `resolve_header`, and
`HeaderResolveError` along with the `E: Env` generic threaded through the
executor. They keep `{{ vars.* }}`, which `RunSettings::substitute_variables`
already substitutes server-side at run creation.
`allowed_env_vars` is removed from the OpenAPI spec and the generated
TypeScript client. The docs example showing `{{ env.* }}` in
`[server.slatedb.s3].bucket` was already wrong — that field is a plain
String and never interpolated — and is now a literal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
