Use /tmp/fabro/runtime for both Docker and Daytona instead of
provider-specific roots. A writable /tmp inside the sandbox is already
a dependency (commit-message files, exec stop-files), it needs no
root-level mkdir for non-root container users, and it makes the two
providers uniform.
The trailing runtime path component stays load-bearing: materialized
blobs at runtime/blobs/{hash}.json are recognized as managed blob
references and normalized back to blob:// in durable context.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Kmn5jyrdpyCdvcfvmEDvA
Remote prompt-value materialization wrote demoted values to
{working_directory}/.fabro/blobs inside the repository checkout, so a
later checkpoint could commit them and leak them into the run pull
request.
Give each sandbox a run-scoped runtime directory outside the source
checkout as part of the Sandbox contract:
- Sandbox::runtime_directory() names the directory; host-local
sandboxes return None because the engine owns a host-side runtime
directory (RunScratch) for those runs.
- Docker creates /fabro/runtime at initialize with umask 077 and
uploads runtime files with mode 0600.
- Daytona creates /home/daytona/fabro/runtime with mode 0700.
- Both remote materialization paths in fabro-workflow share one
materialization-path helper built on the new contract. The paths keep
the runtime/blobs suffix, so durable context still normalizes to
blob://sha256/... references.
- Local materialization now writes owner-private directories and files
on Unix.
Regression coverage: an integration test runs remote-style prompt
demotion against a real git checkout, then a real checkpoint commit,
and asserts the checkout stays clean, the agent-facing file is
readable, and a deleted materialized file is recreated from the
durable blob store. A real-Docker test verifies the runtime directory
and blob file permissions inside a container.
Fixes#798
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Kmn5jyrdpyCdvcfvmEDvA
- Use a derived deserializer for RunTarget by making `None` an empty struct
variant, which keeps `deny_unknown_fields` strict without a hand-rolled impl
- Make clone_source_for_run the single owner of the empty-workspace decision
and drop the duplicated target checks in RunSession::new
- Collapse duplicated target/provider compatibility matches in admission and
start into single matches, using a strum-derived kind name for messages
- Drop the redundant git override in persist_create_run
- Extract a shared helper for the duplicated unavailable-integration test loop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_blob_store was a process-wide OnceLock singleton over one in-memory
SQLite connection, so content-addressed rows written by one test were
visible to every other test in the same process. nextest's
process-per-test model masked the bleed, but plain cargo test failed
(8/24 in fabro-workflow-version) because negative existence assertions
became order-dependent.
test_blob_store now builds a fresh isolated in-memory store per call,
and test_database gives every database its own blob authority.
Reopen-style tests that model one durable blob authority across several
store handles use the new test_blob_store_at, which keeps the blob table
in a SQLite file beside the store directory, plus
test_database_with_blobs to share it explicitly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Blob activation cleanups:
- Reuse fabro-db's append_to_path, remove_file_if_exists, and
set_private_permissions instead of local duplicates.
- Return the store directly from activate_blob_storage; the report
wrapper existed only to be logged internally and then discarded.
- Collapse compute_disk_preflight to return the required free bytes
instead of echoing its inputs back through a struct.
- Deduplicate the "exactly one ok row" PRAGMA integrity_check protocol
into one executor-generic helper used by the backup and live checks.
- Skip re-validating a freshly published backup; the staging copy was
validated immediately before the atomic rename, so only a
concurrently published file needs its own validation.
- Replace the manual anyhow wrapping plus duplicate error log in
serve.rs with a plain .context(), matching other startup errors.
- Extract the disk-candidate enumeration in resource_sampler.rs that
available_space_for_path had copy-pasted from sample_disk_resources.
Test fixture cleanups:
- Route all hand-assembled Database::new(..., test_blob_store()) test
fixtures (32 sites) through fabro_store::test_support::test_database,
and make that helper infallible instead of returning an unconditional
Ok.
- Install the test blob schema from fabro_db::BLOBS_MIGRATION_SQL via a
test-support-gated optional dependency instead of a four-level
relative include_str! into fabro-db's migrations directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Start reconciled the persisted target against its stored GitContext
projection field by field and failed the run on any drift, which forced
every RunSpec writer to keep the pair in lockstep forever. The target is
validated at admission and owns the grammar, so derive the clone source
from it alone; the projection stays persisted as display metadata that
can no longer fail an otherwise-healthy start.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Git-target grammar (slug, branch, and SHA rules plus the derived
origin URL) was implemented twice with no shared code path: once in
server admission and again in sandbox start, so the two could drift and
disagree about which persisted targets are valid.
Own it once as RunTarget::validate() in fabro-types, next to the
primitives it uses, returning the canonical target together with its
derived GitContext projection. Admission consumes it directly, and the
start path re-derives the expected clone source from the same rules
before checking the persisted projection against it. The start path now
also moves the derived strings into the sandbox spec instead of cloning
them.
While reordering admission around the shared validator, run the pure,
in-memory checks (target grammar, environment id) before the blob-store
closure fetch and lowering so malformed requests no longer pay for
version-store I/O.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Apply the cleanup findings from a four-angle review (reuse,
simplification, efficiency, altitude) of the demotion change:
- Share one size gate: serialized_if_over now backs both offload_value
and demote_value_for_prompt, restoring the cheap short-string and
scalar pre-checks so per-node demotion no longer serializes every
small value just to measure it.
- Stop re-writing blobs every node: materialize_value_bytes writes the
sandbox file directly from the in-hand bytes and short-circuits on the
content-addressed file's existence, so an already-demoted value costs
one existence probe instead of a store round-trip per node visit. The
local file write is shared with materialize_blob_ref.
- Demote over the resolved snapshot map instead of re-snapshotting a
Context copy, making the context and outcome loops symmetric and
saving a full deep clone per node; the fidelity lifecycle builds the
Context after the pass.
- Skip the pass entirely for Full and Truncate fidelities (nothing
renders context values), except parallel nodes whose branch stash may
render at a richer fidelity.
- Build is_preamble_hidden_key on is_engine_internal_key instead of
restating its prefixes, and call it directly from the preamble
renderer rather than through a wrapper.
- Document that outcome updates are demoted wholesale and that
BranchWorkItem.item carries the prompt-ready (possibly demoted) item;
drop the item rebinding and redundant test assertions; restore the
local integration test's confinement assertion and make the remote
one non-vacuous.
Skipped by choice: unifying the crate's several truncation helpers and
rendering the marker through the "See:" pointer family (cross-module
coupling out of proportion to the preview cosmetics), per-branch
demotion inside parallel.results (wholesale demotion is what bounds the
total), and cross-node demotion memoization (the file-existence
short-circuit already reduces repeats to a stat).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FH8Jj9Y4E4Tu5g1jwDtHAb
Compact and summary preambles render workflow context values and stage
outputs with no per-value size limit. A late-run node inherits everything
the run has accumulated, and one oversized value (a join result, a jobs
list, a single-line command emit) can push the composed prompt past the
model's context window. A security-review run failed exactly this way:
its dedupe stage assembled a ~1.8M-token prompt against a 1M-token model
limit, made almost entirely of accumulated context the agent never
needed inline.
Reuse the existing blob machinery at the last mile. Before the preamble
builders run, any resolved context or outcome value whose serialized
JSON exceeds 8KB is persisted as a content-addressed blob, materialized
as a real file in the sandbox, and replaced with a small marker holding
a preview, the byte count, and the file path. The agent reads the file
if it needs the data. for_each items get the same treatment at fan-out
with a more generous 64KB budget, since the item is the branch's work
assignment; branch labels still come from the full item. Keys the
preamble never renders are left alone, and a value that fails to demote
stays inline and is logged: demotion bounds prompt size, it does not
gate execution.
The two downstream-resolution integration tests asserted that resolving
text values writes no files; demotion now legitimately materializes the
oversized response for preamble use, so they instead pin that resolution
returned the full inline text and that nothing is written outside the
sandbox blob directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FH8Jj9Y4E4Tu5g1jwDtHAb
- Make the failure watch channel the single record of the latched
failure; drop the worker task's mirrored local state.
- Replace the hand-rolled wait loop with watch::Receiver::wait_for.
- Extract race_persistence/flush_or_stop helpers so the select!/flush
scaffolding in RunSession::run exists once instead of three times.
- Return RunEventPersistenceError from append_event_to_sink and add a
From impl on Error, replacing four hand-written per-event message
strings with the event name derived from the event itself.
- Dedupe the RunCreated test seed literal in initialize.rs and drop the
dead BlockingHandler::simulate override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ryyhtbc1eNtCLw8GjrFQXZ
Make RunCloneSettings::DEFAULT_DEPTH the single owner of the default
depth, and interpret the "0 = full history" sentinel in one place via
RunCloneSettings::depth_limit(). Docker's clone_depth becomes
Option<usize> to match Daytona's encoding, with a shared
depth_argument() helper for both git command builders. Drop the
unreachable Option on the resolved depth field, the hand-written
DaytonaSettings::Default, and the pure-forwarding
daytona_git_clone_options helper. The blob-import test helper reuses
the pool's own connect options instead of rebuilding a partial copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019bgXj5J218RXfiT72qhbLV
Consolidate the copies that review found across the feature:
- One GITHUB_CREDENTIAL_HELPER / GITHUB_CREDENTIAL_HELPER_KEY pair in
fabro-github, with apply_probe_git_env() for probe commands; the runtime
git bridge, server preflight probe, and live contract test all consume it
so the probes exercise exactly what the bridge configures.
- GitHubRepositoryAccess::resolve_verified_token() owns the
resolve-installations-then-mint choreography shared by server preflight,
workflow initialization, and the live test.
- A shared lookup_installation() helper backs both the shared-installation
resolution and the mint's installation lookup.
- The contents = read|write rule lives once as
RunIntegrationsGithubSettings::contents_permission_allows_repository_access.
- The preflight probe paces retries with fabro-sandbox's exported
replication_backoff() (3s/9s) instead of a contradicting 1s/2s loop, and
shares one run_ls_remote() runner with the existing remote-ref check.
Also: collapse the dead Ok(None) arm and repeated error blocks in the
preflight token check, drop the derivable bridge_entry_count(), privatize
resolve_permissions() behind resolve_integration(), make
GitHubRepositorySlug ordering/hashing allocation-free, and use EnvVars
constants for env names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brave stays the default. Shops that already vault VENICE_API_KEY
can drop BRAVE_SEARCH_API_KEY by setting
[server.integrations.search] provider = "venice".
Co-authored-by: Cursor <cursoragent@cursor.com>
Carry the resolved GitHub integration (permissions plus declared
additional repositories) as one value from run materialization into
workflow startup, and make the sandbox environment reach every declared
repository through the single managed GITHUB_TOKEN.
- `StartServices.github_permissions` becomes
`github_integration: ResolvedGithubIntegration`; CLI and server
workers build it with `resolve_integration()` after interpolation and
pass it through `SandboxEnvSpec` as one unit.
- `build_sandbox_env` constructs the validated
`GitHubRepositoryAccess` and scopes the App token source to the whole
effective set. Missing credentials or a missing origin are hard
initialization errors when additional repositories are declared;
legacy permissions-only configuration keeps its best-effort behavior.
- When additional repositories are declared, initialization eagerly
resolves each repository's App installation (naming any repository
the App cannot see) and the token itself, so an inaccessible declared
repository fails before the first workflow stage.
- A new `git_bridge` module injects secret-free `GIT_CONFIG_*` entries
into the stage environment: a github.com credential helper that reads
`$GITHUB_TOKEN` at invocation time, per-repository SSH-to-HTTPS
`insteadOf` rewrites, and `GIT_TERMINAL_PROMPT=0`. Entries append
after a valid user-provided Git config overlay and fail clearly on a
malformed one. Contract tests drive the installed git binary against
local fixtures for the rewrite, credential, prefix-collision, and
overlay-preservation behaviors.
- The long-running ACP notice now says all declared repository access
expires together.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lineage field's `skip_serializing_if` behavior was asserted five times
across three crates. Keep the two assertions in fabro-types, which owns the
attribute, and drop the duplicates:
- Delete `run_created_omits_absent_workflow_version_id` from event/convert.rs,
a copy of the test above it that re-checked another crate's serde attribute.
convert.rs's own responsibility is covered by the existing field assertion.
- Delete `legacy_create_input_persists_without_workflow_version_id`, which ran
the full create() pipeline to prove a hardcoded `None` literal is `None`.
`CreateRunInput` has no such field, so no input could change the result.
- Fold `run_spec_omits_absent_workflow_version_id` into the adjacent legacy-spec
test, which already holds an all-`None` record.
- Drop the off-topic spec re-serialization from run_state.rs's retried_from test.
Add `test_support::test_workflow_version_id()` alongside `test_run_provenance()`
and use it everywhere, replacing eight copies of the same magic seed across five
crates plus two assertion sites that recomputed the hash inline. This also
subsumes retry.rs's private helper of the same shape.
Revert the `run_spec_json` parameterization in the projection round-trip test:
`RunProjection` is a `with_replacement` alias for the canonical type, so the
`Some` and `None` call sites exercise identical code.
Have the two run.created literals that mirror a `RunSpec` read the spec's
lineage field instead of hardcoding `None`, so the mirrors stay accurate once a
producer populates it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sandbox exec logs previously required command_len fingerprinting to tell a
push from a credential refresh or a checkpoint commit. The shared git
helpers now instrument their futures with a git_op span, so Daytona's and
Docker's `exec_command: entered` lines inherit the operation label and the
log renders as `git_op{op=push}: exec_command: entered timeout_ms=...`.
Ops: push (git_push_via_exec), refresh-credentials (both providers'
refresh_push_credentials), checkpoint-commit (checked_git_checkpoint),
fetch (fetch_source_run_ref), and metadata-push (the run-metadata snapshot
write). Spans are attached with #[tracing::instrument] — attached to the
future, never an entered() guard held across an await — so they follow the
task across worker threads. No trait or signature changes.
Plan: .ai/plans/git-push-token-resilience.md (PR 3: item 10).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A Daytona "Sandbox state change in progress" rejection surfacing
through the pipeline lifecycle path ("Pipeline lifecycle operation
failed") matched no transient-infra hint, so the run failure was
categorized deterministic. The condition is a provider lifecycle
transition that finishes on its own — the definition of transient
infrastructure — and the deterministic label misinforms retry
machinery and anyone reading the failure.
Add two transient-infra hints: the provider rejection ("state change
in progress") and the bounded-wait timeout an activation reports when
a stop transition outlives its budget ("sandbox stop still in
progress").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 01M0DH033P2XSTHAGVBHG6922F completed 2.8 hours of work, then failed
terminally because four consecutive publish pushes hit GitHub's
token-replication lag (404 "Repository not found") — the push path had no
retry, the failure was misclassified as deterministic, and the same
fresh-mint-then-push pattern silently disabled metadata snapshots. This
generalizes the clone retry machinery to pushes and makes attempt detail
durable.
- clone_retry -> git_retry: the classifier's boolean becomes a
CredentialContext derived from the token snapshot (fresh App tokens retry
404s as replication lag, mature ones as transient infra, static
credentials fail fast), and the attempt/backoff limits become a RetryPlan
with layered optional bounds. Clone behavior is preserved: Docker keeps
its absolute five-minute deadline, Daytona keeps no deadline.
- Pushes take a scoped CredentialLease before the first attempt: it owns
the embed mutex for the whole operation, pins the single successful
resolve, retries only failed resolves, falls back to the last embedded
token when a mint fails, and force-re-embeds the pinned token once after
the first auth-shaped failure (drift repair). The margin invariant
(REFRESH_MARGIN > every push plan's max_elapsed) guarantees the pinned
token outlives the operation; a unit test asserts it.
- Sandbox::git_push_ref now takes a RetryPlan and returns PushReport /
PushError with per-attempt records (classification, redacted output tail,
token generation/provenance/age, credential action, refresh errors).
Checkpoint pushes use a 90-second budget; the terminal publish push gets
5 attempts over at most 4 minutes.
- The single durable git.push event per push gains a nested attempts array
(GitPushAttemptProps, token snapshot flattened to flat fields); stored
events without it still deserialize. Publish push failures now carry an
explicit failure category — exhausted transient retries stay
transient_infra instead of deterministic — plus one bounded cause line
per attempt and the last successful push time in the message.
- Metadata snapshot degradation records why it degraded: push failures with
retryable classifications leave the writer eligible to re-probe at each
later checkpoint, and a successful snapshot clears the degraded state and
re-arms the warning. Permanent failures keep today's latch.
Plan: .ai/plans/git-push-token-resilience.md (PR 2: items 1, 2, 4, 7 and
the metadata re-probe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every push previously re-minted a fresh GitHub App installation token and
embedded it in the origin URL, so pushes routinely landed inside GitHub's
token-replication lag window (run 01M0DH033P2XSTHAGVBHG6922F failed
terminally on four consecutive fresh-token 404s). Reusing mature tokens
removes the failure trigger and saves two GitHub API calls plus one sandbox
exec per push.
- New fabro_github::token_source::InstallationTokenSource: one cached,
single-flight source per origin repo. Static credentials pass through
(generation 0); App credentials mint through the cache and reuse tokens
until REFRESH_MARGIN (10 min) before expiry. Every resolve returns a
non-secret TokenSnapshot (generation + Minted/Reused/Static provenance),
and the source logs mints at INFO and reuses at DEBUG.
- Docker and Daytona share the source through PushCredentialState: an embed
mutex serializes compare -> set-url -> record, a matching generation skips
the set-url exec, and the generation is recorded only after a successful
exec. The clone still mints its own token, but now seeds the source cache
(generation 1) and the last-embedded state, so a refresh mint failure
falls back to the known embedded token instead of believing nothing was
ever embedded.
- RefreshOutcome now reports the remote action (embedded/unchanged/none)
separately from the token snapshot; git_push_via_exec logs token age and
provenance with each push, and refresh failures log the last embedded
generation.
- The run-metadata writer resolves through the sandbox's shared source
instead of minting per snapshot (with its own cached source on resume).
- The ACP refresh-ahead loop reschedules from the embedded token's
expires_at minus the margin instead of a fixed 45-minute interval, which
a cached source would have broken for long turns; static credentials stop
the loop.
Plan: .ai/plans/git-push-token-resilience.md (PR 1: items 3 and 6).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`RunSpec` has 13 fields and no `Default`, so every test that needed one
spelled out all 13 even when it cared about one or two. That put 64
hand-rolled `RunSpec { .. }` literals in `lib/`, and made a single
additive field cost a mechanical edit at roughly 30 sites.
Add `test_run_spec()` to `fabro-types`' feature-gated `test_support`
module: fixed `fixtures::RUN_1`, default settings, a minimal `test`
graph, `test_run_provenance()`, and every optional field unset. Tests
now spread it and only spell out what they assert on.
Adopt it at the 13 literals where the spread removes real duplication,
including the crate-local `test_run_spec` helpers in `fabro-store` and
`fabro-workflow`, which are now defined in terms of the shared fixture.
Tests that populate every field on purpose — the exhaustive `RunSpec`
serde round-trip in particular — keep spelling it out.
No production code and no behavior changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
reference_kind_for_attribute returned the full ReferenceKind, which
includes the config-sourced Dockerfile kind the classifier can never
yield, so the shared graph walker carried a silent `continue` and an
`unreachable!` for impossible kinds; each new config-sourced kind widens
those filler arms, and a classifier extension that reuses an existing
kind would be dropped by the walker without validation, visitation, or a
compiler error. Return a GraphReferenceKind subset instead (converting
into ReferenceKind for validation), making the walker's matches total
with every arm meaningful.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Config {{ env.NAME }} interpolation was removed workspace-wide (tokens
still parse only to fail with a migration message), but several doc
comments and the server-secrets strategy doc still presented it as a
live mechanism, including run goal file paths where the new
workflow-version validation now makes the contradiction user-visible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
