A Daytona "Sandbox state change in progress" rejection surfacing
through the pipeline lifecycle path ("Pipeline lifecycle operation
failed") matched no transient-infra hint, so the run failure was
categorized deterministic. The condition is a provider lifecycle
transition that finishes on its own — the definition of transient
infrastructure — and the deterministic label misinforms retry
machinery and anyone reading the failure.
Add two transient-infra hints: the provider rejection ("state change
in progress") and the bounded-wait timeout an activation reports when
a stop transition outlives its budget ("sandbox stop still in
progress").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Omitting autoStopInterval from the create-sandbox request inherits
Daytona's server-side default of 15 idle minutes. Daytona counts
inactivity from the last sandbox interaction, and LLM inference never
touches the sandbox, so a single long inference call is enough for the
sandbox to auto-stop mid-run: a workflow failed exactly this way, with
the sandbox entering its stop transition 15 minutes after the last
command while the agent was still thinking.
Send an explicit 120-minute default when lifecycle.auto_stop is unset.
That clears any realistic inference call while still reclaiming
sandboxes leaked by a dead worker. An explicit auto_stop = "0s" still
disables auto-stop entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sandbox exec logs previously required command_len fingerprinting to tell a
push from a credential refresh or a checkpoint commit. The shared git
helpers now instrument their futures with a git_op span, so Daytona's and
Docker's `exec_command: entered` lines inherit the operation label and the
log renders as `git_op{op=push}: exec_command: entered timeout_ms=...`.
Ops: push (git_push_via_exec), refresh-credentials (both providers'
refresh_push_credentials), checkpoint-commit (checked_git_checkpoint),
fetch (fetch_source_run_ref), and metadata-push (the run-metadata snapshot
write). Spans are attached with #[tracing::instrument] — attached to the
future, never an entered() guard held across an await — so they follow the
task across worker threads. No trait or signature changes.
Plan: .ai/plans/git-push-token-resilience.md (PR 3: item 10).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 01M0DH033P2XSTHAGVBHG6922F completed 2.8 hours of work, then failed
terminally because four consecutive publish pushes hit GitHub's
token-replication lag (404 "Repository not found") — the push path had no
retry, the failure was misclassified as deterministic, and the same
fresh-mint-then-push pattern silently disabled metadata snapshots. This
generalizes the clone retry machinery to pushes and makes attempt detail
durable.
- clone_retry -> git_retry: the classifier's boolean becomes a
CredentialContext derived from the token snapshot (fresh App tokens retry
404s as replication lag, mature ones as transient infra, static
credentials fail fast), and the attempt/backoff limits become a RetryPlan
with layered optional bounds. Clone behavior is preserved: Docker keeps
its absolute five-minute deadline, Daytona keeps no deadline.
- Pushes take a scoped CredentialLease before the first attempt: it owns
the embed mutex for the whole operation, pins the single successful
resolve, retries only failed resolves, falls back to the last embedded
token when a mint fails, and force-re-embeds the pinned token once after
the first auth-shaped failure (drift repair). The margin invariant
(REFRESH_MARGIN > every push plan's max_elapsed) guarantees the pinned
token outlives the operation; a unit test asserts it.
- Sandbox::git_push_ref now takes a RetryPlan and returns PushReport /
PushError with per-attempt records (classification, redacted output tail,
token generation/provenance/age, credential action, refresh errors).
Checkpoint pushes use a 90-second budget; the terminal publish push gets
5 attempts over at most 4 minutes.
- The single durable git.push event per push gains a nested attempts array
(GitPushAttemptProps, token snapshot flattened to flat fields); stored
events without it still deserialize. Publish push failures now carry an
explicit failure category — exhausted transient retries stay
transient_infra instead of deterministic — plus one bounded cause line
per attempt and the last successful push time in the message.
- Metadata snapshot degradation records why it degraded: push failures with
retryable classifications leave the writer eligible to re-probe at each
later checkpoint, and a successful snapshot clears the degraded state and
re-arms the warning. Permanent failures keep today's latch.
Plan: .ai/plans/git-push-token-resilience.md (PR 2: items 1, 2, 4, 7 and
the metadata re-probe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every push previously re-minted a fresh GitHub App installation token and
embedded it in the origin URL, so pushes routinely landed inside GitHub's
token-replication lag window (run 01M0DH033P2XSTHAGVBHG6922F failed
terminally on four consecutive fresh-token 404s). Reusing mature tokens
removes the failure trigger and saves two GitHub API calls plus one sandbox
exec per push.
- New fabro_github::token_source::InstallationTokenSource: one cached,
single-flight source per origin repo. Static credentials pass through
(generation 0); App credentials mint through the cache and reuse tokens
until REFRESH_MARGIN (10 min) before expiry. Every resolve returns a
non-secret TokenSnapshot (generation + Minted/Reused/Static provenance),
and the source logs mints at INFO and reuses at DEBUG.
- Docker and Daytona share the source through PushCredentialState: an embed
mutex serializes compare -> set-url -> record, a matching generation skips
the set-url exec, and the generation is recorded only after a successful
exec. The clone still mints its own token, but now seeds the source cache
(generation 1) and the last-embedded state, so a refresh mint failure
falls back to the known embedded token instead of believing nothing was
ever embedded.
- RefreshOutcome now reports the remote action (embedded/unchanged/none)
separately from the token snapshot; git_push_via_exec logs token age and
provenance with each push, and refresh failures log the last embedded
generation.
- The run-metadata writer resolves through the sandbox's shared source
instead of minting per snapshot (with its own cached source on resume).
- The ACP refresh-ahead loop reschedules from the embedded token's
expires_at minus the margin instead of a fixed 45-minute interval, which
a cached source would have broken for long turns; static credentials stop
the loop.
Plan: .ai/plans/git-push-token-resilience.md (PR 1: items 3 and 6).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`RunSpec` has 13 fields and no `Default`, so every test that needed one
spelled out all 13 even when it cared about one or two. That put 64
hand-rolled `RunSpec { .. }` literals in `lib/`, and made a single
additive field cost a mechanical edit at roughly 30 sites.
Add `test_run_spec()` to `fabro-types`' feature-gated `test_support`
module: fixed `fixtures::RUN_1`, default settings, a minimal `test`
graph, `test_run_provenance()`, and every optional field unset. Tests
now spread it and only spell out what they assert on.
Adopt it at the 13 literals where the spread removes real duplication,
including the crate-local `test_run_spec` helpers in `fabro-store` and
`fabro-workflow`, which are now defined in terms of the shared fixture.
Tests that populate every field on purpose — the exhaustive `RunSpec`
serde round-trip in particular — keep spelling it out.
No production code and no behavior changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mark the Slate arm as temporary and record that the SQLite arm's
verified-read and hash-conflict semantics are the intended end state,
so the dual-backend enum reads as a rollout vehicle rather than a
permanent abstraction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LoadedWorkflowVersionClosure owns every file of every version in the
dependency graph, so an advertised Clone invites accidental deep copies
of the whole set. Drop the derive until a consumer needs owned copies.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The positive run-goal tests only asserted fixture shape, so a regression
that stopped pushing the file-goal template root would keep them green
while broken nested includes were silently accepted. Pin the rejection
path directly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Create-time validation of workflow.toml run goals anchored includes at
workflow.toml for inline goals and at the goal file's directory for file
goals, while the run engine inlines the effective goal into the
entrypoint graph and renders it under the entrypoint's template source.
That divergence rejected layouts `fabro run` executes fine and accepted
layouts that fail at render time. Anchor both goal forms at the
entrypoint so validation matches the runtime, and pin the anchor with a
nested-entrypoint test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Batched dependency discovery pre-seeded roots into the path-keyed result
map and reused that map as the traversal-dedup set, so a loaded include
target whose path matched a root (e.g. a goal template including the
graph file that anchors an inline prompt) was recorded but never parsed,
silently accepting invalid template content that per-root discovery used
to reject. Dedup traversal on the full (path, root, content) occurrence
instead, which also stops re-parsing identical duplicate roots.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Replace the discarded dependency-closure map in put/get with a
visitor-based walk so only get_closure retains loaded versions
- Hold the closure root structurally in LoadedWorkflowVersionClosure
instead of asserting its presence in the map with expect()
- Drop the visited-set parameter that guarded against impossible
content-address cycles
- Move template-discovery error source-name extraction into
TemplateDiscoveryError::source_name() where the variants are owned
- Collapse repeated TemplateSource construction into a TemplateRoots
collector and share the config file-reference validation pipeline
between dockerfile and run-goal references
- Deduplicate test helpers (version_id, version_with_goal_file,
impl Into<String> config fixtures)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fireworks reports an account suspension (spending cap reached or unpaid
invoices) as HTTP 412 with code PRECONDITION_FAILED. The status had no
explicit mapping, and the openai_compatible dialect extracts error.type
("error") as the code, so the suspension fell through to InvalidRequest
-- a deterministic request defect -- which suppressed both retry and the
configured model fallback chain. A live run then died mid-stage with
five healthy fallback candidates configured.
No LLM request carries conditional-request preconditions, so a 412 is
never about the request. Map it to AccessDenied, the same family as the
account_deactivated error code: non-retryable on the same provider,
eligible for failover to a provider with independent billing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
reference_kind_for_attribute returned the full ReferenceKind, which
includes the config-sourced Dockerfile kind the classifier can never
yield, so the shared graph walker carried a silent `continue` and an
`unreachable!` for impossible kinds; each new config-sourced kind widens
those filler arms, and a classifier extension that reuses an existing
kind would be dropped by the walker without validation, visitation, or a
compiler error. Return a GraphReferenceKind subset instead (converting
into ReferenceKind for validation), making the walker's matches total
with every arm meaningful.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TemplateDiscoveryError only named a failing source through the Display
strings of its variants: parse and load failures forwarded transparently
to inner errors whose source naming varies (parent for some load
failures, the child path for dynamic dependencies, nothing for I/O
faults), so consumers that need the failing template's path had to
string-round-trip error messages. Carry the parent path on every
variant, exposing a total source_path() accessor, and render parse and
load failures with a parent-naming message above the preserved source
chain.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dependency discovery pre-seeded roots into the path-keyed result map
and reused that map as the traversal-dedup set, so a loaded include
target whose path matched a root was recorded but never parsed (an
include chain that reaches the file anchoring a root silently skips its
content), and a second root occurrence at an already-seeded path was
dropped without parsing. Dedup traversal on the full
(path, root, content) occurrence instead, so every distinct authored
occurrence is parsed exactly once and identical duplicates parse once.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Config {{ env.NAME }} interpolation was removed workspace-wide (tokens
still parse only to fail with a migration message), but several doc
comments and the server-secrets strategy doc still presented it as a
live mechanism, including run goal file paths where the new
workflow-version validation now makes the contradiction user-visible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
It was a one-line passthrough to parse_blob_ref with a single caller,
leaving two names for the same operation; every other consumer calls
parse_blob_ref directly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The manifest_blob and definition_blob filter entries were copy-paste
twins that had to be edited identically; build them from one loop like
the elapsed-ms filters above so the pattern and placeholder cannot
drift apart.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The [BLOB_HASH] placeholder was defined both here and in the shared
json_snapshot_filters regexes, which had to be edited in lockstep. The
fabro_json_snapshot! macro always applies the shared filters to the
rendered string, so the normalizer copies were redundant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
materialize_blob_ref checked is_local_execution for every blob
reference, but the sandbox and run directory are invariant across a
resolution pass, so each check after the first was a redundant (and on
Docker/Daytona, remote) round-trip. The check is now memoized in a
per-pass SandboxLocality threaded through resolve_execution_value.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hydrate_referenced_blobs_with_reader kept a per-call blob cache for the
Json entries but the Text branch bypassed it, so offloaded stage
responses (referenced by both checkpoint values and response.md) were
fetched twice per dump. Both branches now hydrate through the shared
cache, and a test pins the single-fetch behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
