When a run declares additional repositories, preflight now proves the
whole effective set works instead of treating a minted token as proof:
- It constructs the same validated `GitHubRepositoryAccess` used by
runtime initialization, so the two paths cannot disagree.
- In App mode it first resolves every repository's installation with
the App JWT and requires one shared installation ID, naming any
repository the App cannot see before the mint; then it mints the one
scoped token, failing with the raw error on rejection.
- Every effective repository gets a non-interactive
`git ls-remote <url> HEAD` probe through a shared helper that keeps
the token out of the URL, argv, and errors (a credential helper reads
GITHUB_TOKEN from the child environment), retries auth-shaped
failures with the same token to cover replication lag (classified
via fabro_sandbox::classify_failure), and reports one check per
repository in deterministic primary-first order under bounded
concurrency.
- A resolved run environment that defines GH_TOKEN produces a warning
(gh prefers it over the managed token) without failing preflight.
- With no additional repositories declared, the primary-only mint
check is byte-for-byte unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carry the resolved GitHub integration (permissions plus declared
additional repositories) as one value from run materialization into
workflow startup, and make the sandbox environment reach every declared
repository through the single managed GITHUB_TOKEN.
- `StartServices.github_permissions` becomes
`github_integration: ResolvedGithubIntegration`; CLI and server
workers build it with `resolve_integration()` after interpolation and
pass it through `SandboxEnvSpec` as one unit.
- `build_sandbox_env` constructs the validated
`GitHubRepositoryAccess` and scopes the App token source to the whole
effective set. Missing credentials or a missing origin are hard
initialization errors when additional repositories are declared;
legacy permissions-only configuration keeps its best-effort behavior.
- When additional repositories are declared, initialization eagerly
resolves each repository's App installation (naming any repository
the App cannot see) and the token itself, so an inaccessible declared
repository fails before the first workflow stage.
- A new `git_bridge` module injects secret-free `GIT_CONFIG_*` entries
into the stage environment: a github.com credential helper that reads
`$GITHUB_TOKEN` at invocation time, per-repository SSH-to-HTTPS
`insteadOf` rewrites, and `GIT_TERMINAL_PROMPT=0`. Entries append
after a valid user-provided Git config overlay and fail clearly on a
malformed one. Contract tests drive the installed git binary against
local fixtures for the rewrite, credential, prefix-collision, and
overlay-preservation behaviors.
- The long-running ACP notice now says all declared repository access
expires together.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add `GitHubRepositoryAccess`, the secret-free validated value describing a
run's effective GitHub repository set: the primary origin repository plus
the declared additional repositories with the shared permission map.
- The constructor normalizes HTTPS and both SSH origin spellings to one
primary slug, rejects a missing or non-GitHub origin when additional
repositories are declared, rejects primary duplication and cross-owner
additional repositories, and re-checks that interpolated permissions
carry `contents = "read"|"write"` — exposing targets in deterministic
primary-first order.
- `resolve_shared_installation` resolves every target's App installation
with the App JWT and requires one shared installation ID, naming the
repository the App cannot see before any mint.
- The installation-token mint now accepts a repository-name list; the
single-repository entry points delegate to it, and the request body
lists every projected name with the shared permissions.
- `InstallationTokenSource::for_access` builds a source over the access
value; caching, refresh margin, and single-flight are unchanged.
- The scripted `MockHttpClient` and test RSA key move to a shared
crate-internal `tests_mock` module.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add `additional_repositories` to `[run.integrations.github]`: a list of
full `owner/repository` slugs, beyond the implicit run origin, that the
minted GITHUB_TOKEN must cover.
- `GitHubRepositorySlug` gains FromStr, Display, string serde, and
case-insensitive Eq/Ord/Hash identity while preserving the submitted
spelling for display and serialization.
- The config layer keeps raw strings; the higher-precedence list
replaces the lower one wholesale, with `[]` as an explicit clear,
resolving independently from the `permissions` map.
- Resolution validates each entry with indexed error paths: slug
grammar, case-insensitive duplicates, one shared owner, the
499-repository cap, and a required `contents = "read"|"write"`
permission (templated values are re-checked at the runtime boundary).
- `RunIntegrationsGithubSettings` resolves permissions and repositories
together through `resolve_integration()` so consumers cannot pick up
one without the other; the field is omitted from serialization when
empty, keeping single-repository settings byte-identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lineage field's `skip_serializing_if` behavior was asserted five times
across three crates. Keep the two assertions in fabro-types, which owns the
attribute, and drop the duplicates:
- Delete `run_created_omits_absent_workflow_version_id` from event/convert.rs,
a copy of the test above it that re-checked another crate's serde attribute.
convert.rs's own responsibility is covered by the existing field assertion.
- Delete `legacy_create_input_persists_without_workflow_version_id`, which ran
the full create() pipeline to prove a hardcoded `None` literal is `None`.
`CreateRunInput` has no such field, so no input could change the result.
- Fold `run_spec_omits_absent_workflow_version_id` into the adjacent legacy-spec
test, which already holds an all-`None` record.
- Drop the off-topic spec re-serialization from run_state.rs's retried_from test.
Add `test_support::test_workflow_version_id()` alongside `test_run_provenance()`
and use it everywhere, replacing eight copies of the same magic seed across five
crates plus two assertion sites that recomputed the hash inline. This also
subsumes retry.rs's private helper of the same shape.
Revert the `run_spec_json` parameterization in the projection round-trip test:
`RunProjection` is a `with_replacement` alias for the canonical type, so the
`Some` and `None` call sites exercise identical code.
Have the two run.created literals that mirror a `RunSpec` read the spec's
lineage field instead of hardcoding `None`, so the mirrors stay accurate once a
producer populates it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RunId is a ULID, so its embedded timestamp is truncated to whole
milliseconds, while Variable.updated_at comes from Utc::now() with
sub-millisecond precision. When the variable write and the run creation
landed in the same millisecond, the run id compared as earlier and the
assertion failed. Truncate the variable timestamp to milliseconds so
both sides use the same precision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sandbox exec logs previously required command_len fingerprinting to tell a
push from a credential refresh or a checkpoint commit. The shared git
helpers now instrument their futures with a git_op span, so Daytona's and
Docker's `exec_command: entered` lines inherit the operation label and the
log renders as `git_op{op=push}: exec_command: entered timeout_ms=...`.
Ops: push (git_push_via_exec), refresh-credentials (both providers'
refresh_push_credentials), checkpoint-commit (checked_git_checkpoint),
fetch (fetch_source_run_ref), and metadata-push (the run-metadata snapshot
write). Spans are attached with #[tracing::instrument] — attached to the
future, never an entered() guard held across an await — so they follow the
task across worker threads. No trait or signature changes.
Plan: .ai/plans/git-push-token-resilience.md (PR 3: item 10).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A Daytona "Sandbox state change in progress" rejection surfacing
through the pipeline lifecycle path ("Pipeline lifecycle operation
failed") matched no transient-infra hint, so the run failure was
categorized deterministic. The condition is a provider lifecycle
transition that finishes on its own — the definition of transient
infrastructure — and the deterministic label misinforms retry
machinery and anyone reading the failure.
Add two transient-infra hints: the provider rejection ("state change
in progress") and the bounded-wait timeout an activation reports when
a stop transition outlives its budget ("sandbox stop still in
progress").
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Daytona rejects start/stop with HTTP 400 "State change in progress"
while a lifecycle transition is in flight, and activate() only handled
the Started and Starting states: any other state fell through to
start(), which surfaced the rejection as a hard failure. A run died
exactly this way when an inactivity auto-stop began seconds before the
stage finished — activate() saw the sandbox mid-stop and failed the
whole run 35ms later. The cleanup stop() then failed on the same
rejection.
Transitions finish on their own within seconds, so treat them as
wait-and-retry conditions:
- activate() now waits out a Stopping sandbox and dispatches on
whatever state the transition lands on.
- start() and stop() retry the rejected call within a bounded budget,
re-inspecting state between attempts: a transition that lands on
Started needs no further start, and one that lands on Stopped or
Destroyed needs no further stop.
All call sites go through these three provider methods, so no
lifecycle-layer changes are needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Omitting autoStopInterval from the create-sandbox request inherits
Daytona's server-side default of 15 idle minutes. Daytona counts
inactivity from the last sandbox interaction, and LLM inference never
touches the sandbox, so a single long inference call is enough for the
sandbox to auto-stop mid-run: a workflow failed exactly this way, with
the sandbox entering its stop transition 15 minutes after the last
command while the agent was still thinking.
Send an explicit 120-minute default when lifecycle.auto_stop is unset.
That clears any realistic inference call while still reclaiming
sandboxes leaked by a dead worker. An explicit auto_stop = "0s" still
disables auto-stop entirely.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run the local git steps of the Docker exact checkout under the shared
clone deadline instead of a fixed 10s timeout, so materializing a large
working tree cannot time out and abandon a running checkout in the
container.
Check the admitted commit out onto the admitted branch rather than
detaching. A detached HEAD makes `rev-parse --abbrev-ref HEAD` return
"HEAD", which the git setup helper maps to no base branch, silently
dropping it for callers that rely on it. Daytona does the same after its
native clone and now verifies the resulting HEAD the way Docker does.
Fetch the exact commit at the same depth a branch clone uses, so both
paths can reach the same number of parent commits, and stop suggesting
GitHub App credentials when a purely local git step fails.
Document that reachability of the commit from the branch is an
admission-time invariant that the sandbox layer does not re-verify.
Run 01M0DH033P2XSTHAGVBHG6922F completed 2.8 hours of work, then failed
terminally because four consecutive publish pushes hit GitHub's
token-replication lag (404 "Repository not found") — the push path had no
retry, the failure was misclassified as deterministic, and the same
fresh-mint-then-push pattern silently disabled metadata snapshots. This
generalizes the clone retry machinery to pushes and makes attempt detail
durable.
- clone_retry -> git_retry: the classifier's boolean becomes a
CredentialContext derived from the token snapshot (fresh App tokens retry
404s as replication lag, mature ones as transient infra, static
credentials fail fast), and the attempt/backoff limits become a RetryPlan
with layered optional bounds. Clone behavior is preserved: Docker keeps
its absolute five-minute deadline, Daytona keeps no deadline.
- Pushes take a scoped CredentialLease before the first attempt: it owns
the embed mutex for the whole operation, pins the single successful
resolve, retries only failed resolves, falls back to the last embedded
token when a mint fails, and force-re-embeds the pinned token once after
the first auth-shaped failure (drift repair). The margin invariant
(REFRESH_MARGIN > every push plan's max_elapsed) guarantees the pinned
token outlives the operation; a unit test asserts it.
- Sandbox::git_push_ref now takes a RetryPlan and returns PushReport /
PushError with per-attempt records (classification, redacted output tail,
token generation/provenance/age, credential action, refresh errors).
Checkpoint pushes use a 90-second budget; the terminal publish push gets
5 attempts over at most 4 minutes.
- The single durable git.push event per push gains a nested attempts array
(GitPushAttemptProps, token snapshot flattened to flat fields); stored
events without it still deserialize. Publish push failures now carry an
explicit failure category — exhausted transient retries stay
transient_infra instead of deterministic — plus one bounded cause line
per attempt and the last successful push time in the message.
- Metadata snapshot degradation records why it degraded: push failures with
retryable classifications leave the writer eligible to re-probe at each
later checkpoint, and a successful snapshot clears the degraded state and
re-arms the warning. Permanent failures keep today's latch.
Plan: .ai/plans/git-push-token-resilience.md (PR 2: items 1, 2, 4, 7 and
the metadata re-probe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every push previously re-minted a fresh GitHub App installation token and
embedded it in the origin URL, so pushes routinely landed inside GitHub's
token-replication lag window (run 01M0DH033P2XSTHAGVBHG6922F failed
terminally on four consecutive fresh-token 404s). Reusing mature tokens
removes the failure trigger and saves two GitHub API calls plus one sandbox
exec per push.
- New fabro_github::token_source::InstallationTokenSource: one cached,
single-flight source per origin repo. Static credentials pass through
(generation 0); App credentials mint through the cache and reuse tokens
until REFRESH_MARGIN (10 min) before expiry. Every resolve returns a
non-secret TokenSnapshot (generation + Minted/Reused/Static provenance),
and the source logs mints at INFO and reuses at DEBUG.
- Docker and Daytona share the source through PushCredentialState: an embed
mutex serializes compare -> set-url -> record, a matching generation skips
the set-url exec, and the generation is recorded only after a successful
exec. The clone still mints its own token, but now seeds the source cache
(generation 1) and the last-embedded state, so a refresh mint failure
falls back to the known embedded token instead of believing nothing was
ever embedded.
- RefreshOutcome now reports the remote action (embedded/unchanged/none)
separately from the token snapshot; git_push_via_exec logs token age and
provenance with each push, and refresh failures log the last embedded
generation.
- The run-metadata writer resolves through the sandbox's shared source
instead of minting per snapshot (with its own cached source on resume).
- The ACP refresh-ahead loop reschedules from the embedded token's
expires_at minus the margin instead of a fixed 45-minute interval, which
a cached source would have broken for long turns; static credentials stop
the loop.
Plan: .ai/plans/git-push-token-resilience.md (PR 1: items 3 and 6).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`RunSpec` has 13 fields and no `Default`, so every test that needed one
spelled out all 13 even when it cared about one or two. That put 64
hand-rolled `RunSpec { .. }` literals in `lib/`, and made a single
additive field cost a mechanical edit at roughly 30 sites.
Add `test_run_spec()` to `fabro-types`' feature-gated `test_support`
module: fixed `fixtures::RUN_1`, default settings, a minimal `test`
graph, `test_run_provenance()`, and every optional field unset. Tests
now spread it and only spell out what they assert on.
Adopt it at the 13 literals where the spread removes real duplication,
including the crate-local `test_run_spec` helpers in `fabro-store` and
`fabro-workflow`, which are now defined in terms of the shared fixture.
Tests that populate every field on purpose — the exhaustive `RunSpec`
serde round-trip in particular — keep spelling it out.
No production code and no behavior changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Fold Docker's exact-checkout path into clone_github_repo so the auth,
retry, symlink, and bookkeeping skeleton is shared with branch clones
- Skip the Daytona SDK clone for exact checkouts: init and shallow-fetch
the admitted commit directly instead of cloning the default branch and
discarding it
- Combine the detach checkout and HEAD verification into one shell
command, saving an exec round trip per init
- Drop the spec-level decide_clone pre-checks that duplicated the
constructors' fail-fast validation
- Share a CloneAttemptFailure struct in clone_retry and the GIT command
prefix constant across git command builders
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mark the Slate arm as temporary and record that the SQLite arm's
verified-read and hash-conflict semantics are the intended end state,
so the dual-backend enum reads as a rollout vehicle rather than a
permanent abstraction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
