Blob activation cleanups:
- Reuse fabro-db's append_to_path, remove_file_if_exists, and
set_private_permissions instead of local duplicates.
- Return the store directly from activate_blob_storage; the report
wrapper existed only to be logged internally and then discarded.
- Collapse compute_disk_preflight to return the required free bytes
instead of echoing its inputs back through a struct.
- Deduplicate the "exactly one ok row" PRAGMA integrity_check protocol
into one executor-generic helper used by the backup and live checks.
- Skip re-validating a freshly published backup; the staging copy was
validated immediately before the atomic rename, so only a
concurrently published file needs its own validation.
- Replace the manual anyhow wrapping plus duplicate error log in
serve.rs with a plain .context(), matching other startup errors.
- Extract the disk-candidate enumeration in resource_sampler.rs that
available_space_for_path had copy-pasted from sample_disk_resources.
Test fixture cleanups:
- Route all hand-assembled Database::new(..., test_blob_store()) test
fixtures (32 sites) through fabro_store::test_support::test_database,
and make that helper infallible instead of returning an unconditional
Ok.
- Install the test blob schema from fabro_db::BLOBS_MIGRATION_SQL via a
test-support-gated optional dependency instead of a four-level
relative include_str! into fabro-db's migrations directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Make the failure watch channel the single record of the latched
failure; drop the worker task's mirrored local state.
- Replace the hand-rolled wait loop with watch::Receiver::wait_for.
- Extract race_persistence/flush_or_stop helpers so the select!/flush
scaffolding in RunSession::run exists once instead of three times.
- Return RunEventPersistenceError from append_event_to_sink and add a
From impl on Error, replacing four hand-written per-event message
strings with the event name derived from the event itself.
- Dedupe the RunCreated test seed literal in initialize.rs and drop the
dead BlockingHandler::simulate override.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ryyhtbc1eNtCLw8GjrFQXZ
Consolidate the copies that review found across the feature:
- One GITHUB_CREDENTIAL_HELPER / GITHUB_CREDENTIAL_HELPER_KEY pair in
fabro-github, with apply_probe_git_env() for probe commands; the runtime
git bridge, server preflight probe, and live contract test all consume it
so the probes exercise exactly what the bridge configures.
- GitHubRepositoryAccess::resolve_verified_token() owns the
resolve-installations-then-mint choreography shared by server preflight,
workflow initialization, and the live test.
- A shared lookup_installation() helper backs both the shared-installation
resolution and the mint's installation lookup.
- The contents = read|write rule lives once as
RunIntegrationsGithubSettings::contents_permission_allows_repository_access.
- The preflight probe paces retries with fabro-sandbox's exported
replication_backoff() (3s/9s) instead of a contradicting 1s/2s loop, and
shares one run_ls_remote() runner with the existing remote-ref check.
Also: collapse the dead Ok(None) arm and repeated error blocks in the
preflight token check, drop the derivable bridge_entry_count(), privatize
resolve_permissions() behind resolve_integration(), make
GitHubRepositorySlug ordering/hashing allocation-free, and use EnvVars
constants for env names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Brave stays the default. Shops that already vault VENICE_API_KEY
can drop BRAVE_SEARCH_API_KEY by setting
[server.integrations.search] provider = "venice".
Co-authored-by: Cursor <cursoragent@cursor.com>
Carry the resolved GitHub integration (permissions plus declared
additional repositories) as one value from run materialization into
workflow startup, and make the sandbox environment reach every declared
repository through the single managed GITHUB_TOKEN.
- `StartServices.github_permissions` becomes
`github_integration: ResolvedGithubIntegration`; CLI and server
workers build it with `resolve_integration()` after interpolation and
pass it through `SandboxEnvSpec` as one unit.
- `build_sandbox_env` constructs the validated
`GitHubRepositoryAccess` and scopes the App token source to the whole
effective set. Missing credentials or a missing origin are hard
initialization errors when additional repositories are declared;
legacy permissions-only configuration keeps its best-effort behavior.
- When additional repositories are declared, initialization eagerly
resolves each repository's App installation (naming any repository
the App cannot see) and the token itself, so an inaccessible declared
repository fails before the first workflow stage.
- A new `git_bridge` module injects secret-free `GIT_CONFIG_*` entries
into the stage environment: a github.com credential helper that reads
`$GITHUB_TOKEN` at invocation time, per-repository SSH-to-HTTPS
`insteadOf` rewrites, and `GIT_TERMINAL_PROMPT=0`. Entries append
after a valid user-provided Git config overlay and fail clearly on a
malformed one. Contract tests drive the installed git binary against
local fixtures for the rewrite, credential, prefix-collision, and
overlay-preservation behaviors.
- The long-running ACP notice now says all declared repository access
expires together.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The lineage field's `skip_serializing_if` behavior was asserted five times
across three crates. Keep the two assertions in fabro-types, which owns the
attribute, and drop the duplicates:
- Delete `run_created_omits_absent_workflow_version_id` from event/convert.rs,
a copy of the test above it that re-checked another crate's serde attribute.
convert.rs's own responsibility is covered by the existing field assertion.
- Delete `legacy_create_input_persists_without_workflow_version_id`, which ran
the full create() pipeline to prove a hardcoded `None` literal is `None`.
`CreateRunInput` has no such field, so no input could change the result.
- Fold `run_spec_omits_absent_workflow_version_id` into the adjacent legacy-spec
test, which already holds an all-`None` record.
- Drop the off-topic spec re-serialization from run_state.rs's retried_from test.
Add `test_support::test_workflow_version_id()` alongside `test_run_provenance()`
and use it everywhere, replacing eight copies of the same magic seed across five
crates plus two assertion sites that recomputed the hash inline. This also
subsumes retry.rs's private helper of the same shape.
Revert the `run_spec_json` parameterization in the projection round-trip test:
`RunProjection` is a `with_replacement` alias for the canonical type, so the
`Some` and `None` call sites exercise identical code.
Have the two run.created literals that mirror a `RunSpec` read the spec's
lineage field instead of hardcoding `None`, so the mirrors stay accurate once a
producer populates it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 01M0DH033P2XSTHAGVBHG6922F completed 2.8 hours of work, then failed
terminally because four consecutive publish pushes hit GitHub's
token-replication lag (404 "Repository not found") — the push path had no
retry, the failure was misclassified as deterministic, and the same
fresh-mint-then-push pattern silently disabled metadata snapshots. This
generalizes the clone retry machinery to pushes and makes attempt detail
durable.
- clone_retry -> git_retry: the classifier's boolean becomes a
CredentialContext derived from the token snapshot (fresh App tokens retry
404s as replication lag, mature ones as transient infra, static
credentials fail fast), and the attempt/backoff limits become a RetryPlan
with layered optional bounds. Clone behavior is preserved: Docker keeps
its absolute five-minute deadline, Daytona keeps no deadline.
- Pushes take a scoped CredentialLease before the first attempt: it owns
the embed mutex for the whole operation, pins the single successful
resolve, retries only failed resolves, falls back to the last embedded
token when a mint fails, and force-re-embeds the pinned token once after
the first auth-shaped failure (drift repair). The margin invariant
(REFRESH_MARGIN > every push plan's max_elapsed) guarantees the pinned
token outlives the operation; a unit test asserts it.
- Sandbox::git_push_ref now takes a RetryPlan and returns PushReport /
PushError with per-attempt records (classification, redacted output tail,
token generation/provenance/age, credential action, refresh errors).
Checkpoint pushes use a 90-second budget; the terminal publish push gets
5 attempts over at most 4 minutes.
- The single durable git.push event per push gains a nested attempts array
(GitPushAttemptProps, token snapshot flattened to flat fields); stored
events without it still deserialize. Publish push failures now carry an
explicit failure category — exhausted transient retries stay
transient_infra instead of deterministic — plus one bounded cause line
per attempt and the last successful push time in the message.
- Metadata snapshot degradation records why it degraded: push failures with
retryable classifications leave the writer eligible to re-probe at each
later checkpoint, and a successful snapshot clears the degraded state and
re-arms the warning. Permanent failures keep today's latch.
Plan: .ai/plans/git-push-token-resilience.md (PR 2: items 1, 2, 4, 7 and
the metadata re-probe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every push previously re-minted a fresh GitHub App installation token and
embedded it in the origin URL, so pushes routinely landed inside GitHub's
token-replication lag window (run 01M0DH033P2XSTHAGVBHG6922F failed
terminally on four consecutive fresh-token 404s). Reusing mature tokens
removes the failure trigger and saves two GitHub API calls plus one sandbox
exec per push.
- New fabro_github::token_source::InstallationTokenSource: one cached,
single-flight source per origin repo. Static credentials pass through
(generation 0); App credentials mint through the cache and reuse tokens
until REFRESH_MARGIN (10 min) before expiry. Every resolve returns a
non-secret TokenSnapshot (generation + Minted/Reused/Static provenance),
and the source logs mints at INFO and reuses at DEBUG.
- Docker and Daytona share the source through PushCredentialState: an embed
mutex serializes compare -> set-url -> record, a matching generation skips
the set-url exec, and the generation is recorded only after a successful
exec. The clone still mints its own token, but now seeds the source cache
(generation 1) and the last-embedded state, so a refresh mint failure
falls back to the known embedded token instead of believing nothing was
ever embedded.
- RefreshOutcome now reports the remote action (embedded/unchanged/none)
separately from the token snapshot; git_push_via_exec logs token age and
provenance with each push, and refresh failures log the last embedded
generation.
- The run-metadata writer resolves through the sandbox's shared source
instead of minting per snapshot (with its own cached source on resume).
- The ACP refresh-ahead loop reschedules from the embedded token's
expires_at minus the margin instead of a fixed 45-minute interval, which
a cached source would have broken for long turns; static credentials stop
the loop.
Plan: .ai/plans/git-push-token-resilience.md (PR 1: items 3 and 6).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Finish the blob-hash vocabulary unification at the defining signatures:
RunStoreBackend::read_blob, RunStoreHandle, LocalRunStoreBackend, the
HTTP backend impl, RunDatabase::read_blob, and BlobStore::read/exists
all said `id`, which kept re-teaching the old vocabulary at every impl
site and inlay hint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two root-cause fixes for the sandbox failure where an inline Dockerfile
came back from the store as `ARG REDACTED` and the Daytona snapshot
build died on the unset variable.
Entropy redaction measures values, not assignment pairs. The detector
matched `NAME=value` as one token, so an uppercase name merged its
charset into a pure-hex value (which alone can never exceed 4.0 bits)
and pushed the pair over the 4.5-bit threshold — then replaced the
whole pair, destroying the name. `find_entropy_regions` now strips an
identifier-shaped `NAME=` prefix before measuring and redacts only the
value, matching the gitleaks layer's `key=REDACTED` shape.
Execution no longer reads redacted content. Every stored event passes
through the redaction sink, and `load_from_store` rehydrated the
worker's RunSpec from the projection folded from those events — so a
redactor false positive silently rewrote the spec the sandbox builds
from (and changed its snapshot identity). The creation path now writes
the exact spec bytes to the content-addressed blob store and records
`spec_blob` on run.created; `load_from_store` loads the spec from the
blob, keeping the event stream authoritative for run identity,
provenance, and event-recorded blob ids. Retry and fork carry the
source run's `spec_blob` forward, so derived runs stop inheriting the
redacted copy. Runs created before the blob existed fall back to the
folded spec.
The projection and every API surface keep serving the redacted fold;
blobs were already stored unredacted (the workflow bundle carries the
same bytes), so this adds no new exposure at rest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The entropy redactor rewrites NAME=<hex> assignment pairs to a bare
REDACTED, and the worker rehydrates its executable RunSpec from the
projection folded from redacted stored events. Together these broke
Daytona snapshot builds for any run definition whose inline Dockerfile
pins a git SHA: the spec came back as `ARG REDACTED`, the build died on
the unset variable under `set -eu`, and the environment's snapshot
identity silently changed.
Pin the intended contracts with red tests:
- fabro-redact: an assignment whose value alone is below the entropy
threshold survives redaction (pure hex cannot exceed 4.0 bits; only
the name+value charset merge crosses 4.5), and a genuinely
high-entropy value is redacted without destroying the key name.
- fabro-workflow: the spec that load_from_store rehydrates round-trips
byte-identical through the store, including content that looks like
a secret — event redaction must not reach execution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Structural cleanup of the durable pull request creation feature, from a
three-agent review (reuse, quality, efficiency) of the branch:
- Move the supervisor out of handler/ into server/pull_request_supervisor.rs,
collapse its double bookkeeping into one task-id map, and fold the five
copy-pasted failure arms into attempt_pull_request_creation.
- Tag pull_request.failed events with the creation id they resolve, so a
publish-stage failure can never fail an unrelated explicit creation. The
reducer gains PullRequestCreation::succeed/fail transition methods.
- Scan pending creations through a narrow projection-cache accessor instead
of materializing every run summary, raise the scan interval to 30s (notify
covers the live path), and cap retries for runs whose worker cannot even
record a failure.
- Answer "creation already pending" POSTs before taking the per-run create
lock, which a worker can hold for the whole creation.
- Replace the hand-rolled per-run lock map with fabro_store::KeyedMutex.
- Reuse cheap Arc'd projections (cached_run_projection) on the poll endpoint
and in the worker instead of deep-cloning run summaries and diffs.
- Merge ExistingPullRequest into fabro_github::CreatedPullRequest and
extract one reconcile_existing_pull_request helper for both call sites.
- Give the client poll loop a 15-minute deadline; document that Retry-After
and the poll interval are the same constant.
- Resolve a wedged pending creation (run already has a pull request) as a
durable failure instead of skipping it forever.
- Tests: shared wait_for_pull_request_creation helper, a pinned generation-
failure assertion, and a new pipeline test proving reconciliation adopts
an existing PR without an LLM call or create request.
Verified: cargo build --workspace, cargo nextest run --workspace (7,767
passed), nightly clippy -D warnings, fmt --check, insta (no pending), bun
typecheck in fabro-api-client.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The manifest builder's best-effort pre-run push converted every result
into a PreRunPushOutcome that was serialized into GitContext, expanded
into five OpenAPI union arms, and generated into API clients — but no
production path ever read it; every field read was a test.
Delete the concept while preserving the behavior:
- Drop the PreRunPushOutcome enum and GitContext.push_outcome from
fabro-types; GitContext keeps origin_url, branch, optional sha, and
dirty, which remain real execution inputs and provenance.
- Rename the manifest outcome builder to push_manifest_branch_best_effort,
a side-effect-only helper with the same decision rules: skip without an
origin, skip on configured-repository mismatch, skip when the branch is
already synced, otherwise push noninteractively and discard the result
without failing manifest creation or logging raw Git stderr.
- Prove the push through repository state instead of the deleted enum: a
branch ahead of a local bare origin is pushed during manifest build, a
mismatched configured repository is not, and a failing remote helper
still cannot fail manifest creation.
- Remove push_outcome from GitContext in OpenAPI, delete the five-arm
union schemas, and drop the fabro-api type replacement and re-export.
- Keep one regression proving historical run.created events with a nested
push_outcome still deserialize through ordinary unknown-field tolerance
and reserialize to the reduced shape. No migration or event rewrite.
Old JSON carrying the removed field stays readable. Newly generated
clients omit a field older servers required, so new-client-to-old-server
compatibility is intentionally not promised for this pre-1.0 contract.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second pass, from the remaining review findings.
- Wrap the stall watchdog in a `StallWatchdog` type. The call site kept
two parallel `Option`s derived from the same condition and threaded out
an `Option<(CancellationToken, JoinHandle<()>)>`. `monitor_for_stall`
also took two same-typed `CancellationToken` params pointing opposite
directions, where swapping them compiles and yields a run that silently
never stalls.
- Rename `WorkflowAgentQuestionRuntime::stage_id` and
`PendingAgentQuestionBatch::stage_id` to `node_id`. They hold
`node.id`, and the previous commit put them two lines from
`stage_scope.stage_id()`, which returns a real `StageId`.
- Widen the two real-time interview tests. `node_timeout_excludes_
human_input_wait` allowed 20ms of active work against a 50ms budget,
which is tight enough to flake under parallel nextest load. The blocked
wait still outruns the timeout, so both still fail if the pause
regresses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up cleanup on the human-input timeout work.
- Drop the `unresolved_interviews` counter from `InterviewBlockState`. It
duplicated `blocked_stages`, which is non-empty exactly when the run is
blocked.
- Publish block state before emitting `run.blocked` / `run.unblocked` in
both directions, so a listener reading `subscribe()` from an event
callback never sees state that disagrees with the event. The watchdog
still gets a full fresh deadline because it restarts on the unblock
transition.
- Stop panicking in `InterviewBlockState::resolve`. It runs from `Drop`,
where a panic during unwind aborts the process.
- Replace the emitter's `activity_revision` watch channel with a
monotonic timestamp. `record_activity` runs on every agent stream
delta, and the channel woke the watchdog task and re-armed its timer
per event. The watchdog now samples `last_activity()` when its deadline
fires and re-arms only if the run was active, so the hot path is one
clock read and one relaxed store.
- Remove the now-unused `last_event_at()` and `epoch_millis()`.
- Collapse the duplicated blocked/unblocked `select!` arms in
`monitor_for_stall` and `timeout_excluding_interview_wait` into one
loop each, using a branch precondition to park the timer while blocked.
- Handle a dropped block-state sender in
`timeout_excluding_interview_wait` by falling back to a plain deadline
instead of panicking, which also removes a potential busy loop.
- Only compute `stage_id` when the node actually has a timeout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A fork carries a checkpoint from its source run, but its first
run.created event contains only a sandbox plan. Resume previously tried
to reconnect that planned sandbox and failed because no instance
exists. Now a fork resume with a Planned sandbox record builds a fresh
sandbox instead; later fork resumes still reconnect the ready instance,
and a same-run resume with an uninitialized sandbox still fails the
precondition check.
Also consolidates the test module's three near-identical InitOptions
literals into a shared test_init_options helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both from Copilot review feedback on #652.
Do not fall back to `base_sha` for `final_git_commit_sha`:
`base_sha` is where the run started, not what it produced. When a run made
commits but no SHA was tracked, the conclusion reported the base commit as the
run's final commit — a durable, API-exposed field — and publish then checked
the pushed branch against it, failing a branch that was pushed correctly.
The SHA is now only required where it is actually used: verifying the remote
head before opening a pull request. Pushing never needed it, since the refspec
sends whatever the branch points at. A run with no tracked SHA therefore still
pushes its branch and succeeds; it fails only if a pull request is requested,
where an unverifiable head is a real problem.
Route branch names with slashes in the GitHub twin:
Run branches are `fabro/run/<id>`. GitHub routes the branch as the remainder
of the path, but the twin declared a single-segment `{branch}` capture, so
every real run branch 404'd against it. Now a wildcard, with a test covering
the slashed case that the existing single-segment tests missed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three follow-ups from the efficiency review of the publish pipeline.
Check the branch before spending an LLM call:
`open_pull_request` generated the PR title and body first and only then
verified the remote branch pointed at the run's final commit. Every stale
branch therefore cost a full content generation before failing. The
verification is the cheap check, so it now runs first.
Tolerate GitHub read-after-write lag:
`GET /repos/{owner}/{repo}/branches/{branch}` is replica-served and can briefly
report the previous commit, or 404 for a branch that is new on the remote,
right after the push publish just made. It was read once with no retry. Since
publish failures are terminal, a replica that had not caught up yet would
discard a fully successful run. It is now read up to three times.
These two land together on purpose: the LLM call was the only thing buying
slack against the race, so reordering without the retry would have made it
more likely.
Keep commit SHAs out of failure classification:
`classify_failure_reason` substring-matches bare "500", "502", "503" and "504"
as transient-infra hints. Both publish messages embed a commit SHA, and a
40-char hex string contains one of those often enough to matter, so a
deterministic failure could be reported as transient. Long hex runs are now
masked before matching; the three-digit status codes those hints look for are
too short to be affected. The hex regex is shared with
`normalize_failure_reason`, which already had its own copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follow-up cleanup on the publish-failures change.
Error model:
- Collapse `Error::{Engine, Publish, Handler}` into one `Error::Stage` with an
`ErrorStage` discriminator. The three shared a field shape and had to be
edited together in four match groups; nine near-identical constructors
become two private helpers.
- Add `Error::failure_reason()`, replacing the same error -> FailureReason
mapping written out in four places.
- Publish errors are now terminal. Publish runs once, after execution, so no
caller could ever act on the retryable classification.
Publish phase:
- Fix: a branch that was pushed is now still reported when pull request
creation fails afterwards. `PublishOutcome` records what happened and
carries the error separately, instead of hiding both behind a `Result`.
- Drop `PublishOutcome::NoChanges`, which no consumer distinguished from
`Published { pr_url: None }`.
- Move publish onto `Concluded` as methods and replace three near-identical
precondition guards with one `publish_target()`.
Pull requests:
- `maybe_open_pull_request` -> `open_pull_request` returning the record
directly. Both callers already reject empty diffs, so the `Ok(None)` path
was unreachable.
- Drop `CreatedPullRequest.head_sha`, which echoed back its own input.
GitHub client:
- Delete `branch_exists`, which had no callers and duplicated
`branch_head_sha`. Give `branch_head_sha` the `_with_client` split every
sibling has and port the tests to `MockHttpClient`.
- Collapse the copy-pasted credential match in `resolve_clone_credentials`.
Events:
- `PullRequestCreated.head_sha` is `Option<String>` instead of using an empty
string to mean absent.
- Centralize the run-branch refspec in `lifecycle::push_run_branch`, so
`git.push` reports a branch name from both emitters as documented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
