fabro/lib/components/fabro-workflow/src/sandbox_git_runtime.rs
Bryan Helmkamp 1688cd5b91
Retry git pushes with a pinned token and record attempt history
Run 01M0DH033P2XSTHAGVBHG6922F completed 2.8 hours of work, then failed
terminally because four consecutive publish pushes hit GitHub's
token-replication lag (404 "Repository not found") — the push path had no
retry, the failure was misclassified as deterministic, and the same
fresh-mint-then-push pattern silently disabled metadata snapshots. This
generalizes the clone retry machinery to pushes and makes attempt detail
durable.

- clone_retry -> git_retry: the classifier's boolean becomes a
  CredentialContext derived from the token snapshot (fresh App tokens retry
  404s as replication lag, mature ones as transient infra, static
  credentials fail fast), and the attempt/backoff limits become a RetryPlan
  with layered optional bounds. Clone behavior is preserved: Docker keeps
  its absolute five-minute deadline, Daytona keeps no deadline.
- Pushes take a scoped CredentialLease before the first attempt: it owns
  the embed mutex for the whole operation, pins the single successful
  resolve, retries only failed resolves, falls back to the last embedded
  token when a mint fails, and force-re-embeds the pinned token once after
  the first auth-shaped failure (drift repair). The margin invariant
  (REFRESH_MARGIN > every push plan's max_elapsed) guarantees the pinned
  token outlives the operation; a unit test asserts it.
- Sandbox::git_push_ref now takes a RetryPlan and returns PushReport /
  PushError with per-attempt records (classification, redacted output tail,
  token generation/provenance/age, credential action, refresh errors).
  Checkpoint pushes use a 90-second budget; the terminal publish push gets
  5 attempts over at most 4 minutes.
- The single durable git.push event per push gains a nested attempts array
  (GitPushAttemptProps, token snapshot flattened to flat fields); stored
  events without it still deserialize. Publish push failures now carry an
  explicit failure category — exhausted transient retries stay
  transient_infra instead of deterministic — plus one bounded cause line
  per attempt and the last successful push time in the message.
- Metadata snapshot degradation records why it degraded: push failures with
  retryable classifications leave the writer eligible to re-probe at each
  later checkpoint, and a successful snapshot clears the degraded state and
  re-arms the warning. Permanent failures keep today's latch.

Plan: .ai/plans/git-push-token-resilience.md (PR 2: items 1, 2, 4, 7 and
the metadata re-probe).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 10:16:10 -04:00

97 lines
3.1 KiB
Rust

use fabro_agent::Sandbox;
use fabro_sandbox::shell_quote;
use fabro_util::error::SharedError;
use tokio::sync::OnceCell;
use crate::sandbox_git::{GIT_REMOTE, exec_err};
pub(crate) struct SandboxGitRuntime {
probe: OnceCell<Result<(), SharedError>>,
/// When the run last pushed its branch successfully (checkpoint or
/// publish). Read by the publish failure report so "last success 67s
/// before the failure" is visible from the run conclusion.
last_successful_push_at: std::sync::Mutex<Option<chrono::DateTime<chrono::Utc>>>,
}
impl SandboxGitRuntime {
pub(crate) fn new() -> Self {
Self {
probe: OnceCell::new(),
last_successful_push_at: std::sync::Mutex::new(None),
}
}
pub(crate) fn record_successful_push(&self) {
*self
.last_successful_push_at
.lock()
.expect("last push timestamp mutex poisoned") = Some(chrono::Utc::now());
}
pub(crate) fn last_successful_push_at(&self) -> Option<chrono::DateTime<chrono::Utc>> {
*self
.last_successful_push_at
.lock()
.expect("last push timestamp mutex poisoned")
}
pub(crate) async fn ensure_git_available(
&self,
sandbox: &dyn Sandbox,
) -> Result<(), SharedError> {
self.probe
.get_or_init(|| async { probe_sandbox_git(sandbox).await })
.await
.clone()
}
}
impl Default for SandboxGitRuntime {
fn default() -> Self {
Self::new()
}
}
async fn probe_sandbox_git(sandbox: &dyn Sandbox) -> Result<(), SharedError> {
let temp = sandbox_temp_dir(sandbox, "probe", "git");
let index = format!("{temp}/index");
let probe_file = format!("{temp}/probe.txt");
let command = format!(
"set -e\n\
rm -rf {temp_q}\n\
mkdir -p {temp_q}\n\
printf probe > {probe_file_q}\n\
GIT_INDEX_FILE={index_q} {git} read-tree --empty\n\
blob=$({git} hash-object -w {probe_file_q})\n\
GIT_INDEX_FILE={index_q} {git} update-index --add --cacheinfo 100644,$blob,probe.txt\n\
GIT_INDEX_FILE={index_q} {git} write-tree >/dev/null\n\
rm -rf {temp_q}",
temp_q = shell_quote(&temp),
probe_file_q = shell_quote(&probe_file),
index_q = shell_quote(&index),
git = GIT_REMOTE,
);
exec_ok(sandbox, &command).await
}
fn sandbox_temp_dir(sandbox: &dyn Sandbox, run_id: &str, label: &str) -> String {
let cwd = sandbox.working_directory().trim_end_matches('/');
let id = uuid::Uuid::new_v4();
format!("{cwd}/.fabro/tmp/{label}-{run_id}-{id}")
}
async fn exec_ok(sandbox: &dyn Sandbox, command: &str) -> Result<(), SharedError> {
let result = sandbox
.exec_command(command, 30_000, None, None, None)
.await
.map_err(|err| {
SharedError::new(anyhow::Error::new(err).context("sandbox git probe command failed"))
})?;
if result.is_success() {
Ok(())
} else {
Err(SharedError::new(anyhow::Error::new(exec_err(
command, result,
))))
}
}