mirror of
https://github.com/fabro-sh/fabro.git
synced 2026-09-16 23:43:10 +00:00
Run 01M0DH033P2XSTHAGVBHG6922F completed 2.8 hours of work, then failed terminally because four consecutive publish pushes hit GitHub's token-replication lag (404 "Repository not found") — the push path had no retry, the failure was misclassified as deterministic, and the same fresh-mint-then-push pattern silently disabled metadata snapshots. This generalizes the clone retry machinery to pushes and makes attempt detail durable. - clone_retry -> git_retry: the classifier's boolean becomes a CredentialContext derived from the token snapshot (fresh App tokens retry 404s as replication lag, mature ones as transient infra, static credentials fail fast), and the attempt/backoff limits become a RetryPlan with layered optional bounds. Clone behavior is preserved: Docker keeps its absolute five-minute deadline, Daytona keeps no deadline. - Pushes take a scoped CredentialLease before the first attempt: it owns the embed mutex for the whole operation, pins the single successful resolve, retries only failed resolves, falls back to the last embedded token when a mint fails, and force-re-embeds the pinned token once after the first auth-shaped failure (drift repair). The margin invariant (REFRESH_MARGIN > every push plan's max_elapsed) guarantees the pinned token outlives the operation; a unit test asserts it. - Sandbox::git_push_ref now takes a RetryPlan and returns PushReport / PushError with per-attempt records (classification, redacted output tail, token generation/provenance/age, credential action, refresh errors). Checkpoint pushes use a 90-second budget; the terminal publish push gets 5 attempts over at most 4 minutes. - The single durable git.push event per push gains a nested attempts array (GitPushAttemptProps, token snapshot flattened to flat fields); stored events without it still deserialize. Publish push failures now carry an explicit failure category — exhausted transient retries stay transient_infra instead of deterministic — plus one bounded cause line per attempt and the last successful push time in the message. - Metadata snapshot degradation records why it degraded: push failures with retryable classifications leave the writer eligible to re-probe at each later checkpoint, and a successful snapshot clears the degraded state and re-arms the warning. Permanent failures keep today's latch. Plan: .ai/plans/git-push-token-resilience.md (PR 2: items 1, 2, 4, 7 and the metadata re-probe). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
97 lines
3.1 KiB
Rust
97 lines
3.1 KiB
Rust
use fabro_agent::Sandbox;
|
|
use fabro_sandbox::shell_quote;
|
|
use fabro_util::error::SharedError;
|
|
use tokio::sync::OnceCell;
|
|
|
|
use crate::sandbox_git::{GIT_REMOTE, exec_err};
|
|
|
|
pub(crate) struct SandboxGitRuntime {
|
|
probe: OnceCell<Result<(), SharedError>>,
|
|
/// When the run last pushed its branch successfully (checkpoint or
|
|
/// publish). Read by the publish failure report so "last success 67s
|
|
/// before the failure" is visible from the run conclusion.
|
|
last_successful_push_at: std::sync::Mutex<Option<chrono::DateTime<chrono::Utc>>>,
|
|
}
|
|
|
|
impl SandboxGitRuntime {
|
|
pub(crate) fn new() -> Self {
|
|
Self {
|
|
probe: OnceCell::new(),
|
|
last_successful_push_at: std::sync::Mutex::new(None),
|
|
}
|
|
}
|
|
|
|
pub(crate) fn record_successful_push(&self) {
|
|
*self
|
|
.last_successful_push_at
|
|
.lock()
|
|
.expect("last push timestamp mutex poisoned") = Some(chrono::Utc::now());
|
|
}
|
|
|
|
pub(crate) fn last_successful_push_at(&self) -> Option<chrono::DateTime<chrono::Utc>> {
|
|
*self
|
|
.last_successful_push_at
|
|
.lock()
|
|
.expect("last push timestamp mutex poisoned")
|
|
}
|
|
|
|
pub(crate) async fn ensure_git_available(
|
|
&self,
|
|
sandbox: &dyn Sandbox,
|
|
) -> Result<(), SharedError> {
|
|
self.probe
|
|
.get_or_init(|| async { probe_sandbox_git(sandbox).await })
|
|
.await
|
|
.clone()
|
|
}
|
|
}
|
|
|
|
impl Default for SandboxGitRuntime {
|
|
fn default() -> Self {
|
|
Self::new()
|
|
}
|
|
}
|
|
|
|
async fn probe_sandbox_git(sandbox: &dyn Sandbox) -> Result<(), SharedError> {
|
|
let temp = sandbox_temp_dir(sandbox, "probe", "git");
|
|
let index = format!("{temp}/index");
|
|
let probe_file = format!("{temp}/probe.txt");
|
|
let command = format!(
|
|
"set -e\n\
|
|
rm -rf {temp_q}\n\
|
|
mkdir -p {temp_q}\n\
|
|
printf probe > {probe_file_q}\n\
|
|
GIT_INDEX_FILE={index_q} {git} read-tree --empty\n\
|
|
blob=$({git} hash-object -w {probe_file_q})\n\
|
|
GIT_INDEX_FILE={index_q} {git} update-index --add --cacheinfo 100644,$blob,probe.txt\n\
|
|
GIT_INDEX_FILE={index_q} {git} write-tree >/dev/null\n\
|
|
rm -rf {temp_q}",
|
|
temp_q = shell_quote(&temp),
|
|
probe_file_q = shell_quote(&probe_file),
|
|
index_q = shell_quote(&index),
|
|
git = GIT_REMOTE,
|
|
);
|
|
exec_ok(sandbox, &command).await
|
|
}
|
|
|
|
fn sandbox_temp_dir(sandbox: &dyn Sandbox, run_id: &str, label: &str) -> String {
|
|
let cwd = sandbox.working_directory().trim_end_matches('/');
|
|
let id = uuid::Uuid::new_v4();
|
|
format!("{cwd}/.fabro/tmp/{label}-{run_id}-{id}")
|
|
}
|
|
|
|
async fn exec_ok(sandbox: &dyn Sandbox, command: &str) -> Result<(), SharedError> {
|
|
let result = sandbox
|
|
.exec_command(command, 30_000, None, None, None)
|
|
.await
|
|
.map_err(|err| {
|
|
SharedError::new(anyhow::Error::new(err).context("sandbox git probe command failed"))
|
|
})?;
|
|
if result.is_success() {
|
|
Ok(())
|
|
} else {
|
|
Err(SharedError::new(anyhow::Error::new(exec_err(
|
|
command, result,
|
|
))))
|
|
}
|
|
}
|