Petri now ends a run's lifetime at its first failed store write: it
records nothing after it, fails no firing for it, and returns
CoordinatorError::StoreFailed. The run is not over; the next lifetime
resumes it from what the store holds. Fabro read that error as an
unfinished run and failed it.
- engine: RunError::StoreFailed, returned without reading the record
back, and Conclusion::Interrupted for it.
- worker: an interrupted run gets no terminal lifecycle record; the
worker exits with EX_TEMPFAIL (75, the new ExitClass::Interrupted).
- server: WorkerExit carries the exit code. An interrupted worker's run
goes back to the scheduler in resume mode through the relaunch a
restart takes (lease release, recovery, start_requested + runnable),
now shared with reconcile_on_startup. The in-process path does the
same. A run is resumed at most MAX_STORE_INTERRUPTIONS (3) times per
server; the next interruption fails it. A pending cancel, a run that
ended or was deleted, and a shutdown also end it as before.
Tests: an engine run over a store whose first lease write fails is
interrupted with no finish, and a resume finishes it; the server
relaunches an interrupted worker in resume mode, fails the run after the
bound, and fails a worker that exits 1 as before; exit code 75.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petri now refuses to resume a run whose creation a crash cut short (the
key is stored, the root invocation is not) with HostError::NotStarted,
and starts it again when the host runs it under the same key. The engine
used its own guard, check_resumable, which failed the run with
NothingToResume instead.
Execution::Resume now carries the admitted graphs, and a resume Petri
answers with NotStarted starts the run from them. The worker loads the
graphs in resume mode too, as does the server's in-process path. The
guard and RunError::NothingToResume are gone. New test:
a_resume_of_a_run_that_never_started_starts_it_again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rewrite the sandbox credential store only when the token source mints a
new generation, and skip the refresh loop for static tokens. Build the
run's read-only token source once in the worker and share it between the
workspace fetch and stage Git access. Drop unreachable branches in
for_run, reuse the shared contents-permission check and constants, and
fold the duplicated token resolution and test setup into helpers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pin Petri at e46845b, the merge of its executor layer change. The stage
credential layer is now a Petri `SpawnEnv` applied with
`EnvHandle::with_spawn_env`, so Petri forwards every other environment
method and applies the layer to one-shot containers as well as processes.
A container gets the managed GITHUB_TOKEN, but no credential store refresh
or Git helper configuration: the store lives in the scope's sandbox, which
the container does not share.
The managed GITHUB_TOKEN now replaces one set in the workflow environment,
an ACP agent's environment or the sandbox's own, as Fabro's stage
environment did before Petri. The token carries exactly the access the run
declares; a stage that needs other access changes its declaration.
The new pin also keeps a timeout as the failure a partial success came
from: `PartialSuccess.underlying` is now an `UnderlyingFailure`, so the
projection reports "the step timed out" for a partial success converted
from a timeout. The run format moves from 7 to 8, which the attach JSON
snapshot records; runs stored before this pin are refused, as with earlier
format changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petri now treats `petri.launch_model` and `petri.launch_provider` as the
model a run's flags ask for, above the file layers and the graph's
defaults. A host's last-resort default moved to `petri.default_model` and
`petri.default_provider`. Pin Petri at the merge of that change and bind
to it: the explicit `--model`/`--provider` flags go to the launch
variables, and the model the settings resolved (or the catalog default)
goes to the default variables.
`Launch` now names the two pairs `model`/`provider` and
`default_model`/`default_provider`, matching Petri, in place of the
separate override fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host provider manages no networking and refuses any policy but its
default, so sending the resolved AllowAll to local runs failed every
non-dry-run local execution. Apply the run's policy only on container
backends; dry runs, which always use the host backend, are covered by the
same check.
Also fold the Docker environment test helper into one that takes a typed
network mode, share the probe setup between the live Docker and Daytona
network tests, count canary hits per mode, and bind the run environment
once in the worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checkpoints were only enabled for runs with a GitHub source, so local
folder runs, empty Local runs and dry runs stopped committing. `fabro
diff` then failed for them, and their checkpoint, run branch and diff
records disappeared from the event stream.
A run whose workspace is on the host now commits checkpoints there
again, without pushing, as on main. Docker and Daytona runs with no
GitHub source still record execution checkpoints without Git commits,
so a sandbox image without `git` cannot fail the run.
The scenario tests for crash recovery go back to asserting commits. A
workspace deleted while the run is down now fails the resumed run,
since the server keeps no copy to restore it from.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker minted one read token at launch and a fresh push token for
every checkpoint. The read token expired an hour into a run, so a
workspace acquired later fetched with a dead credential. Minting per
push put every push in GitHub's token-replication window, where a token
minted moments earlier is rejected with 404 "Repository not found".
The worker now keeps two InstallationTokenSource caches for the run, a
read-only one for fetches and a contents: write one for pushes. Each
fetch and push resolves through its source, which reuses one token until
it nears expiry and then mints the next. Petri's RunSource asks a
SourceCredentials provider on every fetch instead of holding a fixed
credential.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Share one credential header helper between the in-sandbox fetch and
the run branch push
- Keep only the target branch, goal, and model on the publisher instead
of a full run spec copy
- Load the worker's LLM catalog once, and mint the read token only when
the run checks something out
- Pass the source explicitly to checkpoint fetch helpers, dropping
unreachable branches, and reuse has_object in has_commit
- Move the run patch into the publication instead of cloning it, and
build it only when a publisher exists
- Add test fixture helpers for file sources and recording publishers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pushing the run branch and opening the pull request now happen in the
run's worker, in Fabro's run_finished hook, after the last stage and
before the run's terminal record, as the legacy publish step did. A
failed push or pull request fails the run with publish_failed instead of
leaving a warning on a run that already succeeded.
fabro-petri gains a RunPublisher the hooks call for a successful run with
its run branch, final commit, snapshot repository and patch; the worker's
GitHub publisher pushes from the snapshot repository with a push token it
mints at that moment, opens the pull request its settings ask for, and
records it. The worker resolves the server's GitHub credentials itself
for both the read-only checkout token and the push token, so the server
no longer hands it a clone credential or publishes after the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the Petri cutover, a run with a GitHub target started in an empty
workspace, nothing pushed its run branch to GitHub, and nothing asked for
the automatic pull request when it succeeded.
Fabro's hooks now check a fresh run's GitHub target out inside the
sandbox when Petri hands them the scope, before the first stage: the
workspace fetches the selected commit, tag or branch at the run's clone
depth, with a read-only token the server resolves at each worker launch.
The worker scrubs the token from its environment at startup and presents
it only to the fetch, so it never lands in the repository or its remote.
The files belong to the sandbox user, so git accepts them.
The same checkout seeds the workspace's snapshot repository with the
starting commit. Checkpoint bundles from a shallow clone then import, a
stage's own commits never make a bundle carry the source's history, a
restore into a fresh sandbox fetches the base again and applies the run's
commits, and a fork carries the base with its checkpoints.
When a successful GitHub-target run ends, the server pushes its final
commit from the snapshot repository to fabro/run/<id> with its own write
credentials, then, when the run changed files and asks for one, records
the pull request request for the existing creation supervisor. A failed
push or request is a warning notice on the run. The manual pull request
endpoint shares the request step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retry forked the source run at its last checkpoint and reran the failed
stage. It now creates a new run from the source's saved spec and starts
the workflow from the beginning in a fresh workspace, as retry did
before the Petri cutover. The new run records `retried_from` and no
`fork_source_ref`.
Retry no longer needs a checkpoint, a retained workspace, or a published
run branch, so it works for any terminal run that is not archived. To
continue from where a run stopped, fork it at a checkpoint.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The store checks that captured bytes match their digest in every build,
hashing on a blocking thread, and the upload handler relies on that
check instead of hashing a second time.
- Capture bytes travel as `Bytes` from the hooks through the client and
the store, so uploads and retries share one buffer.
- The artifact writer takes the run ID from the hooks, so objects are
stored under the run their records name.
- Concurrent captures of the same file and content wait on one another,
so the file is uploaded and recorded once.
- When a record append fails, the hooks re-read the run's captures and
treat a record that did land as done, so a lost response does not
record the capture twice.
- An upload that finishes after its run was deleted removes itself,
instead of leaving an object nothing references.
- Listing a run's stage artifacts skips everything under `captures/`, so
an unexpected object there cannot fail the listing or the ZIP.
- The capture record derives its `digest` key from its source instead of
storing it twice, still writing and checking it on the wire.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The artifact writer takes the digest the hooks already computed, so
captured bytes are hashed once on each side, and the dead integrity
error goes away.
- The writer is a required part of HooksSpec, not an optional field on
RunRequest, so a run with capture globs always has a writer and the
no-writer error goes away.
- ArtifactStore routes put/get and the capture methods through shared
put_at/get_at helpers.
- The upload handler parses the digest with parse_blob_hash_path before
any store reads, and builds its size-limit message from the constant.
- The default local artifact root comes from one helper used by both
config resolution and the storage-dir override.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Keep plugin-era Daytona lease fingerprints: read only DAYTONA_API_URL and
DAYTONA_ORGANIZATION_ID (no URL alias, no placement target), and stop
forwarding DAYTONA_SERVER_URL and DAYTONA_TARGET to the worker.
- Take the Docker fingerprint and network from this process's DOCKER_HOST,
the endpoint the Docker client actually connects to; make the provider
configuration's fields private.
- Return an error instead of panicking when Petri supplies no Host registry.
- Run deletion reads the Daytona key only for a Daytona run, and a forced
or restarted delete goes on when the secret store fails, as it does for
every other prune failure.
- Stop putting DAYTONA_API_KEY in the worker's environment; the worker reads
it from the vault. Give the worker's Daytona client the shared HTTP client.
- Fork, rewind and retry no longer read the vault: a fork acquires no sandbox.
- Remove the dead worker plugin forwarding and document that runs execute
only on the built-in providers.
- Build every Petri runtime through providers::standard_runtime or
bare_runtime, with a Clippy lint against Runtime::standard/bare.
- Share the Docker require-or-skip policy in fabro-test, tighten the Host
scope assertion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Load the Daytona key for fork and prune through one AppState method
instead of two copied vault reads, and pass the sandbox configuration
into runtime_spec rather than building it and overwriting it. The
worker reuses the CLI's process_env_var lookup.
Share one Docker availability check and the backend-requirement
variable through fabro-test, drop the built-in plugin path and pin
constants nothing reads any more, and let enabled_plugins() exclude the
bundled kinds itself. Refresh the comments and the spawn_env test that
still described built-in plugins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Register lazy Host, Docker, and Daytona factories for Petri execution,
fork, and prune. Share server provider configuration, preserve lease
fingerprints, and source Daytona credentials from the vault.
Remove built-in plugin setup and skip gates; add a release-mode worker
and prune regression to catch the failure that blocked nightly builds.
Co-Authored-By: Codex <noreply@openai.com>
Reduce the workspace Cargo.toml convention block to three lines: Lithos
libraries track `main`, Cargo.lock picks the commits (move one with
`cargo update -p <crate>`), and unmerged library work is tried with an
uncommitted `[patch]`. Drop the instruction to hand-review lockfile diffs
and trim the restatements in the pebble and petri comments, the
fabro-petri README and module doc, AGENTS.md, and the docker test doc.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every lithoscomputer git dependency (sandbox-driver, pebble, petri,
lithos-llm, twins) now uses `branch = "main"` instead of an exact rev,
matching the libraries, so the workspace resolves one Cargo source per
repository. Cargo.lock is the single place the commits are chosen; move
one with `cargo update -p <crate>`.
The lockfile keeps every commit except sandbox-driver, which moves from
583a164 to b30203c: Daytona removed its paginated sandbox listing, and
b30203c lists through cursors instead (it also moves the driver's
daytona-sdk-rust dependency to 0e69058). The Daytona auth-probe test
mocks now serve the cursor endpoint the driver calls.
CI reads the sandbox-driver commit for the plugin install from the
lockfile through cargo metadata instead of from Cargo.toml.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`lib/components/fabro-pebble-sandbox` moved into pebble as
`pebble_coding_agent::sandbox_driver` (lithoscomputer/pebble#27): the
`Environment` over a driver handle (`SandboxEnvironment`, was
`PebbleSandbox`), the `SandboxExec` policy, the port routes, and
`display_for_log`. The pebble pin moves to that branch head, 6d03b3b,
with the `sandbox-driver` feature on (`sandbox-driver-test-util` for the
server's tests, which take `MockSandbox` from pebble now). Nothing in the
crate was Fabro's by design; what was Fabro's stays: `SecretRedactor`
moves to `fabro-redact` as pebble's `Redactor` over `redact_string`, and
the log renderer takes it where a driver failure is rendered.
`fabro-petri` hands pebble types to Petri's crates, so Petri must pin the
same pebble revision: the petri pins move to lithoscomputer/petri#36
(9ee3f85), which pins pebble at the same head. Both re-pin to the pebble
merge commit together once #27 merges.
The 14 pebble commits between the pins fold the session projection's
lifetime tallies into `SessionProjection::totals` (and `PromptDelta`'s
into a flattened `totals`, which renames the prompt's `subagents` key to
`subagent_counts`, as the projection's already was). The stage progress
fold, the runs handler, the OpenAPI schema, the generated client model,
and the round-trip test follow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worker-backed run's in-memory managed run settled only when the
worker process exited. GET /runs/{id} reads the stored summary, which
the projector ends at Petri's own `run.finished` record, a moment
before the worker stores Fabro's terminal lifecycle record and exits.
The delete precheck prefers the managed run, so a delete issued in that
window was refused with 409 "cannot remove active run". Against a real
worker the window hit about six times in ten.
The server sees both records before they are stored: `run.finished` on
the coordinator log through the worker's records endpoint, and the
terminal lifecycle record through the platform-records endpoint. The
managed run now settles at either, ahead of the store, so the view
never reports the run ended while the managed run still says running.
The stream follower no longer reopens a settled run with the records
that precede its terminal one, and the worker's exit keeps the settled
status: it reaps the process, records a missing terminal record as it
did, and takes the store's status only when the store ended the run
differently. The mapping from Petri's finish to the run's status is the
projection's own, shared with its fold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An intent's goal override, and a `[run.goal]` layer, reached the run's
display graph and its settings but not Petri's check, so the agent stages
executed with the workflow's own goal while the run showed the override.
The launch now carries the run's resolved goal (the settings' inline
`run.goal`, layered as the create path layers it) as Petri's
`petri.launch_goal` compile variable, which Petri binds over the bundle's
`[run] goal` and the graph's own `goal`, so admission's frozen plan carries
the goal the run shows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Four doc comments the projection split cut in half, the scope ledger's
cache miss over an Option<Option>, the Pebble envelope passed by value,
and the Progress enum's large Pebble variant, now boxed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
RecoveryRequest::for_run and HooksSpec::for_run derived the author, the
identity source, the checkpoint settings and host_workspaces from the
run namespace with the same expressions. RunGitSettings, in checkpoint.rs
beside RunWorkspaces, is that derivation once; both specs carry it.
recovery::plan nested the "recorded, else found and reconciled" lookup
two loops deep and tracked a found flag over a tuple list. A Planner
holds the records, the workspace lookup and the recorded checkpoints,
and its methods read in order: targets, snapshot_of, reconcile_record,
newest. Candidate names the (key, sha) pair, and the Target struct that
duplicated its key's execution is gone; last_finish yields the
CheckpointKey itself, whose Display the failure reason now uses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
projector.rs mixed the commit rule with things that are not the pass:
the ordering rule and its tests, the in-process signalling store, the
stream table's rows and reads, and six readers documented "for a test".
projector/mod.rs now holds the pass alone; order.rs, signalling.rs and
stream.rs hold the rest; the test readers (rebuild, stored_projection,
stored_stream, stored_platform_records, event_json) live in test_support
behind the test-support feature, as the test-support boundary rule asks,
and the two test callers reach them there. The fault injection and the
cache's test-only reader are gated the same way, so neither ships.
pass() reads as three steps: at_head, read_new_events (a NewEvents
struct in place of a tuple) and stream::stream_rows, which rebuild
shares instead of repeating the fold loop. PassReport::skipped and
PassReport::contended replace the zero-filled literals.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
stage_key formatted "<execution>:<firing>" and five places re-derived or
re-parsed that string: the engine, progress and platform folds, the
projector's ordering rule, and fork.rs's stage_labels, which split it
back apart. FiringKey is the fact itself, with of_event for the event
side and From<StagePosition> for the platform-record side. It still
serializes as "<execution>:<firing>", so the fold_json a stored view
holds keeps its shape and needs no migration; a unit test pins that.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
fold_progress matched five payload kinds by string and walked each
payload's JSON by hand. Progress is now one internally tagged enum over
those kinds, with a struct per payload (PlannedRoute, OfferedTool,
ForkOccurrence) as the Attractor steps document them, and a test holds
each kind literal to the constant petri_attractor_steps exports, so a
rename there fails a test here instead of projecting nothing.
The fallback plan's original route carries its reasoning effort and
speed in the same lithos-llm types StageModelUsage holds, and the fold
now keeps them; it set both to None before. That is visible on the
stage's provider_used for an agent stage that ran under a fallback plan.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Four small duplications in the projection fold: a StageCompletion
literal built three times from an attempt's status, a StageModelUsage
literal built three times with no request controls, the pause and
unpause status arithmetic written four times across the coordinator and
lifecycle folds, and a bare Conclusion built beside the full one. Each is
now one function: completion() in the engine fold, StageModelUsage::new,
RunStatus::paused and RunStatus::unpaused beside blocked_reason (with
settle_control for the pending control they clear), and
Conclusion::outcome_only.
One case reads differently: a coordinator RunPaused that lands on a run
already paused behind a block now keeps that block for the unpause, as
the lifecycle Paused already did, instead of dropping it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
projection.rs held 1,834 lines: the fold state, the platform-record fold,
the lifecycle fold, the coordinator fold, the engine and view folds, the
progress and Pebble folds, the sandbox mapping, the tool mapping and the
model parsing, in one file. VIEWS.md is organised by source; the code now
is too. projection/mod.rs keeps RunView, FoldState and the helpers every
fold shares; platform.rs, coordinator.rs, engine.rs, progress.rs,
sandbox.rs and model.rs each hold one source's rows. A pure move: no
function body changed, only the visibility the cross-module calls need.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Eleven private hook methods returned Result<_, String> and rendered
their causes with the same collect_chain(...).join(": ") in fourteen
places. The error-handling strategy reserves String for rendered
projections. HookError keeps every failure with its source; it is
rendered once, by HookError::render, at the four Petri boundaries that
carry text: the adjusted outcome of a failed checkpoint, a transition's
problems, a scope acquisition's error, and the log. CheckpointKey gains
a Display so three messages stop spelling it out by hand. The rendered
text is the same as before, which the failed-checkpoint tests assert on.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
FabroHooks held twenty-one flat fields. Three of them were "a set read
from the store once, then kept current", in two shapes: a Mutex beside a
OnceCell<()> that had to be initialised first, and, for the restore plan,
a OnceCell<Mutex<_>> that could not be read uninitialised. The recorded
checkpoints and the collected artifacts now use the second shape too.
The checkpoint state (committed, recorded, last, branch), the artifact
state (globs, collected) and the scope state (environments, inherited
workspaces, per-workspace locks) are three structs with their own methods,
so each invariant lives in one type instead of in every caller.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
RunWorkspaces already ran every git command through a private Site
(a host path or a sandbox environment) but exposed each operation twice,
as commit/commit_in, matches/matches_in, has_commit/has_commit_in,
reset/reset_in, restore/restore_in and workspace_head/workspace_head_in.
The pairs propagated into every caller: recovery had bring_host_to and
bring_sandbox_to, the hooks had snapshot and snapshot_in_sandbox, and
restore_host and restore_sandbox. Site is now the public parameter, each
operation exists once, and the callers collapse to one function each.
The hooks resolve a scope's workspace and site in one place, site_of.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Seven crates' files each carried the same four-line lock function that
recovers a poisoned mutex. fabro_util::sync::lock is that function, once;
the copies in fabro-petri and fabro-server are gone. fabro-template's
helper panics on poison instead, a different policy, and is left as is.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Clippy's absolute-paths lint on the rendered prune error, the sync
directory reads the prune test documents, and a redundant rustdoc link.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Run deletion called the driver's `provider.delete(id)` under the run's
`petri.run` scope, a delete of Fabro's own over a sandbox whose lease
record Petri owns. It now goes the way `petri sandbox prune` goes:
`fabro_petri::prune` builds the run's Petri runtime over the server's
store (the run key, the run directory, the sandbox backend) and calls
Petri's prune, which opens the run for writing, checks each lease's
provider fingerprint, writes the delete intent and the tombstone beside
the run's other records, and lets each provider remove its managed
workspace, a host workspace included.
A run a live process holds answers 409 unless the delete is forced; a
lease Petri could not prune answers 409 with the problem text, or is
warned and skipped under force or a delete that already started. The
server drops the worker's handles before the prune, on the store
instance the prune opens, so the lease a stopped worker held is released
first. The projection reads only the coordinator and execution logs, so
the resource records change nothing it reports.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
