A failed checkpoint cancels the run, so Petri recorded it as cancelled
and the worker overrode its own outcome in memory. Runs that checkpoint
now declare required finalization, and finalize_run rejects with
checkpoint_failed before publishing. The projection and the engine
outcome report a cancelled finish carrying that failure as a workflow
failure with the checkpoint's message, and the in-memory override is
gone. A run whose checkpoint failed is never published.
Retry a host failure that storage rejects with a short backoff, so a
brief storage fault does not leave the run active and holding its
scheduler slot. A failure that never commits still leaves the run's
status alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- stop_lock_holder returns the stopped pid as Option<u32> instead of a
LockHolder enum that only wrapped it
- stop_previous_worker uses with_context, and one run_scratch helper
replaces three spellings of the run's scratch path
- relaunch builds its runnable record through run_records::runnable,
moved out of the lifecycle handler so both callers share it
- WorkerExit derives success from its exit code instead of storing both
- the scenario tests share one worker-pid lookup and wait loop
- small readability fixes in the engine's resume arm and the resume test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker leads a process group of its own, so it outlives a server
crash. The restarted server released the run's lease from outside and
launched a resume while that worker could still be running: a worker
whose lease is released keeps acting until its next write, beside its
successor. A delete likewise dropped the lease without knowing the
worker was gone.
Now each worker holds a lock on `worker.lock` in its run's scratch
directory for its whole life, taken before anything else. It is a POSIX
record lock: the kernel frees it only when the worker exits, and names
the process that holds it. Before the server ends a lease from outside
(the relaunch after a restart or a store interruption, and a delete), it
kills whatever process still holds the lock, with its process group, and
waits until the lock is free. A worker that finds the lock held does not
start.
- fabro-proc: ProcessLock::try_hold and stop_lock_holder, tested with
this test binary as the holding process.
- fabro-config: RunScratch::worker_lock_path.
- New scenario: a worker that outlives the server is gone before the
resume's worker launches, and the run succeeds once. It fails without
the server-side stop.
A host stage process runs in a process group of its own and still
outlives its killed worker, as it did at a worker crash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pin Petri at e46845b, the merge of its executor layer change. The stage
credential layer is now a Petri `SpawnEnv` applied with
`EnvHandle::with_spawn_env`, so Petri forwards every other environment
method and applies the layer to one-shot containers as well as processes.
A container gets the managed GITHUB_TOKEN, but no credential store refresh
or Git helper configuration: the store lives in the scope's sandbox, which
the container does not share.
The managed GITHUB_TOKEN now replaces one set in the workflow environment,
an ACP agent's environment or the sandbox's own, as Fabro's stage
environment did before Petri. The token carries exactly the access the run
declares; a stage that needs other access changes its declaration.
The new pin also keeps a timeout as the failure a partial success came
from: `PartialSuccess.underlying` is now an `UnderlyingFailure`, so the
projection reports "the step timed out" for a partial success converted
from a timeout. The run format moves from 7 to 8, which the attach JSON
snapshot records; runs stored before this pin are refused, as with earlier
format changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retry now starts the workflow over, and a local-folder run commits its
checkpoints, so the retry scenario checks that every stage commits again
under the retry's run id. The scenario for retrying without Git
checkpoints goes: local runs have them now, and the retry scenario
covers starting over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Checkpoints were only enabled for runs with a GitHub source, so local
folder runs, empty Local runs and dry runs stopped committing. `fabro
diff` then failed for them, and their checkpoint, run branch and diff
records disappeared from the event stream.
A run whose workspace is on the host now commits checkpoints there
again, without pushing, as on main. Docker and Daytona runs with no
GitHub source still record execution checkpoints without Git commits,
so a sandbox image without `git` cannot fail the run.
The scenario tests for crash recovery go back to asserting commits. A
workspace deleted while the run is down now fails the resumed run,
since the server keeps no copy to restore it from.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retry forked the source run at its last checkpoint and reran the failed
stage. It now creates a new run from the source's saved spec and starts
the workflow from the beginning in a fresh workspace, as retry did
before the Petri cutover. The new run records `retried_from` and no
`fork_source_ref`.
Retry no longer needs a checkpoint, a retained workspace, or a published
run branch, so it works for any terminal run that is not archived. To
continue from where a run stopped, fork it at a checkpoint.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Servers have always moved a local artifact store under the storage
directory at startup, whatever `local.root` said. Browser-wizard installs
write `local.root = "<storage>/objects"`, so honoring that root would move
their store and hide every artifact already written, with nothing to
migrate it. Restore the storage-directory override for local roots and
leave honoring custom roots to a change that migrates existing objects.
Installer metadata still goes through the override, so it lands where the
server reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Keep plugin-era Daytona lease fingerprints: read only DAYTONA_API_URL and
DAYTONA_ORGANIZATION_ID (no URL alias, no placement target), and stop
forwarding DAYTONA_SERVER_URL and DAYTONA_TARGET to the worker.
- Take the Docker fingerprint and network from this process's DOCKER_HOST,
the endpoint the Docker client actually connects to; make the provider
configuration's fields private.
- Return an error instead of panicking when Petri supplies no Host registry.
- Run deletion reads the Daytona key only for a Daytona run, and a forced
or restarted delete goes on when the secret store fails, as it does for
every other prune failure.
- Stop putting DAYTONA_API_KEY in the worker's environment; the worker reads
it from the vault. Give the worker's Daytona client the shared HTTP client.
- Fork, rewind and retry no longer read the vault: a fork acquires no sandbox.
- Remove the dead worker plugin forwarding and document that runs execute
only on the built-in providers.
- Build every Petri runtime through providers::standard_runtime or
bare_runtime, with a Clippy lint against Runtime::standard/bare.
- Share the Docker require-or-skip policy in fabro-test, tighten the Host
scope assertion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Load the Daytona key for fork and prune through one AppState method
instead of two copied vault reads, and pass the sandbox configuration
into runtime_spec rather than building it and overwriting it. The
worker reuses the CLI's process_env_var lookup.
Share one Docker availability check and the backend-requirement
variable through fabro-test, drop the built-in plugin path and pin
constants nothing reads any more, and let enabled_plugins() exclude the
bundled kinds itself. Refresh the comments and the spawn_env test that
still described built-in plugins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Register lazy Host, Docker, and Daytona factories for Petri execution,
fork, and prune. Share server provider configuration, preserve lease
fingerprints, and source Daytona credentials from the vault.
Remove built-in plugin setup and skip gates; add a release-mode worker
and prune regression to catch the failure that blocked nightly builds.
Co-Authored-By: Codex <noreply@openai.com>
Reduce the workspace Cargo.toml convention block to three lines: Lithos
libraries track `main`, Cargo.lock picks the commits (move one with
`cargo update -p <crate>`), and unmerged library work is tried with an
uncommitted `[patch]`. Drop the instruction to hand-review lockfile diffs
and trim the restatements in the pebble and petri comments, the
fabro-petri README and module doc, AGENTS.md, and the docker test doc.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every lithoscomputer git dependency (sandbox-driver, pebble, petri,
lithos-llm, twins) now uses `branch = "main"` instead of an exact rev,
matching the libraries, so the workspace resolves one Cargo source per
repository. Cargo.lock is the single place the commits are chosen; move
one with `cargo update -p <crate>`.
The lockfile keeps every commit except sandbox-driver, which moves from
583a164 to b30203c: Daytona removed its paginated sandbox listing, and
b30203c lists through cursors instead (it also moves the driver's
daytona-sdk-rust dependency to 0e69058). The Daytona auth-probe test
mocks now serve the cursor endpoint the driver calls.
CI reads the sandbox-driver commit for the plugin install from the
lockfile through cargo metadata instead of from Cargo.toml.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker-backed run's in-memory managed run settled only when the
worker process exited. GET /runs/{id} reads the stored summary, which
the projector ends at Petri's own `run.finished` record, a moment
before the worker stores Fabro's terminal lifecycle record and exits.
The delete precheck prefers the managed run, so a delete issued in that
window was refused with 409 "cannot remove active run". Against a real
worker the window hit about six times in ten.
The server sees both records before they are stored: `run.finished` on
the coordinator log through the worker's records endpoint, and the
terminal lifecycle record through the platform-records endpoint. The
managed run now settles at either, ahead of the store, so the view
never reports the run ended while the managed run still says running.
The stream follower no longer reopens a settled run with the records
that precede its terminal one, and the worker's exit keeps the settled
status: it reaps the process, records a missing terminal record as it
did, and takes the store's status only when the store ended the run
differently. The mapping from Petri's finish to the run's status is the
projection's own, shared with its fold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Docker recovery scenarios scraped the worker log for "sandbox
workspace brought to its durable snapshot"; since the host and sandbox
restores share one path, the line reads "workspace brought to its
durable snapshot" with the site as a field.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The include error names the partial relative to the run's working
directory, so its `../` run is as long as that directory is deep: eight
on this machine's temp dir, three on the CI runner's. Collapse the run
to a token before the snapshot compares.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The detached cancel test waited for the run's status to read `failed`
and then asserted on the stored `run.lifecycle` record. The projection
concludes the run from Petri's `run.finished` coordinator record, and
the worker stores the platform's terminal lifecycle record a moment
later, so the read raced the write and the assertion failed about once
in thirty runs. Wait for the record itself, and print the stored events
and the run state when the assertion fails.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The host_plugin_ and docker_plugin_ variants ran each scenario under
Fabro's old plugin transport with the provider kinds `host` and
`docker-plugin`, which Petri's Fabro frontend rejects. Under Petri every
provider is already served by a sandbox-driver plugin, so those variants
test nothing distinct. A single docker_ variant replaces them: an
environment with provider `docker` on buildpack-deps:noble, created on
an isolated server, skipping without the sandbox-driver-docker
executable or a daemon with the image unless
FABRO_REQUIRE_SANDBOX_PLUGINS is set.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every Petri run takes its scope through a sandbox-driver plugin
executable that Petri finds on PATH, so the test jobs need
sandbox-driver-host and sandbox-driver-docker installed at the rev the
workspace pins. The three jobs share one from-source install through an
actions/cache entry keyed on the OS and the rev.
The Linux test job also pre-pulls Petri's default runner image, which
the suite's Docker scenarios leave to Petri: the plugin pulls it on
first use, but a 1 GiB pull inside a run's timeout is a flake.
The stdio plugin job was built for the deleted fabro-sandbox layer. It
becomes the Docker providers job: the `docker_` scenario variants and
the fabro-petri suite, with the fabro-sandbox and fabro-workflow steps
whose tests no longer exist removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A live-server scenario runs a one-stage workflow on Docker whose command
writes a file into the workspace, opens an Ask Fabro session on the
finished run, and sends one turn. The session attaches to the container
Petri created, stopped at the run's end, starts it again, and its tool
reads the file inside it; the turn succeeds, the tool's output and the
model's reply carry the file's content, and the twin's follow-up request
shows the model read it from the tool. Ask Fabro's tool policy is
read-only, so the shell tool is hidden from the model and refused; the
turn reads the file with the `read_file` tool, scripted on the twin,
instead of a shell `cat`.
The scenario skips, and says why, without the Docker plugin or a daemon,
as the other Docker scenarios do.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Nothing imports it any more: the Pebble glue lives in
fabro-pebble-sandbox, the server reaches run sandboxes through
sandbox_access, and Petri creates every run sandbox. The crate, its
test-support, its integration tests and every dependency edge go with
it. The `[server.sandbox.providers.<kind>.plugin]` settings stay: the
server still launches a plugin executable through them to attach to a
sandbox of a non-bundled kind.
AGENTS.md names the new crate and the direct-access pattern in place of
`RunSandbox`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>