A run whose sandbox cleanup fails is kept for a retry, and only an explicit
force deletes it without its sandboxes. Say so in the 409 so a caller with a
run stuck behind an unavailable provider knows how to proceed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ManagedRun::settle owns the rule that a run's first terminal status
sticks, and the lifecycle fold, the finish settle, fail_managed_run and
the in-process finish all go through it. One release_managed_run
replaces the two helpers that released a run's live state and also
frees its scheduler slot. The engine reads the finish through the
projection's finished_status, so a cancelled finish carrying a
checkpoint failure, and a failed publication, are mapped in one place.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed checkpoint cancels the run, so Petri recorded it as cancelled
and the worker overrode its own outcome in memory. Runs that checkpoint
now declare required finalization, and finalize_run rejects with
checkpoint_failed before publishing. The projection and the engine
outcome report a cancelled finish carrying that failure as a workflow
failure with the checkpoint's message, and the in-memory override is
gone. A run whose checkpoint failed is never published.
Retry a host failure that storage rejects with a short backoff, so a
brief storage fault does not leave the run active and holding its
scheduler slot. A failure that never commits still leaves the run's
status alone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Share the managed-run cleanup between persist_run_failure and
fail_managed_run, return the committed projection from
commit_host_failure, and keep the run's status when the failure cannot
be stored. Restore the cancelled-launch message and keep the spawn
error's cause in the launch failure.
Rebuild Cargo.lock from main with only the Petri bump so the Windows
crates stay on their newer versions. Import the projection module
rather than the function, wrap long tracing calls, and document the
finalization test fixture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Name the publish failure once, settle host failures through one flat
helper, share the append-then-finish path for in-process runs, reuse
the source state already read when checking a fork, and drop the
unused finished-run status shim.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Track Petri main now that required run finalization has merged there,
read workflow execution from the engine state, resume the failed-publication
regression with its admitted workflow, and keep the worker lifecycle helper
usable for rejected appends.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- stop_lock_holder returns the stopped pid as Option<u32> instead of a
LockHolder enum that only wrapped it
- stop_previous_worker uses with_context, and one run_scratch helper
replaces three spellings of the run's scratch path
- relaunch builds its runnable record through run_records::runnable,
moved out of the lifecycle handler so both callers share it
- WorkerExit derives success from its exit code instead of storing both
- the scenario tests share one worker-pid lookup and wait loop
- small readability fixes in the engine's resume arm and the resume test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker leads a process group of its own, so it outlives a server
crash. The restarted server released the run's lease from outside and
launched a resume while that worker could still be running: a worker
whose lease is released keeps acting until its next write, beside its
successor. A delete likewise dropped the lease without knowing the
worker was gone.
Now each worker holds a lock on `worker.lock` in its run's scratch
directory for its whole life, taken before anything else. It is a POSIX
record lock: the kernel frees it only when the worker exits, and names
the process that holds it. Before the server ends a lease from outside
(the relaunch after a restart or a store interruption, and a delete), it
kills whatever process still holds the lock, with its process group, and
waits until the lock is free. A worker that finds the lock held does not
start.
- fabro-proc: ProcessLock::try_hold and stop_lock_holder, tested with
this test binary as the holding process.
- fabro-config: RunScratch::worker_lock_path.
- New scenario: a worker that outlives the server is gone before the
resume's worker launches, and the run succeeds once. It fails without
the server-side stop.
A host stage process runs in a process group of its own and still
outlives its killed worker, as it did at a worker crash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petri now ends a run's lifetime at its first failed store write: it
records nothing after it, fails no firing for it, and returns
CoordinatorError::StoreFailed. The run is not over; the next lifetime
resumes it from what the store holds. Fabro read that error as an
unfinished run and failed it.
- engine: RunError::StoreFailed, returned without reading the record
back, and Conclusion::Interrupted for it.
- worker: an interrupted run gets no terminal lifecycle record; the
worker exits with EX_TEMPFAIL (75, the new ExitClass::Interrupted).
- server: WorkerExit carries the exit code. An interrupted worker's run
goes back to the scheduler in resume mode through the relaunch a
restart takes (lease release, recovery, start_requested + runnable),
now shared with reconcile_on_startup. The in-process path does the
same. A run is resumed at most MAX_STORE_INTERRUPTIONS (3) times per
server; the next interruption fails it. A pending cancel, a run that
ended or was deleted, and a shutdown also end it as before.
Tests: an engine run over a store whose first lease write fails is
interrupted with no finish, and a resume finishes it; the server
relaunches an interrupted worker in resume mode, fails the run after the
bound, and fails a worker that exits 1 as before; exit code 75.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petri now refuses to resume a run whose creation a crash cut short (the
key is stored, the root invocation is not) with HostError::NotStarted,
and starts it again when the host runs it under the same key. The engine
used its own guard, check_resumable, which failed the run with
NothingToResume instead.
Execution::Resume now carries the admitted graphs, and a resume Petri
answers with NotStarted starts the run from them. The worker loads the
graphs in resume mode too, as does the server's in-process path. The
guard and RunError::NothingToResume are gone. New test:
a_resume_of_a_run_that_never_started_starts_it_again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Creating a pull request for a finished run read the projection's final
patch field directly. Petri stores that patch in the blob table and the
field holds a blob reference, so the description model was handed the
reference string instead of the diff, and the empty-diff check could
never fire.
Move the reference resolution out of Files Changed into a shared
final_patch::load helper and use it for both readers. Pull request input
extraction now reads the real patch, judges emptiness by its text, and
reports a missing or unreadable blob instead of describing a placeholder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The create, validate and preflight paths each wrapped their launch with
the run's --model and --provider flags before building the check
request, so a new caller could build a check without them. The check
request now takes the flags as a required argument and binds them onto
the launch itself, and unit tests cover the flags, a provider-only flag,
and no flags.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petri now treats `petri.launch_model` and `petri.launch_provider` as the
model a run's flags ask for, above the file layers and the graph's
defaults. A host's last-resort default moved to `petri.default_model` and
`petri.default_provider`. Pin Petri at the merge of that change and bind
to it: the explicit `--model`/`--provider` flags go to the launch
variables, and the model the settings resolved (or the catalog default)
goes to the default variables.
`Launch` now names the two pairs `model`/`provider` and
`default_model`/`default_provider`, matching Petri, in place of the
separate override fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host provider manages no networking and refuses any policy but its
default, so sending the resolved AllowAll to local runs failed every
non-dry-run local execution. Apply the run's policy only on container
backends; dry runs, which always use the host backend, are covered by the
same check.
Also fold the Docker environment test helper into one that takes a typed
network mode, share the probe setup between the live Docker and Daytona
network tests, count canary hits per mode, and bind the run environment
once in the worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pushing the run branch and opening the pull request now happen in the
run's worker, in Fabro's run_finished hook, after the last stage and
before the run's terminal record, as the legacy publish step did. A
failed push or pull request fails the run with publish_failed instead of
leaving a warning on a run that already succeeded.
fabro-petri gains a RunPublisher the hooks call for a successful run with
its run branch, final commit, snapshot repository and patch; the worker's
GitHub publisher pushes from the snapshot repository with a push token it
mints at that moment, opens the pull request its settings ask for, and
records it. The worker resolves the server's GitHub credentials itself
for both the read-only checkout token and the push token, so the server
no longer hands it a clone credential or publishes after the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the Petri cutover, a run with a GitHub target started in an empty
workspace, nothing pushed its run branch to GitHub, and nothing asked for
the automatic pull request when it succeeded.
Fabro's hooks now check a fresh run's GitHub target out inside the
sandbox when Petri hands them the scope, before the first stage: the
workspace fetches the selected commit, tag or branch at the run's clone
depth, with a read-only token the server resolves at each worker launch.
The worker scrubs the token from its environment at startup and presents
it only to the fetch, so it never lands in the repository or its remote.
The files belong to the sandbox user, so git accepts them.
The same checkout seeds the workspace's snapshot repository with the
starting commit. Checkpoint bundles from a shallow clone then import, a
stage's own commits never make a bundle carry the source's history, a
restore into a fresh sandbox fetches the base again and applies the run's
commits, and a fork carries the base with its checkpoints.
When a successful GitHub-target run ends, the server pushes its final
commit from the snapshot repository to fabro/run/<id> with its own write
credentials, then, when the run changed files and asks for one, records
the pull request request for the existing creation supervisor. A failed
push or request is a warning notice on the run. The manual pull request
endpoint shares the request step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retry forked the source run at its last checkpoint and reran the failed
stage. It now creates a new run from the source's saved spec and starts
the workflow from the beginning in a fresh workspace, as retry did
before the Petri cutover. The new run records `retried_from` and no
`fork_source_ref`.
Retry no longer needs a checkpoint, a retained workspace, or a published
run branch, so it works for any terminal run that is not archived. To
continue from where a run stopped, fork it at a checkpoint.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The store checks that captured bytes match their digest in every build,
hashing on a blocking thread, and the upload handler relies on that
check instead of hashing a second time.
- Capture bytes travel as `Bytes` from the hooks through the client and
the store, so uploads and retries share one buffer.
- The artifact writer takes the run ID from the hooks, so objects are
stored under the run their records name.
- Concurrent captures of the same file and content wait on one another,
so the file is uploaded and recorded once.
- When a record append fails, the hooks re-read the run's captures and
treat a record that did land as done, so a lost response does not
record the capture twice.
- An upload that finishes after its run was deleted removes itself,
instead of leaving an object nothing references.
- Listing a run's stage artifacts skips everything under `captures/`, so
an unexpected object there cannot fail the listing or the ZIP.
- The capture record derives its `digest` key from its source instead of
storing it twice, still writing and checking it on the wire.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Servers have always moved a local artifact store under the storage
directory at startup, whatever `local.root` said. Browser-wizard installs
write `local.root = "<storage>/objects"`, so honoring that root would move
their store and hide every artifact already written, with nothing to
migrate it. Restore the storage-directory override for local roots and
leave honoring custom roots to a change that migrates existing objects.
Installer metadata still goes through the override, so it lands where the
server reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The artifact writer takes the digest the hooks already computed, so
captured bytes are hashed once on each side, and the dead integrity
error goes away.
- The writer is a required part of HooksSpec, not an optional field on
RunRequest, so a run with capture globs always has a writer and the
no-writer error goes away.
- ArtifactStore routes put/get and the capture methods through shared
put_at/get_at helpers.
- The upload handler parses the digest with parse_blob_hash_path before
any store reads, and builds its size-limit message from the constant.
- The default local artifact root comes from one helper used by both
config resolution and the storage-dir override.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Keep plugin-era Daytona lease fingerprints: read only DAYTONA_API_URL and
DAYTONA_ORGANIZATION_ID (no URL alias, no placement target), and stop
forwarding DAYTONA_SERVER_URL and DAYTONA_TARGET to the worker.
- Take the Docker fingerprint and network from this process's DOCKER_HOST,
the endpoint the Docker client actually connects to; make the provider
configuration's fields private.
- Return an error instead of panicking when Petri supplies no Host registry.
- Run deletion reads the Daytona key only for a Daytona run, and a forced
or restarted delete goes on when the secret store fails, as it does for
every other prune failure.
- Stop putting DAYTONA_API_KEY in the worker's environment; the worker reads
it from the vault. Give the worker's Daytona client the shared HTTP client.
- Fork, rewind and retry no longer read the vault: a fork acquires no sandbox.
- Remove the dead worker plugin forwarding and document that runs execute
only on the built-in providers.
- Build every Petri runtime through providers::standard_runtime or
bare_runtime, with a Clippy lint against Runtime::standard/bare.
- Share the Docker require-or-skip policy in fabro-test, tighten the Host
scope assertion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Load the Daytona key for fork and prune through one AppState method
instead of two copied vault reads, and pass the sandbox configuration
into runtime_spec rather than building it and overwriting it. The
worker reuses the CLI's process_env_var lookup.
Share one Docker availability check and the backend-requirement
variable through fabro-test, drop the built-in plugin path and pin
constants nothing reads any more, and let enabled_plugins() exclude the
bundled kinds itself. Refresh the comments and the spawn_env test that
still described built-in plugins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Register lazy Host, Docker, and Daytona factories for Petri execution,
fork, and prune. Share server provider configuration, preserve lease
fingerprints, and source Daytona credentials from the vault.
Remove built-in plugin setup and skip gates; add a release-mode worker
and prune regression to catch the failure that blocked nightly builds.
Co-Authored-By: Codex <noreply@openai.com>
Every lithoscomputer git dependency (sandbox-driver, pebble, petri,
lithos-llm, twins) now uses `branch = "main"` instead of an exact rev,
matching the libraries, so the workspace resolves one Cargo source per
repository. Cargo.lock is the single place the commits are chosen; move
one with `cargo update -p <crate>`.
The lockfile keeps every commit except sandbox-driver, which moves from
583a164 to b30203c: Daytona removed its paginated sandbox listing, and
b30203c lists through cursors instead (it also moves the driver's
daytona-sdk-rust dependency to 0e69058). The Daytona auth-probe test
mocks now serve the cursor endpoint the driver calls.
CI reads the sandbox-driver commit for the plugin install from the
lockfile through cargo metadata instead of from Cargo.toml.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
main (#891) upstreamed the Pebble sandbox adapter and deleted
fabro-pebble-sandbox, so Fabro now builds Pebble's sandbox-driver feature
beside Petri's crates and must hold one sandbox-driver copy. The three
pins move together: sandbox-driver to its main after #61 (host
attach-by-path, merged), Pebble to pebble#27's head after it moved its
own sandbox-driver pin to the same revision, and Petri to petri#36's
head, which carries both bumps.
Resolution: Cargo.toml keeps main's shape with the three revisions
rewritten; Cargo.lock regenerated from main's copy and holds one copy of
each library.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A run's host sandbox is a managed directory the worker recorded, with
Petri's labels, in the host registry inside the run's Petri directory.
The server reached it only by designating the directory again: a handle
with no record, so no labels and no ownership check, unlike Docker and
Daytona where `petri.run` is checked on every attach.
The server now observes the run's registry (`HostProvider::
observe_registry`, read and never written, so the worker stays its only
writer), resolves the directory to its record by path
(`attach_directory`), and runs the same `petri.run` ownership check as
on Docker (`OwnedProvider::check`). The handle refuses every lifecycle
change, so starting, stopping, and deleting stay the worker's, and a
stopped sandbox's retained workspace is usable through it; the access
paths therefore return a host handle as attached instead of activating
it. `ProviderAccess` carries the storage root the registry is found
under; a directory no registry records (a run older than this driver, a
caller without a storage root, a pruned Petri directory) is designated
again as before, without labels, for the read paths only.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`lib/components/fabro-pebble-sandbox` moved into pebble as
`pebble_coding_agent::sandbox_driver` (lithoscomputer/pebble#27): the
`Environment` over a driver handle (`SandboxEnvironment`, was
`PebbleSandbox`), the `SandboxExec` policy, the port routes, and
`display_for_log`. The pebble pin moves to that branch head, 6d03b3b,
with the `sandbox-driver` feature on (`sandbox-driver-test-util` for the
server's tests, which take `MockSandbox` from pebble now). Nothing in the
crate was Fabro's by design; what was Fabro's stays: `SecretRedactor`
moves to `fabro-redact` as pebble's `Redactor` over `redact_string`, and
the log renderer takes it where a driver failure is rendered.
`fabro-petri` hands pebble types to Petri's crates, so Petri must pin the
same pebble revision: the petri pins move to lithoscomputer/petri#36
(9ee3f85), which pins pebble at the same head. Both re-pin to the pebble
merge commit together once #27 merges.
The 14 pebble commits between the pins fold the session projection's
lifetime tallies into `SessionProjection::totals` (and `PromptDelta`'s
into a flattened `totals`, which renames the prompt's `subagents` key to
`subagent_counts`, as the projection's already was). The stage progress
fold, the runs handler, the OpenAPI schema, the generated client model,
and the round-trip test follow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The in-process Petri path settled the run's in-memory managed run once
Fabro's terminal lifecycle record was stored, after the engine returned.
GET /runs/{id} reads the stored summary, which the projector ends at
Petri's own `run.finished` record, a moment earlier, so a delete issued
the moment the run read as ended could reach the delete precheck, which
prefers the managed run, while it still said running, and was refused
with 409 "cannot remove active run". Against the real engine the window
hit eight times in thirty.
The worker path settles its run at the worker's records endpoint, ahead
of the store (#888). The in-process run now settles at the same record
through the run store it executes over: a store whose coordinator
appends settle the managed run at the `run.finished` record before the
record reaches the store and the projector's signal, with the same
finish mapping and settle the worker path uses. The settle is in memory
only. The terminal lifecycle record stored once the engine returns
refines the status and error and ends the run's live state as before,
and stays the settle of a run that ends without an engine finish.
The prune scenario no longer waits for the managed run to settle before
its delete: the wait guarded only this window.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worker-backed run's in-memory managed run settled only when the
worker process exited. GET /runs/{id} reads the stored summary, which
the projector ends at Petri's own `run.finished` record, a moment
before the worker stores Fabro's terminal lifecycle record and exits.
The delete precheck prefers the managed run, so a delete issued in that
window was refused with 409 "cannot remove active run". Against a real
worker the window hit about six times in ten.
The server sees both records before they are stored: `run.finished` on
the coordinator log through the worker's records endpoint, and the
terminal lifecycle record through the platform-records endpoint. The
managed run now settles at either, ahead of the store, so the view
never reports the run ended while the managed run still says running.
The stream follower no longer reopens a settled run with the records
that precede its terminal one, and the worker's exit keeps the settled
status: it reaps the process, records a missing terminal record as it
did, and takes the store's status only when the store ended the run
differently. The mapping from Petri's finish to the run's status is the
projection's own, shared with its fold.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An intent's goal override, and a `[run.goal]` layer, reached the run's
display graph and its settings but not Petri's check, so the agent stages
executed with the workflow's own goal while the run showed the override.
The launch now carries the run's resolved goal (the settings' inline
`run.goal`, layered as the create path layers it) as Petri's
`petri.launch_goal` compile variable, which Petri binds over the bundle's
`[run] goal` and the graph's own `goal`, so admission's frozen plan carries
the goal the run shows.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
