Share the managed-run cleanup between persist_run_failure and
fail_managed_run, return the committed projection from
commit_host_failure, and keep the run's status when the failure cannot
be stored. Restore the cancelled-launch message and keep the spawn
error's cause in the launch failure.
Rebuild Cargo.lock from main with only the Petri bump so the Windows
crates stay on their newer versions. Import the projection module
rather than the function, wrap long tracing calls, and document the
finalization test fixture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker leads a process group of its own, so it outlives a server
crash. The restarted server released the run's lease from outside and
launched a resume while that worker could still be running: a worker
whose lease is released keeps acting until its next write, beside its
successor. A delete likewise dropped the lease without knowing the
worker was gone.
Now each worker holds a lock on `worker.lock` in its run's scratch
directory for its whole life, taken before anything else. It is a POSIX
record lock: the kernel frees it only when the worker exits, and names
the process that holds it. Before the server ends a lease from outside
(the relaunch after a restart or a store interruption, and a delete), it
kills whatever process still holds the lock, with its process group, and
waits until the lock is free. A worker that finds the lock held does not
start.
- fabro-proc: ProcessLock::try_hold and stop_lock_holder, tested with
this test binary as the holding process.
- fabro-config: RunScratch::worker_lock_path.
- New scenario: a worker that outlives the server is gone before the
resume's worker launches, and the run succeeds once. It fails without
the server-side stop.
A host stage process runs in a process group of its own and still
outlives its killed worker, as it did at a worker crash.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petri now ends a run's lifetime at its first failed store write: it
records nothing after it, fails no firing for it, and returns
CoordinatorError::StoreFailed. The run is not over; the next lifetime
resumes it from what the store holds. Fabro read that error as an
unfinished run and failed it.
- engine: RunError::StoreFailed, returned without reading the record
back, and Conclusion::Interrupted for it.
- worker: an interrupted run gets no terminal lifecycle record; the
worker exits with EX_TEMPFAIL (75, the new ExitClass::Interrupted).
- server: WorkerExit carries the exit code. An interrupted worker's run
goes back to the scheduler in resume mode through the relaunch a
restart takes (lease release, recovery, start_requested + runnable),
now shared with reconcile_on_startup. The in-process path does the
same. A run is resumed at most MAX_STORE_INTERRUPTIONS (3) times per
server; the next interruption fails it. A pending cancel, a run that
ended or was deleted, and a shutdown also end it as before.
Tests: an engine run over a store whose first lease write fails is
interrupted with no finish, and a resume finishes it; the server
relaunches an interrupted worker in resume mode, fails the run after the
bound, and fails a worker that exits 1 as before; exit code 75.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Petri now refuses to resume a run whose creation a crash cut short (the
key is stored, the root invocation is not) with HostError::NotStarted,
and starts it again when the host runs it under the same key. The engine
used its own guard, check_resumable, which failed the run with
NothingToResume instead.
Execution::Resume now carries the admitted graphs, and a resume Petri
answers with NotStarted starts the run from them. The worker loads the
graphs in resume mode too, as does the server's in-process path. The
guard and RunError::NothingToResume are gone. New test:
a_resume_of_a_run_that_never_started_starts_it_again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rewrite the sandbox credential store only when the token source mints a
new generation, and skip the refresh loop for static tokens. Build the
run's read-only token source once in the worker and share it between the
workspace fetch and stage Git access. Drop unreachable branches in
for_run, reuse the shared contents-permission check and constants, and
fold the duplicated token resolution and test setup into helpers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The host provider manages no networking and refuses any policy but its
default, so sending the resolved AllowAll to local runs failed every
non-dry-run local execution. Apply the run's policy only on container
backends; dry runs, which always use the host backend, are covered by the
same check.
Also fold the Docker environment test helper into one that takes a typed
network mode, share the probe setup between the live Docker and Daytona
network tests, count canary hits per mode, and bind the run environment
once in the worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker minted one read token at launch and a fresh push token for
every checkpoint. The read token expired an hour into a run, so a
workspace acquired later fetched with a dead credential. Minting per
push put every push in GitHub's token-replication window, where a token
minted moments earlier is rejected with 404 "Repository not found".
The worker now keeps two InstallationTokenSource caches for the run, a
read-only one for fetches and a contents: write one for pushes. Each
fetch and push resolves through its source, which reuses one token until
it nears expiry and then mints the next. Petri's RunSource asks a
SourceCredentials provider on every fetch instead of holding a fixed
credential.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Share one credential header helper between the in-sandbox fetch and
the run branch push
- Keep only the target branch, goal, and model on the publisher instead
of a full run spec copy
- Load the worker's LLM catalog once, and mint the read token only when
the run checks something out
- Pass the source explicitly to checkpoint fetch helpers, dropping
unreachable branches, and reuse has_object in has_commit
- Move the run patch into the publication instead of cloning it, and
build it only when a publisher exists
- Add test fixture helpers for file sources and recording publishers
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pushing the run branch and opening the pull request now happen in the
run's worker, in Fabro's run_finished hook, after the last stage and
before the run's terminal record, as the legacy publish step did. A
failed push or pull request fails the run with publish_failed instead of
leaving a warning on a run that already succeeded.
fabro-petri gains a RunPublisher the hooks call for a successful run with
its run branch, final commit, snapshot repository and patch; the worker's
GitHub publisher pushes from the snapshot repository with a push token it
mints at that moment, opens the pull request its settings ask for, and
records it. The worker resolves the server's GitHub credentials itself
for both the read-only checkout token and the push token, so the server
no longer hands it a clone credential or publishes after the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since the Petri cutover, a run with a GitHub target started in an empty
workspace, nothing pushed its run branch to GitHub, and nothing asked for
the automatic pull request when it succeeded.
Fabro's hooks now check a fresh run's GitHub target out inside the
sandbox when Petri hands them the scope, before the first stage: the
workspace fetches the selected commit, tag or branch at the run's clone
depth, with a read-only token the server resolves at each worker launch.
The worker scrubs the token from its environment at startup and presents
it only to the fetch, so it never lands in the repository or its remote.
The files belong to the sandbox user, so git accepts them.
The same checkout seeds the workspace's snapshot repository with the
starting commit. Checkpoint bundles from a shallow clone then import, a
stage's own commits never make a bundle carry the source's history, a
restore into a fresh sandbox fetches the base again and applies the run's
commits, and a fork carries the base with its checkpoints.
When a successful GitHub-target run ends, the server pushes its final
commit from the snapshot repository to fabro/run/<id> with its own write
credentials, then, when the run changed files and asks for one, records
the pull request request for the existing creation supervisor. A failed
push or request is a warning notice on the run. The manual pull request
endpoint shares the request step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retry forked the source run at its last checkpoint and reran the failed
stage. It now creates a new run from the source's saved spec and starts
the workflow from the beginning in a fresh workspace, as retry did
before the Petri cutover. The new run records `retried_from` and no
`fork_source_ref`.
Retry no longer needs a checkpoint, a retained workspace, or a published
run branch, so it works for any terminal run that is not archived. To
continue from where a run stopped, fork it at a checkpoint.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The store checks that captured bytes match their digest in every build,
hashing on a blocking thread, and the upload handler relies on that
check instead of hashing a second time.
- Capture bytes travel as `Bytes` from the hooks through the client and
the store, so uploads and retries share one buffer.
- The artifact writer takes the run ID from the hooks, so objects are
stored under the run their records name.
- Concurrent captures of the same file and content wait on one another,
so the file is uploaded and recorded once.
- When a record append fails, the hooks re-read the run's captures and
treat a record that did land as done, so a lost response does not
record the capture twice.
- An upload that finishes after its run was deleted removes itself,
instead of leaving an object nothing references.
- Listing a run's stage artifacts skips everything under `captures/`, so
an unexpected object there cannot fail the listing or the ZIP.
- The capture record derives its `digest` key from its source instead of
storing it twice, still writing and checking it on the wire.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The artifact writer takes the digest the hooks already computed, so
captured bytes are hashed once on each side, and the dead integrity
error goes away.
- The writer is a required part of HooksSpec, not an optional field on
RunRequest, so a run with capture globs always has a writer and the
no-writer error goes away.
- ArtifactStore routes put/get and the capture methods through shared
put_at/get_at helpers.
- The upload handler parses the digest with parse_blob_hash_path before
any store reads, and builds its size-limit message from the constant.
- The default local artifact root comes from one helper used by both
config resolution and the storage-dir override.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Keep plugin-era Daytona lease fingerprints: read only DAYTONA_API_URL and
DAYTONA_ORGANIZATION_ID (no URL alias, no placement target), and stop
forwarding DAYTONA_SERVER_URL and DAYTONA_TARGET to the worker.
- Take the Docker fingerprint and network from this process's DOCKER_HOST,
the endpoint the Docker client actually connects to; make the provider
configuration's fields private.
- Return an error instead of panicking when Petri supplies no Host registry.
- Run deletion reads the Daytona key only for a Daytona run, and a forced
or restarted delete goes on when the secret store fails, as it does for
every other prune failure.
- Stop putting DAYTONA_API_KEY in the worker's environment; the worker reads
it from the vault. Give the worker's Daytona client the shared HTTP client.
- Fork, rewind and retry no longer read the vault: a fork acquires no sandbox.
- Remove the dead worker plugin forwarding and document that runs execute
only on the built-in providers.
- Build every Petri runtime through providers::standard_runtime or
bare_runtime, with a Clippy lint against Runtime::standard/bare.
- Share the Docker require-or-skip policy in fabro-test, tighten the Host
scope assertion.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Load the Daytona key for fork and prune through one AppState method
instead of two copied vault reads, and pass the sandbox configuration
into runtime_spec rather than building it and overwriting it. The
worker reuses the CLI's process_env_var lookup.
Share one Docker availability check and the backend-requirement
variable through fabro-test, drop the built-in plugin path and pin
constants nothing reads any more, and let enabled_plugins() exclude the
bundled kinds itself. Refresh the comments and the spawn_env test that
still described built-in plugins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Register lazy Host, Docker, and Daytona factories for Petri execution,
fork, and prune. Share server provider configuration, preserve lease
fingerprints, and source Daytona credentials from the vault.
Remove built-in plugin setup and skip gates; add a release-mode worker
and prune regression to catch the failure that blocked nightly builds.
Co-Authored-By: Codex <noreply@openai.com>
`lib/components/fabro-pebble-sandbox` moved into pebble as
`pebble_coding_agent::sandbox_driver` (lithoscomputer/pebble#27): the
`Environment` over a driver handle (`SandboxEnvironment`, was
`PebbleSandbox`), the `SandboxExec` policy, the port routes, and
`display_for_log`. The pebble pin moves to that branch head, 6d03b3b,
with the `sandbox-driver` feature on (`sandbox-driver-test-util` for the
server's tests, which take `MockSandbox` from pebble now). Nothing in the
crate was Fabro's by design; what was Fabro's stays: `SecretRedactor`
moves to `fabro-redact` as pebble's `Redactor` over `redact_string`, and
the log renderer takes it where a driver failure is rendered.
`fabro-petri` hands pebble types to Petri's crates, so Petri must pin the
same pebble revision: the petri pins move to lithoscomputer/petri#36
(9ee3f85), which pins pebble at the same head. Both re-pin to the pebble
merge commit together once #27 merges.
The 14 pebble commits between the pins fold the session projection's
lifetime tallies into `SessionProjection::totals` (and `PromptDelta`'s
into a flattened `totals`, which renames the prompt's `subagents` key to
`subagent_counts`, as the projection's already was). The stage progress
fold, the runs handler, the OpenAPI schema, the generated client model,
and the round-trip test follow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Petri creates and owns every run sandbox through the sandbox driver, so
what Fabro still needs around a driver handle is the Pebble glue: the
Environment pebble's coding agent runs its tools through, the exec policy
(stop grace, working directory, StripAll, the termination mapping, the
redacted output tail), pebble's port routes over the driver's preview
URLs, the secret redactor, the path helpers, and a log rendering that
appends a failed command's redacted tail. This crate holds that glue,
moved from fabro-sandbox, over `Arc<dyn Sandbox>` plus a working
directory instead of `RunSandbox`, with a `MockSandbox` double behind
`test-support`.
`fabro exec` creates its host sandbox directly on the driver's Host
provider and activates it; Ask Fabro wraps the attached handle. Both
keep the provider alive beside the sandbox where the session's processes
are the provider's process groups.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The rest of the change whose deletions the previous commit carries (its
`git add` stopped at an already-removed path): `fabro_types::RunGraph`
and the `fabro-petri` builder that reads it off the admitted graph, the
server's create, validate, preflight and render paths on Petri's check
alone, the consumers moved to the new shape, the OpenAPI `RunGraph`
schemas with their parity tests, the regenerated TS client, and the
docs naming Petri's diagnostic codes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
GET /runs/{id}/checkpoint duplicated what /state serves; the hidden
fabro parse command had no user; records, run_status, outcome, and
usage_rollup in fabro-workflow only re-exported fabro_types. The
importers now name fabro_types directly. format_cost keeps its two
callers (the pull request body and the CLI stage display) and moves to
fabro_types::usage; the usage rollup tests move beside the function in
fabro-types, with test_usage in its test support.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A worker whose server is killed outright retried its control-stream
connect forever at a 5 s capped backoff, so it never finalized, never
stopped its sandbox, and lived until reboot. The worker now tracks the
start of each run of continuous connection failure and gives up after
60 s (10 s once its parent is pid 1), through the existing fatal
control-loss path that interrupts interviews and cancels the run. After
that fatal fires, the runner keeps driving the cancelled pipeline for a
bounded grace so `conclude` can stop the sandbox before the process
exits, instead of dropping the pipeline future mid-flight.
The test harness's stale-daemon reaper only matched `fabro server`
titles bound under `/tmp/.tmp*`, but `TestContext` roots are
`.ft-<label>-*` under `std::env::temp_dir()`, so daemons bound there
were never reaped. The regex now also matches sockets below a `.ft-`
directory component under any parent, and the unit test covers both
roots plus real-looking paths and `.ft-` outside a directory component.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Petri's Fabro frontend refused a bundle naming an environment it did not
declare and every MCP catalog reference, so the fixtures declared
`[environments.local]` and the server's catalogs never reached Petri.
Pin Petri at c874b86, where the frontend reads `[environments.<id>]` and
`[run.environment]` from every settings layer (bundle over project over the
host's layer, key by key), takes the environment a launch selected over the
layers, and resolves `[run.agent.mcps.<name>] id = "..."` against a catalog
the host binds. The server hands Petri its environment catalog as
`[environments.<id>]` tables of the settings layer it already passes, the
intent's environment as the launch's selection (`Launch::environment`, as
the intent overrides the bundle in Fabro's own resolution), and its MCP
catalog as `RuntimeSpec::mcp_catalog_toml`, one inline entry per definition
keyed by id. Offline validation hands Petri the seeded catalog the same
way, so `fabro validate` accepts `[run.environment] id = "local"`.
The fixtures drop the `[environments.local]` tables they carried for this;
the secrets test keeps its own, on purpose. Scenario tests cover a bundle
naming a catalog environment (its image lowered, and run on Docker when the
plugin and a daemon are there), a bundle's own table winning key by key,
the server refusing an unknown environment before Petri, and a catalog MCP
reference whose tool the agent session lists (an echo server under
`test/mcp/`).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The worker control bus was publish-only: the steer and interrupt
endpoints answered 202 once the control was forwarded, and a refusal
showed up only later as a `run.notice` on the run's stream.
A steer or an interrupt now carries a request id. The worker answers it
over the control stream it arrived on with `{request_id, outcome}`,
where the outcome is `delivered` (with the stage's label) or `refused`
(with the code and the reason). The server keeps the outstanding
requests in a registry and waits up to 5 s for the answer: the endpoint
answers 202 `{"outcome":"delivered","stage":…}`, 409 with the refusal's
code (`no_live_turn`, `no_such_stage`, `steer_refused`,
`interrupt_refused`) and message, or 202 `{"outcome":"pending"}` when
the worker gave no answer in time. The `run.notice` record on refusal
stays, under the same code, so a steer to a stage that is not running is
now `no_such_stage` there too. Pause and unpause are unchanged.
The in-process test path answers a steer or an interrupt from the run's
own controls at once. `FABRO_TEST_CONTROL_ACKS_MUTED=1` on the server
mutes the worker's answers, so a test can see the pending fallback.
`fabro steer` prints the worker's answer, and a refusal is its error.
The OpenAPI spec documents the 202 body and the 409 codes; the Rust and
TypeScript clients are regenerated.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`fabro timeline <run>` prints the checkpoint table (or JSON); `fabro fork
<run> [target]` and `fabro rewind <run> <target>` make and start the new
run and name it, `--list` showing the timeline instead; `fabro retry
<run>` starts the retry and prints its id. The top-level help snapshot
and the generated CLI reference follow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The worker's `run.notice` for an interrupt it could not deliver now reads
"Interrupt of stage `gate` refused: the stage has no model turn to
interrupt" (or "Interrupt refused: …" when the control named no stage),
so the web and the CLI can show which stage and why, with the reason as
Petri's `ControlError` spells it. The gate scenario asserts both messages.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Petri 639ce3e added `ControlService::interrupt_firing` and the `LiveTurns`
capability a host installs beside the pause hooks. Fabro now drives it:
`RunControls::interrupt(stage, text)` resolves its stage the way a steer
does (a label, a node name, or the run's one live agent stage) and stops
that stage's current model turn, keeping the session; the text, when
given, is the stage's next input. `engine::run` installs the live-turn set
as a runtime capability, so without it no interrupt could ever land.
The worker maps `run.interrupt` and `run.interrupt_then_steer`, both of
which now carry an optional `stage`, to that call. The control bus is
one-way, so a refusal is recorded the way a refused steer is: a
`run.notice` on the run's stream whose code says why (`no_live_turn` when
Petri refuses a stage with no turn in flight, `no_such_stage`,
`interrupt_refused`).
The server's `POST /runs/{id}/interrupt` and `POST /runs/{id}/steer` with
`interrupt=true` forward the control and answer 202, replacing the 501
`interrupt_unsupported` stub. The interrupt endpoint takes an optional
body (`stage`, `text`), refuses a finished run with 409
`run_not_interruptible`, and forwards an interrupt of a blocked run, since
an agent stage may be running a turn beside the question and the worker
judges each stage itself. `fabro events --pretty` prints the delivered
interrupt and the stage's `attractor.turn.interrupted` report.
Verified on the twin: `fabro steer --interrupt` during a long tool call
ends the turn, the text is the agent's next request, and the stream
carries the `$interrupt` record and the interrupted-turn report; an
interrupt of a gate stage is refused with `no_live_turn` and the gate's
question is untouched.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Fabro halves of D1, D2 and D3. A stage's `agent_tools` is the union, by
name, of the `attractor.tools` payloads its native sessions record, with
`invoked` flipped by the envelope's `ToolCallStarted`; the payload carries
Petri's origin category, so Pebble's category is `subagent` for a sub-agent
tool and `other` for the rest. A pending question carries each option's
description and preview and the question's context, and its reference as
the review target when Fabro's validation admits it; the interview dock and
the Q&A renderer show the previews beside the descriptions, and the attach
prompt prints both under each choice. The web's command view reads the
script from the node's `meta.script`, the decision renderer the matched
condition from `meta.edges[edge].condition`, and `run events --pretty`
prints the condition on the transition line and one line per session
naming its tool count. The command view notes what the output capture did
not keep, from the final `step.finished` loss metrics.
The web fixtures are recaptured at the pin, so they carry the new facts.
VIEWS.md loses the two gap rows Petri filled and names the sources; the
README's list of what the fold leaves default shrinks to match.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`SteerRunRequest` takes an optional `stage`: the label the projection
shows (`node@visit`, or `node/e<execution>@visit` when two executions
share one) or the node's name. The server passes it on the worker control
message; the worker's `RunControls` resolves a label to the live agent
firing and steers that firing, and a node name through Petri's own
live-stage index. Unnamed, the one-live-agent rule stays, and the refusal
now names the live stages by their labels. `fabro steer --stage` sets it.
A controls scenario runs two agent stages side by side, sees the unnamed
steer refused with both named, and steers each apart, one over the API
and one through the flag.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The worker's signal handlers pause and unpause the run through its
`RunControls`, the same path the server's pause and unpause take, so a
signal holds admission, records Petri's `run.paused` and the lifecycle
mirror, and releases it on the unpause. The legacy pause state the plan
named no longer exists in the tree; nothing was left to delete. A controls
scenario sends both signals to a real worker and reads the records back.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
