Commit graph

32 commits

Author SHA1 Message Date
Scott Werner
04309977e5 Commit checkpoint failures through required finalization
A failed checkpoint cancels the run, so Petri recorded it as cancelled
and the worker overrode its own outcome in memory. Runs that checkpoint
now declare required finalization, and finalize_run rejects with
checkpoint_failed before publishing. The projection and the engine
outcome report a cancelled finish carrying that failure as a workflow
failure with the checkpoint's message, and the in-memory override is
gone. A run whose checkpoint failed is never published.

Retry a host failure that storage rejects with a short backoff, so a
brief storage fault does not leave the run active and holding its
scheduler slot. A failure that never commits still leaves the run's
status alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:00:01 -04:00
Scott Werner
599b674ec1 Build required publication on Petri main's finalization
Track Petri main now that required run finalization has merged there,
read workflow execution from the engine state, resume the failed-publication
regression with its admitted workflow, and keep the worker lifecycle helper
usable for rejected appends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 12:45:25 -04:00
Scott Werner
3443a923c0 Carry required publication into fork declarations
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 12:45:25 -04:00
Scott Werner
59f9088b04 Preserve committed outcomes during host failure cleanup
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 12:45:24 -04:00
Scott Werner
21d4db6766 Commit publication failures through required run finalization 2026-10-08 12:45:24 -04:00
Scott Werner
f12065e6ae Adapt recovery to current resume callers and API errors 2026-10-06 12:29:34 -04:00
Scott Werner
cd5f56a4ed Enforce resolved network policy for Docker and Daytona runs 2026-10-01 17:12:03 -04:00
Scott Werner
6e0ef2c402 Keep Git checkpoints for runs on host workspaces
Checkpoints were only enabled for runs with a GitHub source, so local
folder runs, empty Local runs and dry runs stopped committing. `fabro
diff` then failed for them, and their checkpoint, run branch and diff
records disappeared from the event stream.

A run whose workspace is on the host now commits checkpoints there
again, without pushing, as on main. Docker and Daytona runs with no
GitHub source still record execution checkpoints without Git commits,
so a sandbox image without `git` cannot fail the run.

The scenario tests for crash recovery go back to asserting commits. A
workspace deleted while the run is down now fails the resumed run,
since the server keeps no copy to restore it from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 13:01:53 -04:00
Scott Werner
23591d550f Reuse cached GitHub installation tokens for run fetches and pushes
The worker minted one read token at launch and a fresh push token for
every checkpoint. The read token expired an hour into a run, so a
workspace acquired later fetched with a dead credential. Minting per
push put every push in GitHub's token-replication window, where a token
minted moments earlier is rejected with 404 "Repository not found".

The worker now keeps two InstallationTokenSource caches for the run, a
read-only one for fetches and a contents: write one for pushes. Each
fetch and push resolves through its source, which reuses one token until
it nears expiry and then mints the next. Petri's RunSource asks a
SourceCredentials provider on every fetch instead of holding a fixed
credential.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 13:01:13 -04:00
Scott Werner
aad954693e Keep checkpoints and run branch pushes in the sandbox 2026-10-01 13:01:13 -04:00
Scott Werner
886065957c Simplify GitHub checkout and run publication
- Share one credential header helper between the in-sandbox fetch and
  the run branch push
- Keep only the target branch, goal, and model on the publisher instead
  of a full run spec copy
- Load the worker's LLM catalog once, and mint the read token only when
  the run checks something out
- Pass the source explicitly to checkpoint fetch helpers, dropping
  unreachable branches, and reuse has_object in has_commit
- Move the run patch into the publication instead of cloning it, and
  build it only when a publisher exists
- Add test fixture helpers for file sources and recording publishers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 13:00:26 -04:00
Scott Werner
0250b40586 Publish a successful run from its run_finished hook
Pushing the run branch and opening the pull request now happen in the
run's worker, in Fabro's run_finished hook, after the last stage and
before the run's terminal record, as the legacy publish step did. A
failed push or pull request fails the run with publish_failed instead of
leaving a warning on a run that already succeeded.

fabro-petri gains a RunPublisher the hooks call for a successful run with
its run branch, final commit, snapshot repository and patch; the worker's
GitHub publisher pushes from the snapshot repository with a push token it
mints at that moment, opens the pull request its settings ask for, and
records it. The worker resolves the server's GitHub credentials itself
for both the read-only checkout token and the push token, so the server
no longer hands it a clone credential or publishes after the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 13:00:26 -04:00
Scott Werner
5c6195a352 Check GitHub targets out in the sandbox and publish their run branch
Since the Petri cutover, a run with a GitHub target started in an empty
workspace, nothing pushed its run branch to GitHub, and nothing asked for
the automatic pull request when it succeeded.

Fabro's hooks now check a fresh run's GitHub target out inside the
sandbox when Petri hands them the scope, before the first stage: the
workspace fetches the selected commit, tag or branch at the run's clone
depth, with a read-only token the server resolves at each worker launch.
The worker scrubs the token from its environment at startup and presents
it only to the fetch, so it never lands in the repository or its remote.
The files belong to the sandbox user, so git accepts them.

The same checkout seeds the workspace's snapshot repository with the
starting commit. Checkpoint bundles from a shallow clone then import, a
stage's own commits never make a bundle carry the source's history, a
restore into a fresh sandbox fetches the base again and applies the run's
commits, and a fork carries the base with its checkpoints.

When a successful GitHub-target run ends, the server pushes its final
commit from the snapshot repository to fabro/run/<id> with its own write
credentials, then, when the run changed files and asks for one, records
the pull request request for the existing creation supervisor. A failed
push or request is a warning notice on the run. The manual pull request
endpoint shares the request step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 13:00:26 -04:00
Scott Werner
c6464fb646 Restore Daytona container runs and configured resources 2026-09-28 23:02:34 -04:00
Scott Werner
d26b2324f0 Merge remote-tracking branch 'origin/main' into codex/restore-configured-artifact-storage
# Conflicts:
#	lib/apps/fabro-cli/tests/it/scenario/artifacts.rs
2026-09-28 16:27:30 -04:00
Scott Werner
7f55dd87b5 Harden artifact capture writes
- The store checks that captured bytes match their digest in every build,
  hashing on a blocking thread, and the upload handler relies on that
  check instead of hashing a second time.
- Capture bytes travel as `Bytes` from the hooks through the client and
  the store, so uploads and retries share one buffer.
- The artifact writer takes the run ID from the hooks, so objects are
  stored under the run their records name.
- Concurrent captures of the same file and content wait on one another,
  so the file is uploaded and recorded once.
- When a record append fails, the hooks re-read the run's captures and
  treat a record that did land as done, so a lost response does not
  record the capture twice.
- An upload that finishes after its run was deleted removes itself,
  instead of leaving an object nothing references.
- Listing a run's stage artifacts skips everything under `captures/`, so
  an unexpected object there cannot fail the listing or the ZIP.
- The capture record derives its `digest` key from its source instead of
  storing it twice, still writing and checking it on the wire.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 14:30:36 -04:00
Scott Werner
c7176bd572 Fix platform blockers in nightly releases 2026-09-28 11:37:15 -04:00
Scott Werner
2e500b0e2f Simplify artifact capture writer and storage helpers
- The artifact writer takes the digest the hooks already computed, so
  captured bytes are hashed once on each side, and the dead integrity
  error goes away.
- The writer is a required part of HooksSpec, not an optional field on
  RunRequest, so a run with capture globs always has a writer and the
  no-writer error goes away.
- ArtifactStore routes put/get and the capture methods through shared
  put_at/get_at helpers.
- The upload handler parses the digest with parse_blob_hash_path before
  any store reads, and builds its size-limit message from the constant.
- The default local artifact root comes from one helper used by both
  config resolution and the storage-dir override.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 13:32:13 -04:00
Scott Werner
8a09e8fa08 refactor: tidy the in-process sandbox provider wiring
Load the Daytona key for fork and prune through one AppState method
instead of two copied vault reads, and pass the sandbox configuration
into runtime_spec rather than building it and overwriting it. The
worker reuses the CLI's process_env_var lookup.

Share one Docker availability check and the backend-requirement
variable through fabro-test, drop the built-in plugin path and pin
constants nothing reads any more, and let enabled_plugins() exclude the
bundled kinds itself. Refresh the comments and the spawn_env test that
still described built-in plugins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 11:29:29 -04:00
Scott Werner
6ac6d5495f fix: run built-in sandbox providers in process
Register lazy Host, Docker, and Daytona factories for Petri execution,
fork, and prune. Share server provider configuration, preserve lease
fingerprints, and source Daytona credentials from the vault.

Remove built-in plugin setup and skip gates; add a release-mode worker
and prune regression to catch the failure that blocked nightly builds.

Co-Authored-By: Codex <noreply@openai.com>
2026-09-24 17:14:47 -04:00
Scott Werner
a1926c8632 Restore configured artifact storage for workflow captures 2026-09-24 13:53:46 -04:00
Bryan Helmkamp
12483e081e
Plan recovery through a Planner and share the run's Git settings with the hooks
RecoveryRequest::for_run and HooksSpec::for_run derived the author, the
identity source, the checkpoint settings and host_workspaces from the
run namespace with the same expressions. RunGitSettings, in checkpoint.rs
beside RunWorkspaces, is that derivation once; both specs carry it.

recovery::plan nested the "recorded, else found and reconciled" lookup
two loops deep and tracked a found flag over a tuple list. A Planner
holds the records, the workspace lookup and the recorded checkpoints,
and its methods read in order: targets, snapshot_of, reconcile_record,
newest. Candidate names the (key, sha) pair, and the Target struct that
duplicated its key's execution is gone; last_finish yields the
CheckpointKey itself, whose Display the failure reason now uses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-20 15:15:11 -04:00
Bryan Helmkamp
baa2fc8b5c
Keep every workspace under Petri's retention and say why
Petri's retention decides whether a released workspace is kept or removed.
Fabro's lifecycle settings decide whether a sandbox keeps running after
the run and whether a delete may remove it; none asks for removal at the
run's end, and the sandbox tab, `fabro cp`, the run's delete and the
sandbox scenarios read the container after the run. So the mapping is
`Retention::Always` for every setting, named once as `engine::RETENTION`
with the reasoning, instead of a per-setting function that released a
finished sandbox.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 18:39:58 -04:00
Bryan Helmkamp
67ba595b01
Record the run branch, identity, artifacts and diffs from the hooks
The commit that creates a workspace's run branch records `run.branch`
(the base commit, or the first checkpoint in a workspace with no history)
and `git.identity`. Every checkpoint record after the first carries the
stage's diff from its parent commit, with the patch as a text blob. After
the checkpoint record, the transition hook lists the stage's workspace
through the scope's environment, on the host and in a sandbox alike, and
collects every file under `[run.artifacts] include` into the blob table as
an `artifact.collected` record, skipping a file already collected under
the same path and digest. At the run's end the hooks diff the branch's
last checkpoint against its base in the snapshot repository and record
`run.diff`.

`engine::retention` maps the environment's lifecycle settings onto
Petri's workspace retention instead of always keeping every workspace:
`preserve`, `stop_on_terminal = false` and the local provider keep them,
anything else keeps a failed scope's only. The hooks docs no longer name a
redundant link target, so rustdoc passes with warnings denied.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 16:00:11 -04:00
Bryan Helmkamp
af38d68942
Delete fabro-validate and fabro-acp; validate on Petri's check
Petri judges a workflow at admission, so Fabro's lint rules go:
`fabro-validate` (its 35 rules and the `LintRule` trait) is deleted, and
with it `fabro-acp` (only a rule and two legacy executor tests used it),
the model-resolution transform, the legacy `create`, `compile_create_run`
and `materialize_create_run` stages, and `fabro-graphviz`'s `condition`
and `fidelity` modules. `Diagnostic`, `RelatedDiagnostic` and `Severity`
move to `fabro_types::diagnostic`, the one shape every diagnostic takes.

Validation is now the same question the create handler asks. A new
server module, `petri_check`, builds Petri's check request from a
workflow bundle and the run's settings (every workflow of the bundle at
its bundle-relative path, the inputs, the run variables, the launch),
runs the check, and maps the diagnostics; Fabro adds one rule of its
own, `fabro.model.no_ready_provider`, refusing a model node when no
provider is ready. Admission, the validate and preflight endpoints and
the offline `fabro validate` all go through it:

- `validate_prepared_manifest` runs Fabro's structural pass (parse and
  transform, whose diagnostics stay) and then Petri's check, on the
  blocking pool from the handlers;
- the offline `validate_manifest` checks with no model client and with
  an unbound input as a warning (`CheckRequest.unbound_is_warning`), so
  a workflow validates before its inputs exist; a collected workflow
  before upload checks with unbound inputs as errors, as before;
- preflight resolves each LLM node's selector against the ready
  providers and the catalog for its probe, as the deleted transform did,
  and no longer probes a model Petri refused;
- the graph render endpoint needs only the structural pass;
- a run manifest now carries its `[run.goal] file`, which Petri reads
  from the bundle as it does for a version.

The transforms keep the authored model selector (`sonnet` stays
`sonnet`): Petri pins the catalog model in the admitted graph, not in
the graph Fabro displays or in the settings snapshot. Tests assert that,
and the CLI's validate, preflight and graph snapshots carry Petri's
diagnostics (`attractor.no_start`, `attractor.undeclared_node`,
`attractor.bad_on_failure`, ...) in place of the lint rules' text.

Known gaps, Petri's side: a `workflow.toml` whose `[run.environment]`
names an environment the server catalog defines but the file does not
is refused (`unsupported.workflow_toml.run.environment`), as admission
already refused it; an unbound input inside an included template
partial is a render error rather than the unbound-input warning.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 11:32:04 -04:00
Bryan Helmkamp
a36bea15d2
Remove the engine flag: every run is a Petri run
Delete `Engine`, `RunEngine`, `[workflow] engine`, `[server.execution]
engine`, `FABRO_SERVER_ENGINE` and `fabro server start --engine`. The run
spec records what Petri admitted as `admission: PetriAdmission`; the
create handler always admits through `Runtime::check`; `execute_run`
always launches the Petri worker (or executes in process under the test
override); the CLI runner takes only the Petri worker path, and its
legacy control arm, artifact uploader, signal pause handlers and
credential helpers go with it. The CLI's `attach` and `events` read the
run stream only.

Two gaps this surfaced are closed here: the check adapter binds the
server's run variables as Petri compile variables (`{{ vars.* }}` in a
prompt no longer fails admission), and deleting a run removes its Petri
records, lease, platform records, projection and stream.

Tests: the config engine tests are replaced (an engine key is unknown),
the API round-trip test covers `PetriAdmission`, the server and CLI
Petri scenarios drop their engine settings, and the API tests that read
legacy event names now read the run stream or the session events. The
remaining red tests are fixtures and scenarios of the legacy executor
and the legacy event store (`fabro-store` `slate` and `run_state`,
`fabro-types` legacy `run.created` JSON, the server's handler-registry
scenarios, the CLI dry-run snapshots), which the next steps of the F4.3
series delete or port.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 09:58:21 -04:00
Bryan Helmkamp
b5fdc015f0
Merge branch 'petri-integration-sandbox' into petri-integration
# Conflicts:
#	lib/apps/fabro-cli/tests/it/scenario/mod.rs
2026-09-18 09:11:19 -04:00
Bryan Helmkamp
d60744b4d6
Checkpoint and recover Petri workspaces inside Docker and Daytona sandboxes
A Petri run on Docker or Daytona keeps its workspace inside the scope's
sandbox. Fabro's hooks now take the environment Petri hands them at
`scope_acquired`, run `git` inside the scope through it (the path Petri's
own sandbox-placed hooks take), and commit each stage on the run branch
with the same message and trailers as the host path. The commit leaves
the sandbox as a Git bundle, created against the newest ancestor the
snapshot repository already holds, split into 8 MiB parts (the plugin
transport reads one file up to 16 MiB), read out through the
environment's file transfer, fetched into the bare snapshot repository
on the host and named there under the checkpoint's ref. The repository
holds every checkpoint whatever the provider, and the platform records
name the same commits. The host path is unchanged; both sites share one
runner and the same commands.

Recovery is split: `recovery::plan` decides, over the records and the
snapshot repository alone, what every live workspace must sit on and
reconciles a lost record; `recover` applies it to host workspaces on the
server, as before, and reports a sandbox workspace's target as deferred.
The worker's hooks read the same plan at the scope's first acquisition
after a resume and bring the sandbox workspace to it before any attempt
runs there: verified or reset in a retained sandbox that still holds the
commit, else restored from a bundle of the checkpoint written into the
sandbox. Petri replaces a lease's lost sandbox on Fabro's request
(`LostSandbox::Replace`), so a removed container comes back fresh and
restored.

The checkpoint records are written for every provider now. A Docker
variant of the in-process hooks test moves a 20 MiB file through the
split transfer; the same test runs on Daytona when live credentials are
present. Three CLI scenarios run on a Docker environment: every stage's
checkpoint published from the container, a retained container whose
workspace drifted reset on restart, and a removed container replaced and
restored from the snapshot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 09:10:29 -04:00
Bryan Helmkamp
5093265efb
Wire pause, unpause and steer into the Petri worker
A Petri run answered only cancel and answers; pause, unpause and steer
were ignored with a warning. `fabro_petri::controls::RunControls` now
wraps Petri's `ControlService` per run: `engine::run` installs its pause
gate over the run's hooks, observes the run through it and wires it to
the coordinator, on a start and a resume alike, so a run paused when its
worker died resumes paused.

The worker's control channel takes a `WorkerControls` enum: the legacy
hub and pause flag, or the Petri run's controls. On Petri, `run.pause`
holds admission, `run.unpause` releases it once the record is durable,
and `run.steer` goes to the one live agent stage (Fabro's steer names no
stage); with none or several it is refused with a `run.notice` record.
The paused state is mirrored to Fabro's lifecycle as `run.paused` and
`run.unpaused` events, so the server's live status and the projection
follow Petri's own records.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 07:49:54 -04:00
Bryan Helmkamp
c9c4c27753
Merge branch 'petri-integration-hooks' into petri-integration
# Conflicts:
#	Cargo.lock
#	lib/apps/fabro-cli/src/commands/run/petri_worker.rs
#	lib/apps/fabro-cli/tests/it/scenario/petri.rs
#	lib/apps/fabro-server/src/server/petri_runs.rs
#	lib/components/fabro-petri/Cargo.toml
#	lib/components/fabro-petri/src/engine.rs
#	lib/components/fabro-petri/src/lib.rs
#	lib/components/fabro-store/src/platform_records.rs
2026-09-18 00:51:21 -04:00
Bryan Helmkamp
34139b1936
Prove the checkpoint hooks and the recovery protocol
In-process tests over the memory store: every finish is committed on the
run branch with its identity trailers and recorded with its commit, the
run-end hooks reach Petri's local service through Fabro's wrapper, a
stage that fails on its own terms is committed and its failure route runs
on the committed files, a failed checkpoint records `checkpoint_failed`
with no route taken and a restart reports the run failed, and a
`[[run.hooks]]` hook blocks an agent's tool call through the forwarded
service, with the model told why.

Real-binary scenarios crash the server and its worker with SIGKILL: after
a durable finish the stage's commit is not repeated and the interrupted
stage reruns on its snapshot; a crash held before the commit reruns the
stage once; a crash held after the commit but before its record
reconciles the record from the snapshot repository; a deleted workspace
is restored; a failure route sees the same committed files after a
crash; a failed checkpoint fails the run and a restart leaves it failed.

Recovery selects the executions `inspect_run` reports incomplete, and a
run whose coordinator log is still empty is left to the worker's resume.
The worker's platform record endpoints get an API test and the generated
TypeScript client.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 00:25:10 -04:00
Bryan Helmkamp
01beea0a6c
Checkpoint a Petri run's stages and recover its workspaces on restart
Fabro's hooks on a Petri run wrap the hooks the runtime installed for
`[[run.hooks]]` and forward every point. In `prepare_result`, before the
finish is recorded, they commit the stage's files on the run branch of
its host workspace with Fabro's author identity and the run, execution,
firing and attempt as trailers, and publish the commit to a snapshot
repository beside the run's workspaces under a ref per checkpoint. A
stage that failed on its own terms is committed like a successful one; a
commit that fails is fatal: the outcome becomes a `checkpoint_failed`
failure, the run is cancelled through the coordinator handle, and the
transition refuses the firing's routes. In `transition` they write the
platform checkpoint record, keyed on the Petri position and the
checkpoint's operation identity, and a failed write is a recorded
problem.

On restart the server runs the recovery protocol before it relaunches a
worker: a run with a failed checkpoint is reported failed; otherwise
every live execution's last durable finish names the snapshot its
workspace is verified against, reset to, or restored from, with a lost
record reconciled from the snapshot repository, and a finish with no
snapshot fails the run rather than resume it on stale files.

The worker reaches the platform records over two new worker-scoped
endpoints; the server reaches the table directly. A test gate directory
lets the CLI scenarios hold a checkpoint at a named point.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 00:12:14 -04:00