fabro/docs/public/execution/checkpoints.mdx

---
title: "Checkpoints"
description: "How Fabro uses Git to checkpoint and resume workflow runs"
---

Fabro checkpoints every workflow run using Git plus the durable run store. After each node completes, Fabro commits file changes to the run branch and records execution state in the durable event stream so that interrupted runs can be resumed exactly where they left off. This happens automatically — no configuration required beyond running inside a Git repository.

## Code and execution history

Fabro stores code and execution state separately:

| Storage | Contains |
|---|---|
| **Run branch** (`fabro/run/{run_id}`) | File changes made by agents and commands |
| **Durable run store** | Events and event-derived projections for checkpoints, stages, configuration, and conclusions |
| **Content-addressed store (CAS)** | Artifact and offloaded context payloads referenced by events and checkpoints |

The run branch is a regular Git branch. Checkpoint events record its commit SHAs, which link execution state to the corresponding code.

### Run branch commits

After each node finishes, Fabro stages file changes and creates a commit on the run branch. Files matching `[run.checkpoint] exclude_globs` patterns (configured in [run.toml](/execution/run-configuration#runcheckpoint) or [settings.toml](/administration/server-configuration#runcheckpoint-section)) are excluded from staging:

```
fabro(01JKXYZ...): plan (succeeded)

Fabro-Run: 01JKXYZ...
Fabro-Completed: 2
```

The commit message follows a structured format:

| Part | Description |
|---|---|
| Subject line | `fabro({run_id}): {node_id} ({status})` |
| `Fabro-Run` trailer | The run ID |
| `Fabro-Completed` trailer | Number of completed nodes so far |

The `git_commit_sha` in a `checkpoint.completed` event identifies the run branch commit. New commits do not include a `Fabro-Checkpoint` trailer.

Fabro disables Git commit and tag signing for checkpoint commits created inside a sandbox. Your personal or repository-level signing settings can stay enabled, but sandbox bookkeeping does not need access to your signing key.

### Durable execution state

Fabro derives run state from persisted events. The projection includes the run spec, start and status records, checkpoints, stage results, conclusion, and sandbox information. Artifact payloads live in CAS.

Use `fabro dump` to export a run as files, including `run.json`, `graph.fabro`, and per-stage artifacts.

### Metadata branch

Fabro no longer creates or pushes `fabro/meta/{run_id}` branches. Existing metadata branches remain untouched. Historical metadata snapshot events remain readable.

## What's in a checkpoint

The checkpoint projection captures the execution state needed to resume a run:

| Field | Description |
|---|---|
| `timestamp` | When the checkpoint was created |
| `current_node` | The node that just completed |
| `next_node_id` | The next node the engine would execute |
| `completed_nodes` | Ordered list of all completed node IDs |
| `node_retries` | How many retry attempts each node has used |
| `node_outcomes` | Full outcome (status, context updates, usage) for each completed node |
| `context_values` | Snapshot of the entire [run context](/execution/context) |
| `git_commit_sha` | SHA of the run branch commit at this checkpoint |
| `loop_failure_signatures` | Failure signature counts for loop detection |
| `restart_failure_signatures` | Failure signature counts across loop-restart edges |

The durable run store also keeps the current checkpoint so `resume`, `inspect`, and API reads do not need to rely on scratch files.

## Worktrees

Fabro uses Git worktrees to isolate workflow runs from your working directory. When a local run starts in a Git repository:

1. Fabro records the current HEAD as the **base SHA**
2. Creates a new branch `fabro/run/{run_id}` at that SHA
3. Adds a worktree at `{run_dir}/worktree` on that branch
4. Changes into the worktree directory for the duration of the run

This means your original working directory stays untouched while the agent makes changes in the worktree. When the run completes, Fabro removes the worktree and restores your original directory.

<Note>
If the working directory has uncommitted changes, the worktree starts from committed `HEAD` and those uncommitted changes are not included. Fabro logs a warning so you can commit, stash, or run explicitly in place when that is what you want.
</Note>

For Docker and Daytona sandboxes, the repository is cloned into the sandbox and checkpoint Git operations run there. The run branch is pushed to origin from the sandbox after each checkpoint when pushing is configured.

## Resuming a run

Resume an interrupted run from its durable checkpoint:

```bash
fabro resume 01JKXYZ
```

Fabro resolves the run by ID prefix, validates that durable state contains a checkpoint, and asks the server to continue execution. No workflow file or override flags are needed — all configuration is restored from persisted state.

<Accordion title="What happens during resume">
1. Fabro looks up the run directory by ID prefix
2. Validates that a checkpoint exists in durable state and no engine process is already running
3. Cleans stale local artifacts from the previous execution
4. Resets status to `Submitted` and spawns a new engine subprocess with `--resume`
5. The engine restores the full context, completed node list, retry counts, and failure signatures from durable state
6. If the checkpointed node used `full` fidelity, downgrades the first resumed node to `summary:high` (since the original conversation thread no longer exists in memory)
7. Continues execution from `next_node_id`
</Accordion>

### Resuming an in-flight stage

If the latest checkpoint captured a stage that was still running, resume allocates a new stage execution ID instead of rewriting the original history. For example, a resumed `work@1` stage continues as `work@2`, and the new record sets `resumed_from_stage_id = "work@1"`. The earlier stage remains immutable.

The number after `@` is the stage execution ordinal. It is separate from the graph visit count and from a handler's retry attempt, so loops, retries, and resume ancestry can be inspected independently.

## The checkpoint cycle

After a node completes, Fabro:

1. Stores offloaded context payloads in CAS.
2. Creates a code checkpoint commit when Git checkpointing is enabled.
3. Pushes the run branch when configured and collects the code diff.
4. Emits a checkpoint event with execution state and the code commit SHA. The run store persists this event and updates the projection.

A checkpoint commit failure stops execution. Intermediate push and diff failures emit warning notices. A required final publish failure marks the run as failed.

## Inspecting run history

Because checkpoints are plain Git commits, you can inspect them with standard Git tools:

```bash
# View the commit log for a run
git log fabro/run/01JKXYZ... --oneline

# See what an agent changed at a specific node
git show fabro/run/01JKXYZ...

# Diff the full run against the starting point
git diff main..fabro/run/01JKXYZ...

# Inspect execution state and events
fabro inspect 01JKXYZ
fabro events 01JKXYZ

# Export the run projection and artifacts
fabro dump 01JKXYZ --output ./run-dump
```

## Rewinding to an earlier checkpoint

If a later stage goes off-track, you can rewind a terminal run to an earlier checkpoint and resume from there instead of restarting the entire workflow. Rewind creates a replacement run at the target checkpoint, archives the source run, and prints the new run ID to resume:

```bash
# List the checkpoint timeline
fabro rewind <RUN_ID> --list

# Rewind to a specific checkpoint
fabro rewind <RUN_ID> plan@2

# Resume from the rewound point
fabro resume <NEW_RUN_ID>
```

The source run must already be terminal (`succeeded`, `failed`, or `dead`). If the source is archived, unarchive it first. If archive fails after the replacement run is created, do not retry `fabro rewind`; archive the source run manually.

See [`fabro rewind`](/reference/cli#fabro-rewind) for the full command reference.

## Forking a run

If you want to explore an alternate path from a checkpoint without archiving the original run, use `fabro fork` instead of `fabro rewind`. Fork creates a new independent run branching from the target checkpoint; the original run stays intact.

```bash
# List checkpoints
fabro fork <RUN_ID> --list

# Fork from a specific checkpoint
fabro fork <RUN_ID> plan@2

# Resume the forked run
fabro resume <NEW_RUN_ID>
```

Use **rewind** when a terminal run should be abandoned and replaced from an earlier point. Use **fork** when you want to try a different approach while keeping the original run as a reference.

`fabro rewind --list`, `fabro fork --list`, `fabro rewind`, and `fabro fork` are server-backed. Timeline listing reads checkpoints from the durable run store.

See [`fabro fork`](/reference/cli#fabro-fork) for the full command reference.

## When checkpointing is active

Git checkpointing activates automatically when:

- The run uses a Git repository and checkpointing has not been explicitly disabled
- Local runs can create a Git worktree under the run scratch directory
- Docker or Daytona can clone the configured GitHub origin into the sandbox

It is skipped when:

- The working directory is not a Git repository
- The run uses `--dry-run`
- The run is explicitly started in place with checkpointing disabled