mirror of
https://github.com/fabro-sh/fabro.git
synced 2026-09-17 23:52:34 +00:00
refactor(scratch): remove stale scratch file refs
Drop scratch-only compatibility paths and legacy test scaffolding now that SlateDB-backed state is authoritative. This removes scratch file fallbacks, updates docs and UI labels, and moves tests onto durable store-backed helpers.
This commit is contained in:
parent
326e0c27fa
commit
4bcd8f7647
49 changed files with 456 additions and 534 deletions
|
|
@ -95,7 +95,7 @@ export default function RunSettingsPage({ loaderData }: any) {
|
|||
|
||||
<div className="min-w-0 flex-1">
|
||||
<CollapsibleFile
|
||||
file={{ name: "run.json", contents: JSON.stringify(settings, null, 2), lang: "json" }}
|
||||
file={{ name: "settings.json", contents: JSON.stringify(settings, null, 2), lang: "json" }}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ export default function WorkflowDefinition() {
|
|||
return (
|
||||
<div className="flex flex-col gap-6">
|
||||
<CollapsibleFile
|
||||
file={{ name: "run.json", contents: JSON.stringify(workflow.settings, null, 2), lang: "json" }}
|
||||
file={{ name: "settings.json", contents: JSON.stringify(workflow.settings, null, 2), lang: "json" }}
|
||||
defaultOpen={false}
|
||||
/>
|
||||
{dotReady && (
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Fabro Events Strategy
|
||||
|
||||
Fabro emits structured **workflow run events** during execution for observability. Events are the durable audit trail for a run: they drive `progress.jsonl`, `live.json`, the run store, SSE streaming, CLI progress rendering, and retro analysis.
|
||||
Fabro emits structured **workflow run events** during execution for observability. Events are the durable audit trail for a run: they drive the run store, SSE streaming, CLI progress rendering, retro analysis, and optional JSONL sinks.
|
||||
|
||||
Events are distinct from tracing logs. Tracing is developer diagnostics; events are product-facing state transitions and activity records that other systems consume.
|
||||
|
||||
|
|
@ -10,12 +10,12 @@ Detached runs rely on this distinction. If something needs to be visible after r
|
|||
|
||||
```text
|
||||
Engine/Handler -> Event -> Emitter::emit()
|
||||
|- trace(raw event)
|
||||
|- canonicalize -> RunEvent
|
||||
`- on_event(&RunEvent)
|
||||
|- progress.jsonl + live.json
|
||||
|- trace(raw event)
|
||||
|- canonicalize -> RunEvent
|
||||
`- on_event(&RunEvent)
|
||||
|- run store
|
||||
|- SSE
|
||||
|- optional JSONL/debug sinks
|
||||
`- CLI / tests / metrics listeners
|
||||
```
|
||||
|
||||
|
|
@ -29,7 +29,7 @@ The canonical `RunEvent` is built exactly once in `fabro-workflow/src/event.rs`.
|
|||
|
||||
## Canonical Envelope
|
||||
|
||||
Each line in `progress.jsonl` is a serialized `RunEvent`:
|
||||
Each serialized `RunEvent` uses this canonical envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
@ -171,6 +171,6 @@ Do not rebuild or mutate the `RunEvent` in downstream listeners.
|
|||
|
||||
## Bypass And Persistence Guarantees
|
||||
|
||||
`progress.jsonl`, the run store, and SSE should reflect the same canonical envelope bytes after redaction.
|
||||
Any JSONL sink, the run store, and SSE should reflect the same canonical envelope bytes after redaction.
|
||||
|
||||
`status.json` remains the authoritative completion signal for detached runs. Terminal run status should only be written after all post-run work is finished.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Events
|
||||
|
||||
Every event in `progress.jsonl` is a JSON object with this envelope structure:
|
||||
Every serialized run event envelope, whether streamed over SSE, returned by `fabro logs`, or written to a JSONL sink, uses this structure:
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ Production runs at INFO level. INFO should be low-volume and high-signal — the
|
|||
|
||||
- Hot loops or per-token streaming events (use DEBUG only if truly needed for diagnosis)
|
||||
- Data that belongs in user-facing output (`eprintln!` for interactive CLI feedback, not tracing)
|
||||
- Detached user-visible warnings or errors that need to survive `attach`/`logs` (`detach.log` is debug-only; emit a `Event` into `progress.jsonl` instead)
|
||||
- Detached user-visible warnings or errors that need to survive `attach`/`logs` (`detach.log` is debug-only; emit an `Event` into the run event stream instead)
|
||||
- Redundant information already captured by a parent event (if you logged "starting X", you don't need to log every sub-step at the same level)
|
||||
- Events that are already traced via `EventEnum::trace()` — the event enums (`AgentEvent`, `PipelineEvent`, `ExecutionEnvEvent`) each have a `trace()` method called automatically at their emit site; do not add manual `info!`/`debug!` calls that duplicate what `trace()` already emits
|
||||
- Wrapper/forwarding variants that re-emit an inner event — `PipelineEvent::Agent`, `PipelineEvent::ExecutionEnv`, and `AgentEvent::SubAgentEvent` are no-ops in `trace()` because the inner event is already traced at its origin
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Run Scratch Files
|
||||
|
||||
This document maps the files written under a run scratch directory to their event sources.
|
||||
This document maps the files that still live under a run scratch directory. Durable run state lives in the run store and metadata branch; scratch is mostly local runtime state and caches.
|
||||
|
||||
Scope:
|
||||
- Scratch root: `~/.fabro/scratch/YYYYMMDD-{run_id}/`
|
||||
|
|
@ -12,19 +12,10 @@ There is no `_init.json` anymore. Run existence in the database is determined by
|
|||
|
||||
## Root-Level Files
|
||||
|
||||
| File | Purpose | Event source |
|
||||
| File | Purpose | Source |
|
||||
|---|---|---|
|
||||
| `run.json` | Run metadata snapshot: run id, resolved settings, graph, workflow slug, working directory, repo info, labels | Primarily `run.created`, plus request-time inputs persisted during create |
|
||||
| `start.json` | Start timestamp and git context | `run.started` |
|
||||
| `workflow.fabro` | Original dot source when available | `run.created.properties.workflow_source` |
|
||||
| `workflow.toml` | Original workflow TOML when available | `run.created.properties.workflow_config` |
|
||||
| `progress.jsonl` | Append-only event log | Every emitted run event |
|
||||
| `live.json` | Latest live state snapshot | Derived continuously from emitted events |
|
||||
| `checkpoint.json` | Crash-recovery snapshot | Derived from accumulated stage/checkpoint events plus engine state |
|
||||
| `conclusion.json` | Final outcome summary | `run.completed` and `run.failed` |
|
||||
| `retro.json` | Post-run retro output | `retro.completed` |
|
||||
| `workflow_bundle.json` | Bundled workflow input used by `start` to restore `workflow_path` and bundled child workflows/files | Written during create from the resolved workflow bundle |
|
||||
| `final.patch` | Final git diff for checkpointed runs | Local git state at finalize time, not a direct event payload |
|
||||
| `cli.log` | Per-run tracing log | Local tracing output, not event-derived |
|
||||
| `run.pid` | Legacy detached-run pid file from older runs | Legacy only; current flows do not rely on it |
|
||||
|
||||
## Local-Only Directories
|
||||
|
|
@ -35,45 +26,18 @@ These paths are local runtime state, not canonical event projections.
|
|||
|---|---|
|
||||
| `worktree/` | Git worktree used by checkpointed runs |
|
||||
| `runtime/blobs/` | Materialized local blob payloads for file-backed `fabro+blob://` references |
|
||||
| `cache/artifacts/values/` | Large context values spilled to the filesystem |
|
||||
| `runtime/worker.stderr.log` | Server-managed worker stderr capture |
|
||||
| `cache/artifacts/files/` | Captured artifact files organized by node and retry |
|
||||
| `nodes/{manager_node}_{visit}/child/` | Nested scratch root for manager-loop child workflows |
|
||||
|
||||
## Node Directories
|
||||
## Reconstructed / Exported Files
|
||||
|
||||
Per-node outputs are written under:
|
||||
- `nodes/{node_id}/`
|
||||
- `nodes/{node_id}-visit_{N}/` for retries, where `N` starts at `2`
|
||||
These names are still real, but they are no longer live scratch files by default:
|
||||
|
||||
### Agent and prompt nodes
|
||||
|
||||
| File | Purpose | Event source |
|
||||
|---|---|---|
|
||||
| `prompt.md` | Rendered prompt text | `stage.prompt.properties.text` is the closest event source |
|
||||
| `response.md` | Final model response text | Reconstructable from message events, but written as a local convenience file |
|
||||
| `status.json` | Final node status, notes, failure reason, timestamp | `stage.completed` |
|
||||
|
||||
### Command nodes
|
||||
|
||||
| File | Purpose | Event source |
|
||||
|---|---|---|
|
||||
| `script_invocation.json` | Command metadata: command, language, timeout | Partly from node config; not fully represented by a single event |
|
||||
| `stdout.log` | Captured stdout | Local process output |
|
||||
| `stderr.log` | Captured stderr | Local process output |
|
||||
| `script_timing.json` | Duration, exit code, timeout result | Partly `stage.completed.properties.duration_ms`, otherwise local process state |
|
||||
| `status.json` | Final node status | `stage.completed` |
|
||||
|
||||
### Nodes with git checkpointing
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `diff.patch` | Per-node git diff captured at checkpoint time |
|
||||
|
||||
### Manager / child workflow nodes
|
||||
|
||||
Manager nodes may create a nested `child/` directory containing a full run scratch structure for the child workflow.
|
||||
- Metadata branch files such as `run.json`, `start.json`, `checkpoint.json`, and `retro.json`
|
||||
- `fabro store dump` exports such as `run.json`, `start.json`, `status.json`, `checkpoint.json`, `conclusion.json`, `retro.json`, `events.jsonl`, and per-node prompt/response/status/stdout/stderr files
|
||||
- Retro-agent temp uploads named `progress.jsonl`, `checkpoint.json`, `run.json`, and `start.json` inside the retro sandbox
|
||||
|
||||
## Notes
|
||||
|
||||
- `progress.jsonl` is the event log; many other files are denormalized convenience snapshots derived from that stream plus local runtime state.
|
||||
- `checkpoint.json` and `live.json` are projections, not single-event payloads.
|
||||
- Artifact binaries are no longer stored in the SlateDB keyspace. They live in `ArtifactStore`; the run scratch tree only contains local cached copies when a workflow stage writes them to disk.
|
||||
|
|
|
|||
|
|
@ -119,7 +119,7 @@ The tracked paths are stored as `files_touched` on the stage outcome:
|
|||
|
||||
| Location | How it's used |
|
||||
|---|---|
|
||||
| `StageCompleted` event | Emitted with `files_touched` in the event stream and `progress.jsonl` |
|
||||
| `StageCompleted` event | Emitted with `files_touched` in the event stream and surfaced by `fabro logs` / exported event streams |
|
||||
| Preambles | Listed under each completed stage so downstream agents know what changed |
|
||||
| Retros | Included per-stage and aggregated across the full run |
|
||||
| `status.json` | Written to the stage's logs directory after each node completes |
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ title: "Checkpoints"
|
|||
description: "How Fabro uses Git to checkpoint and resume workflow runs"
|
||||
---
|
||||
|
||||
Fabro checkpoints every workflow run using Git. After each node completes, Fabro commits the file changes and execution state so that interrupted runs can be resumed exactly where they left off. This happens automatically — no configuration required beyond running inside a Git repository.
|
||||
Fabro checkpoints every workflow run using Git plus the durable run store. After each node completes, Fabro commits the file changes and execution state so that interrupted runs can be resumed exactly where they left off. This happens automatically — no configuration required beyond running inside a Git repository.
|
||||
|
||||
## Two branches, two purposes
|
||||
|
||||
|
|
@ -69,7 +69,7 @@ The `checkpoint.json` captures everything needed to resume a run:
|
|||
| `loop_failure_signatures` | Failure signature counts for loop detection |
|
||||
| `restart_failure_signatures` | Failure signature counts across loop-restart edges |
|
||||
|
||||
The checkpoint is also saved to `checkpoint.json` in the run directory for quick local access.
|
||||
The durable run store also keeps the current checkpoint so `resume`, `inspect`, and API reads do not need to rely on scratch files.
|
||||
|
||||
## Worktrees
|
||||
|
||||
|
|
@ -90,20 +90,20 @@ For Daytona sandboxes, the worktree is created inside the remote sandbox instead
|
|||
|
||||
## Resuming a run
|
||||
|
||||
Resume an interrupted run from its checkpoint on disk:
|
||||
Resume an interrupted run from its durable checkpoint:
|
||||
|
||||
```bash
|
||||
fabro resume 01JKXYZ
|
||||
```
|
||||
|
||||
Fabro looks up the run directory by ID prefix, loads `checkpoint.json` and `run.json` from the run directory, and spawns a new engine process to continue execution. No workflow file or override flags are needed — all configuration is read from the persisted run state.
|
||||
Fabro resolves the run by ID prefix, validates that durable state contains a checkpoint, and asks the server to continue execution. No workflow file or override flags are needed — all configuration is restored from persisted state.
|
||||
|
||||
<Accordion title="What happens during resume">
|
||||
1. Fabro looks up the run directory by ID prefix
|
||||
2. Validates that `checkpoint.json` exists and no engine process is already running
|
||||
3. Cleans stale artifacts from the previous execution (conclusion, PID file, etc.)
|
||||
2. Validates that a checkpoint exists in durable state and no engine process is already running
|
||||
3. Cleans stale local artifacts from the previous execution
|
||||
4. Resets status to `Submitted` and spawns a new engine subprocess with `--resume`
|
||||
5. The engine loads `run.json` and `checkpoint.json`, restores the full context, completed node list, retry counts, and failure signatures
|
||||
5. The engine restores the full context, completed node list, retry counts, and failure signatures from durable state
|
||||
6. If the checkpointed node used `full` fidelity, downgrades the first resumed node to `summary:high` (since the original conversation thread no longer exists in memory)
|
||||
7. Continues execution from `next_node_id`
|
||||
</Accordion>
|
||||
|
|
@ -112,12 +112,12 @@ Fabro looks up the run directory by ID prefix, loads `checkpoint.json` and `run.
|
|||
|
||||
Here's the full sequence that runs after every node completes:
|
||||
|
||||
1. **Save checkpoint to disk** — Write `checkpoint.json` to the run directory
|
||||
1. **Append checkpoint event** — Persist the new checkpoint into durable run state
|
||||
2. **Write metadata branch** — Serialize the checkpoint and any new artifacts to the metadata branch (shadow commit)
|
||||
3. **Commit to run branch** — Stage all file changes, commit with structured trailers linking to the shadow commit SHA
|
||||
4. **Update checkpoint** — Re-save `checkpoint.json` with the `git_commit_sha` field set
|
||||
4. **Update durable checkpoint** — Persist the `git_commit_sha` associated with the run-branch commit
|
||||
|
||||
Steps 2-4 are best-effort — if any Git operation fails, the run continues and emits a `RunNotice` warning event. The disk checkpoint from step 1 is always available as a fallback.
|
||||
Steps 2-4 are best-effort — if any Git operation fails, the run continues and emits a `RunNotice` warning event. Resume still uses the durable checkpoint in the run store.
|
||||
|
||||
## Inspecting run history
|
||||
|
||||
|
|
|
|||
|
|
@ -9,10 +9,11 @@ Fabro captures a structured event for every significant action during a workflow
|
|||
|
||||
Every workflow run emits a sequence of canonical **run event envelopes** that are:
|
||||
|
||||
- Written to `progress.jsonl` in the run directory
|
||||
- Stored durably in the run store
|
||||
- Broadcast over SSE to connected API clients
|
||||
- Stored for later analysis and retro generation
|
||||
- Rendered by CLI progress and log tooling
|
||||
- Optionally materialized into JSONL by export/debug paths
|
||||
|
||||
### Event names
|
||||
|
||||
|
|
@ -28,7 +29,7 @@ Event names use lowercase dot notation, for example:
|
|||
|
||||
### Envelope format
|
||||
|
||||
Each line in `progress.jsonl` is a JSON object with a stable envelope:
|
||||
Each serialized event envelope has a stable JSON shape:
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
@ -64,25 +65,23 @@ Envelope fields:
|
|||
|
||||
Only `id`, `ts`, `run_id`, and `event` are always present. Optional fields are omitted when they do not apply.
|
||||
|
||||
## Reading `progress.jsonl`
|
||||
## Reading the event stream
|
||||
|
||||
Because event payload lives in `properties`, most shell queries should look there.
|
||||
|
||||
```bash
|
||||
# Count tool calls in a run
|
||||
jq -r 'select(.event == "agent.tool.started") | .properties.tool_name' \
|
||||
~/.fabro/scratch/01JKXYZ.../progress.jsonl | wc -l
|
||||
fabro logs 01JKXYZ... | jq -r 'select(.event == "agent.tool.started") | .properties.tool_name' | wc -l
|
||||
|
||||
# Find stage failures
|
||||
jq 'select(.event == "stage.failed")' \
|
||||
~/.fabro/scratch/01JKXYZ.../progress.jsonl
|
||||
fabro logs 01JKXYZ... | jq 'select(.event == "stage.failed")'
|
||||
|
||||
# See which edges were taken
|
||||
jq '{from: .properties.from_node, to: .properties.to_node, label: .properties.label}' \
|
||||
~/.fabro/scratch/01JKXYZ.../progress.jsonl | head
|
||||
<(fabro logs 01JKXYZ...) | head
|
||||
```
|
||||
|
||||
`live.json` is still a pretty-printed copy of the most recent event envelope.
|
||||
If you need files on disk for offline analysis, `fabro store dump` exports `events.jsonl` plus run-state projections.
|
||||
|
||||
## Event categories
|
||||
|
||||
|
|
@ -111,7 +110,7 @@ Lifecycle events such as `agent.sub.spawned` and `agent.sub.completed` are emitt
|
|||
|
||||
### API: Server-Sent Events
|
||||
|
||||
When running workflows through the API server, subscribe to the [run events endpoint](/api-reference/runs/stream-run-events). Each SSE payload is a serialized run event envelope in the same shape used by `progress.jsonl`.
|
||||
When running workflows through the API server, subscribe to the [run events endpoint](/api-reference/runs/stream-run-events). Each SSE payload is a serialized run event envelope in the same shape used by `fabro logs` and `events.jsonl` exports.
|
||||
|
||||
### Web UI
|
||||
|
||||
|
|
@ -127,16 +126,12 @@ The CLI renders live progress from the same envelope format. This is written to
|
|||
|
||||
## Post-run analysis
|
||||
|
||||
Run artifacts still include:
|
||||
Post-run analysis surfaces include:
|
||||
|
||||
| File | Description |
|
||||
| Surface | Description |
|
||||
|---|---|
|
||||
| `run.json` | Run metadata and graph |
|
||||
| `start.json` | Start record |
|
||||
| `progress.jsonl` | Full event envelope stream |
|
||||
| `live.json` | Latest event envelope |
|
||||
| `checkpoint.json` | Final execution state |
|
||||
| `retro.json` | Retrospective, when enabled |
|
||||
| `conclusion.json` | Terminal summary |
|
||||
| `fabro logs <RUN>` | Full event envelope stream as NDJSON |
|
||||
| `fabro inspect <RUN>` | Current durable run state, including run/start/checkpoint/conclusion records |
|
||||
| `fabro store dump --output <DIR> <RUN>` | Exported `events.jsonl` plus reconstructed JSON and node files |
|
||||
|
||||
See [retros](/execution/retros), [stages](/api-reference/run-internals/list-run-stages), and [turns](/api-reference/run-internals/list-stage-turns) for higher-level analysis views built on top of this event stream.
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ The quantitative layer is extracted directly from the [checkpoint](/execution/ch
|
|||
|
||||
### Narrative layer
|
||||
|
||||
An LLM agent reads the run's `progress.jsonl` event stream and produces a structured analysis:
|
||||
An LLM agent reads the run's full event stream and produces a structured analysis:
|
||||
|
||||
| Field | Description |
|
||||
|---|---|
|
||||
|
|
@ -99,9 +99,9 @@ Open items capture follow-up work identified during the run:
|
|||
|
||||
Retro generation happens in two phases after a run completes:
|
||||
|
||||
1. **Derive** — Fabro extracts stage durations from `progress.jsonl` and builds a retro from the checkpoint data. This is deterministic, fast, and produces the quantitative layer. The retro is saved immediately as `retro.json` in the run's directory.
|
||||
1. **Derive** — Fabro extracts stage durations from durable run events and builds a retro from the checkpoint data. This is deterministic, fast, and produces the quantitative layer.
|
||||
|
||||
2. **Narrate** — An LLM agent session analyzes the run data. The agent has read access to `progress.jsonl`, `checkpoint.json`, `run.json`, and `start.json`. It uses grep and read tools to find interesting signals — failures, retries, errors, approach changes — then calls a `submit_retro` tool with its structured analysis. The narrative fields are merged into the existing retro and saved.
|
||||
2. **Narrate** — An LLM agent session analyzes the run data. The agent receives temp files named `progress.jsonl`, `checkpoint.json`, `run.json`, and `start.json` inside its sandbox so it can grep and read the event stream and run state. The narrative fields are merged back into durable retro state.
|
||||
|
||||
Both phases run automatically at the end of every CLI run. The API server derives the quantitative layer but does not currently run the narrative agent.
|
||||
|
||||
|
|
@ -113,13 +113,6 @@ Both phases run automatically at the end of every CLI run. The API server derive
|
|||
|
||||
### CLI
|
||||
|
||||
Retros are saved to `{run_dir}/retro.json` after every run. The path is printed at the end of the run output:
|
||||
|
||||
```
|
||||
Retro: smooth — Successfully implemented the feature
|
||||
Retro saved to ~/fabro-logs/01JKXYZ.../retro.json
|
||||
```
|
||||
|
||||
To enable retros for your project, set `retros = true` in the `[features]` section of your `fabro.toml`:
|
||||
|
||||
```toml title="fabro.toml"
|
||||
|
|
@ -148,4 +141,4 @@ Retros are also available via the REST API. See the [list retros](/api-reference
|
|||
|
||||
## Storage
|
||||
|
||||
Retros are stored as `retro.json` in the run's directory alongside `checkpoint.json` and `progress.jsonl`. They are plain JSON files — easy to parse, query, or pipe into other tools.
|
||||
Retros are stored in durable run state. If you need files on disk, `fabro store dump` materializes the retro as `retro.json` alongside other exported run data.
|
||||
|
|
|
|||
|
|
@ -198,7 +198,7 @@ The table shows run ID, status, workflow name, goal, and timing.
|
|||
| `--before <DATE>` | Only show runs started before this date (YYYY-MM-DD prefix match) |
|
||||
| `--workflow <NAME>` | Filter by workflow name (substring match) |
|
||||
| `--label <KEY=VALUE>` | Filter by label (repeatable, AND semantics) |
|
||||
| `--orphans` | Include orphan directories (no `run.json`) |
|
||||
| `--orphans` | Include orphan directories (no matching durable run) |
|
||||
| `--json` | Output as JSON |
|
||||
| `-q, --quiet` | Only display full run IDs, one per line (no headers or footers). Takes precedence over `--json`. |
|
||||
|
||||
|
|
@ -255,7 +255,7 @@ fabro system prune --orphans --yes
|
|||
| `--older-than <DURATION>` | Only prune runs older than this duration (e.g. `24h`, `7d`). Default when no explicit filters are set: `24h` |
|
||||
| `--workflow <NAME>` | Filter by workflow name (substring match) |
|
||||
| `--label <KEY=VALUE>` | Filter by label (repeatable, AND semantics) |
|
||||
| `--orphans` | Include orphan directories (no `run.json`) |
|
||||
| `--orphans` | Include orphan directories (no matching durable run) |
|
||||
| `--yes` | Actually delete (default is dry-run) |
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -4,13 +4,13 @@ description: "Structure of Fabro's per-run directory"
|
|||
---
|
||||
|
||||
<Warning>
|
||||
The run directory structure and file formats described here are internal implementation details and subject to change without notice. Do not build tooling that relies on them. The authoritative source of run state is the event-sourced run store — files in the scratch directory are disk projections for debugging convenience.
|
||||
The run directory structure and file formats described here are internal implementation details and subject to change without notice. Do not build tooling that relies on them. The authoritative source of run state is the event-sourced run store; the scratch directory is now mostly local runtime state and caches.
|
||||
</Warning>
|
||||
|
||||
Each `fabro run` invocation creates a timestamped directory under `~/.fabro/scratch/`:
|
||||
Each `fabro run` invocation creates a timestamped directory under `~/.fabro/storage/scratch/`:
|
||||
|
||||
```
|
||||
~/.fabro/scratch/20260307-01JQXYZ123ABC456DEF789/
|
||||
~/.fabro/storage/scratch/20260307-01JQXYZ123ABC456DEF789/
|
||||
```
|
||||
|
||||
The naming format is `YYYYMMDD-{run_id}`, where `run_id` is the ULID assigned to the run. You can override the base storage directory with the global `--storage-dir` flag (the scratch directory will be `<storage-dir>/scratch/`).
|
||||
|
|
@ -19,62 +19,28 @@ The naming format is `YYYYMMDD-{run_id}`, where `run_id` is the ULID assigned to
|
|||
|
||||
| File | Format | When written | Description |
|
||||
|---|---|---|---|
|
||||
| `run.json` | JSON | Run create | Run metadata — `run_id`, `created_at`, `config` (resolved configuration), `graph` (Graph), `workflow_slug`, `working_directory`, `host_repo_path`, `base_branch`, `labels` |
|
||||
| `start.json` | JSON | Run start | Start metadata — `run_id`, `start_time`, `run_branch`, `base_sha` |
|
||||
| `workflow.fabro` | Graphviz | Run create | Copy of the original workflow graph when the raw DOT source is available |
|
||||
| `run.pid` | Text | Legacy only | Legacy process ID file from older runs. Current detached launches use launcher records instead, and current attach/resume no longer read `run.pid`. |
|
||||
| `workflow.toml` | TOML | Run create | Copy of the original workflow file (only when the workflow is defined in TOML) |
|
||||
| `progress.jsonl` | JSONL | Continuous | Event stream — one JSON object per line for every significant event (stage starts, completions, tool calls, retries, etc.). See [Observability](/execution/observability) for the full event catalog. |
|
||||
| `live.json` | JSON | Continuous | Current execution state snapshot, overwritten on each event. Used for live monitoring. |
|
||||
| `checkpoint.json` | JSON | After each node | Crash recovery state — `current_node`, `completed_nodes`, `node_retries`, `context_values`, `node_outcomes`, `next_node_id`, `git_commit_sha`, failure signatures. See [Checkpoints](/execution/checkpoints). |
|
||||
| `conclusion.json` | JSON | Run end | Final result — `status`, `duration_ms`, `failure_reason`, `final_git_commit_sha`. Only present when the run completes (not for crashed or interrupted runs). |
|
||||
| `workflow_bundle.json` | JSON | Run create | Bundled workflow input used to restart the run without re-reading the original workflow files. Includes the root workflow path plus bundled child workflow sources and inline files. |
|
||||
| `final.patch` | Diff | Run end | Git diff from `base_sha` to final HEAD. Only present in git checkpoint mode. |
|
||||
| `retro.json` | JSON | Run end | Post-run retrospective analysis — `smoothness_rating`, `learnings`, `friction_points`, `stages`. Omitted if `--no-retro` is passed. See [Retros](/execution/retros). |
|
||||
| `cli.log` | Text | Continuous | Per-run tracing log. Contains the same tracing output as the daily log file, scoped to this run. |
|
||||
| `run.pid` | Text | Legacy only | Legacy process ID file from older runs. Current detached launches use launcher records instead, and current attach/resume no longer read `run.pid`. |
|
||||
|
||||
## `nodes/` subdirectory
|
||||
## Local-only directories
|
||||
|
||||
Each node execution writes artifacts into `nodes/{node_id}/`. When a node is retried, subsequent visits use `nodes/{node_id}-visit_{N}/` (where N starts at 2).
|
||||
These paths are local runtime state and caches, not the canonical run record.
|
||||
|
||||
Every node gets a `status.json` after completion containing `status`, `notes`, `failure_reason`, and `timestamp`. The remaining files depend on the handler type:
|
||||
- **`worktree/`** — When running in worktree mode, Fabro creates a Git worktree here as the working directory for agents and commands.
|
||||
- **`runtime/`** — Local runtime files. Today this is mainly materialized blob payloads under `runtime/blobs/` and worker stderr logs for server-managed subprocesses.
|
||||
- **`cache/artifacts/files/`** — Captured artifact files organized by node and retry, plus a `manifest.json` for each retry directory.
|
||||
- **`nodes/{manager_node}_{visit}/child/`** — Nested scratch directories for manager-loop child workflows.
|
||||
|
||||
**Agent and prompt nodes:**
|
||||
Large durable values, event streams, checkpoints, conclusions, and retros are no longer projected into live scratch by default. Use `fabro logs`, `fabro inspect`, the API, or `fabro store dump` for those surfaces.
|
||||
|
||||
| File | Description |
|
||||
|---|---|
|
||||
| `prompt.md` | The full prompt sent to the LLM |
|
||||
| `response.md` | The LLM's response text |
|
||||
| `status.json` | Execution status with routing outcome |
|
||||
## Reconstructed and export-only layouts
|
||||
|
||||
**Command nodes:**
|
||||
Some file names you may have seen in older runs or older docs still exist in reconstructed metadata branches or `fabro store dump` exports:
|
||||
|
||||
| File | Description |
|
||||
|---|---|
|
||||
| `script_invocation.json` | Command metadata — `command`, `language`, `timeout_ms` |
|
||||
| `stdout.log` | Standard output |
|
||||
| `stderr.log` | Standard error |
|
||||
| `script_timing.json` | Timing info — `duration_ms`, `exit_code`, `timed_out` |
|
||||
|
||||
**Nodes with git checkpointing:**
|
||||
|
||||
| File | Description |
|
||||
|---|---|
|
||||
| `diff.patch` | Git diff of changes made during this stage |
|
||||
|
||||
**Manager loop nodes:**
|
||||
|
||||
Manager nodes that run sub-workflows write a nested `child/` directory containing a full run structure (run.json, start.json, checkpoint, nodes, etc.).
|
||||
|
||||
## Other directories
|
||||
|
||||
**`worktree/`** — When running in git checkpoint mode, Fabro creates a Git worktree here as the working directory for agents and commands.
|
||||
|
||||
**`runtime/`** — Local-only runtime files such as materialized blob payloads under `runtime/blobs/`.
|
||||
|
||||
**`cache/`** — Local filesystem cache for file-backed artifacts and captured test artifacts:
|
||||
|
||||
- `cache/artifacts/values/` — large context values offloaded to file-backed artifacts
|
||||
- `cache/artifacts/files/` — captured test artifacts organized by node and retry
|
||||
- `run.json`, `start.json`, and `checkpoint.json` on metadata branches for rewind and fork
|
||||
- `run.json`, `start.json`, `checkpoint.json`, `conclusion.json`, `retro.json`, and `events.jsonl` in `fabro store dump` output
|
||||
- Per-node prompt, response, status, stdout, and stderr files in `fabro store dump` output and metadata rebuilds
|
||||
|
||||
## Browsing runs
|
||||
|
||||
|
|
@ -89,28 +55,16 @@ fabro ps --filter workflow=my-workflow
|
|||
## Full directory tree
|
||||
|
||||
```
|
||||
~/.fabro/scratch/
|
||||
~/.fabro/storage/scratch/
|
||||
├── 20260307-01JQXYZ123ABC456DEF789/ # One directory per run
|
||||
│ ├── run.json
|
||||
│ ├── start.json
|
||||
│ ├── workflow.fabro
|
||||
│ ├── workflow_bundle.json
|
||||
│ ├── run.pid # Legacy only; older runs may contain this
|
||||
│ ├── workflow.toml
|
||||
│ ├── progress.jsonl
|
||||
│ ├── live.json
|
||||
│ ├── checkpoint.json
|
||||
│ ├── conclusion.json
|
||||
│ ├── final.patch
|
||||
│ ├── retro.json
|
||||
│ ├── cli.log
|
||||
│ ├── runtime/
|
||||
│ │ └── blobs/
|
||||
│ │ └── 01JT5Y3KJ0N5S9E1Y7YFBR2G4D.json
|
||||
│ ├── cache/
|
||||
│ │ └── artifacts/
|
||||
│ │ ├── values/
|
||||
│ │ │ ├── response.plan.json
|
||||
│ │ │ └── command.output.json
|
||||
│ │ └── files/
|
||||
│ │ └── test/
|
||||
│ │ └── retry_1/
|
||||
|
|
@ -118,32 +72,11 @@ fabro ps --filter workflow=my-workflow
|
|||
│ │ │ └── screenshot.png
|
||||
│ │ └── manifest.json
|
||||
│ ├── nodes/
|
||||
│ │ ├── plan/
|
||||
│ │ │ ├── prompt.md
|
||||
│ │ │ ├── response.md
|
||||
│ │ │ └── status.json
|
||||
│ │ ├── work/
|
||||
│ │ │ ├── prompt.md
|
||||
│ │ │ ├── response.md
|
||||
│ │ │ ├── status.json
|
||||
│ │ │ └── diff.patch
|
||||
│ │ ├── work-visit_2/ # Retry of "work" node
|
||||
│ │ │ ├── prompt.md
|
||||
│ │ │ ├── response.md
|
||||
│ │ │ ├── status.json
|
||||
│ │ │ └── diff.patch
|
||||
│ │ ├── test/
|
||||
│ │ │ ├── script_invocation.json
|
||||
│ │ │ ├── stdout.log
|
||||
│ │ │ ├── stderr.log
|
||||
│ │ │ ├── script_timing.json
|
||||
│ │ │ └── status.json
|
||||
│ │ └── manager/
|
||||
│ │ ├── status.json
|
||||
│ │ └── child/
|
||||
│ │ ├── run.json
|
||||
│ │ ├── start.json
|
||||
│ │ ├── checkpoint.json
|
||||
│ │ └── nodes/
|
||||
│ │ ├── workflow_bundle.json
|
||||
│ │ ├── runtime/
|
||||
│ │ ├── cache/
|
||||
│ │ └── worktree/
|
||||
│ └── worktree/ # Git worktree (git checkpoint mode)
|
||||
```
|
||||
|
|
|
|||
|
|
@ -219,7 +219,7 @@ pub(crate) struct RunFilterArgs {
|
|||
#[arg(long = "label", value_name = "KEY=VALUE")]
|
||||
pub(crate) label: Vec<String>,
|
||||
|
||||
/// Include orphan directories (no run.json)
|
||||
/// Include orphan directories (no matching durable run)
|
||||
#[arg(long)]
|
||||
pub(crate) orphans: bool,
|
||||
}
|
||||
|
|
|
|||
|
|
@ -365,9 +365,10 @@ fn infer_storage_dir(run_dir: &Path) -> Option<PathBuf> {
|
|||
|
||||
#[cfg(test)]
|
||||
fn infer_run_id(run_dir: &Path) -> Option<RunId> {
|
||||
std::fs::read_to_string(run_dir.join("id.txt"))
|
||||
.ok()
|
||||
.map(|run_id| run_id.trim().to_string())
|
||||
run_dir
|
||||
.file_name()
|
||||
.map(|name| name.to_string_lossy().to_string())
|
||||
.and_then(|name| name.rsplit('-').next().map(ToOwned::to_owned))
|
||||
.filter(|run_id| !run_id.is_empty())
|
||||
.and_then(|run_id| run_id.parse().ok())
|
||||
}
|
||||
|
|
@ -461,18 +462,16 @@ mod tests {
|
|||
}
|
||||
|
||||
#[test]
|
||||
fn infer_run_id_reads_id_txt() {
|
||||
fn infer_run_id_reads_run_dir_suffix() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let storage_dir = dir.path().join("storage");
|
||||
let run_dir = storage_dir.join("scratch").join("20260401-test");
|
||||
let run_id = fabro_types::fixtures::RUN_1;
|
||||
let run_dir = storage_dir
|
||||
.join("scratch")
|
||||
.join(format!("20260401-{run_id}"));
|
||||
std::fs::create_dir_all(&run_dir).unwrap();
|
||||
std::fs::write(
|
||||
run_dir.join("id.txt"),
|
||||
format!("{}\n", fabro_types::fixtures::RUN_1),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(infer_run_id(&run_dir), Some(fabro_types::fixtures::RUN_1));
|
||||
assert_eq!(infer_run_id(&run_dir), Some(run_id));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
|
|
@ -363,10 +363,6 @@ fn attach_replays_from_store_without_run_json_or_progress_jsonl() {
|
|||
.assert()
|
||||
.success();
|
||||
|
||||
let run = resolve_run(&context, &run_id);
|
||||
let _ = std::fs::remove_file(run.run_dir.join("run.json"));
|
||||
let _ = std::fs::remove_file(run.run_dir.join("progress.jsonl"));
|
||||
|
||||
let mut cmd = context.command();
|
||||
cmd.args(["attach", &run_id]);
|
||||
cmd.timeout(SHARED_DAEMON_TIMEOUT);
|
||||
|
|
|
|||
|
|
@ -137,7 +137,6 @@ fn diff_node_outputs_specific_patch() {
|
|||
fn diff_node_reads_store_patch_without_disk_file() {
|
||||
let context = test_context!();
|
||||
let setup = setup_git_backed_changed_run(&context);
|
||||
let _ = std::fs::remove_file(setup.run.run_dir.join("nodes/step_one/diff.patch"));
|
||||
|
||||
let mut cmd = context.command();
|
||||
cmd.args(["diff", &setup.run.run_id, "--node", "step_one"]);
|
||||
|
|
|
|||
|
|
@ -138,15 +138,6 @@ fn inspect_json_omits_run_dir() {
|
|||
fn inspect_completed_run_reads_store_without_disk_metadata_files() {
|
||||
let context = test_context!();
|
||||
let run = setup_completed_fast_dry_run(&context);
|
||||
for name in [
|
||||
"run.json",
|
||||
"start.json",
|
||||
"conclusion.json",
|
||||
"checkpoint.json",
|
||||
"sandbox.json",
|
||||
] {
|
||||
let _ = std::fs::remove_file(run.run_dir.join(name));
|
||||
}
|
||||
let output = run_success(&context, &["inspect", &run.run_id]);
|
||||
|
||||
assert_snapshot!(serde_json::to_string_pretty(&compact_inspect(&output)).unwrap(), @r#"
|
||||
|
|
|
|||
|
|
@ -108,7 +108,6 @@ fn logs_completed_run_outputs_raw_ndjson() {
|
|||
fn logs_completed_run_reads_store_without_progress_jsonl() {
|
||||
let context = test_context!();
|
||||
let run = setup_completed_dry_run(&context);
|
||||
let _ = std::fs::remove_file(run.run_dir.join("progress.jsonl"));
|
||||
|
||||
let mut filters = context.filters();
|
||||
filters.push((
|
||||
|
|
|
|||
|
|
@ -67,9 +67,6 @@ fn pr_create_completed_dry_run_without_run_branch_errors() {
|
|||
fn pr_create_uses_store_run_record_without_run_json() {
|
||||
let context = test_context!();
|
||||
let run = setup_completed_fast_dry_run(&context);
|
||||
let _ = std::fs::remove_file(run.run_dir.join("run.json"));
|
||||
let _ = std::fs::remove_file(run.run_dir.join("start.json"));
|
||||
let _ = std::fs::remove_file(run.run_dir.join("conclusion.json"));
|
||||
|
||||
let mut cmd = context.command();
|
||||
cmd.args(["pr", "create", &run.run_id]);
|
||||
|
|
|
|||
|
|
@ -113,11 +113,6 @@ fn pr_view_reads_pull_request_from_store_without_pull_request_json() {
|
|||
.unwrap();
|
||||
});
|
||||
|
||||
let pr_path = run.run_dir.join("pull_request.json");
|
||||
if pr_path.exists() {
|
||||
std::fs::remove_file(pr_path).unwrap();
|
||||
}
|
||||
|
||||
let mut cmd = context.command();
|
||||
cmd.args(["pr", "view", &run.run_id]);
|
||||
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ fn help() {
|
|||
--no-upgrade-check Disable automatic upgrade check [env: FABRO_NO_UPGRADE_CHECK=true]
|
||||
--workflow <WORKFLOW> Filter by workflow name (substring match)
|
||||
--label <KEY=VALUE> Filter by label (KEY=VALUE, repeatable, AND semantics)
|
||||
--orphans Include orphan directories (no run.json)
|
||||
--orphans Include orphan directories (no matching durable run)
|
||||
--verbose Enable verbose output [env: FABRO_VERBOSE=]
|
||||
-a, --all Show all runs, not just running (like docker ps -a)
|
||||
-q, --quiet Only display run IDs
|
||||
|
|
|
|||
|
|
@ -73,7 +73,6 @@ fn resume_rewound_run_succeeds() {
|
|||
&setup.repo_dir,
|
||||
&["rev-parse", &format!("fabro/run/{}", setup.run.run_id)],
|
||||
);
|
||||
let _ = std::fs::remove_file(setup.run.run_dir.join("run.json"));
|
||||
|
||||
let mut resume_cmd = context.command();
|
||||
resume_cmd.current_dir(&setup.repo_dir);
|
||||
|
|
|
|||
|
|
@ -85,7 +85,6 @@ fn rewind_target_updates_metadata_and_resume_hint() {
|
|||
|
||||
let mut cmd = context.command();
|
||||
cmd.current_dir(&setup.repo_dir);
|
||||
let _ = std::fs::remove_file(setup.run.run_dir.join("run.json"));
|
||||
cmd.args(["rewind", &setup.run.run_id, "@1", "--no-push"]);
|
||||
|
||||
let (snapshot, output) = run_and_format(&mut cmd, &git_filters(&context));
|
||||
|
|
|
|||
|
|
@ -127,7 +127,6 @@ fn rm_force_deletes_submitted_run() {
|
|||
fn rm_force_deletes_run_without_sandbox_json_when_store_has_sandbox() {
|
||||
let context = test_context!();
|
||||
let setup = setup_local_sandbox_run(&context);
|
||||
let _ = std::fs::remove_file(setup.run.run_dir.join("sandbox.json"));
|
||||
|
||||
let mut filters = context.filters();
|
||||
filters.push((
|
||||
|
|
@ -146,7 +145,7 @@ fn rm_force_deletes_run_without_sandbox_json_when_store_has_sandbox() {
|
|||
");
|
||||
assert!(
|
||||
!setup.run.run_dir.exists(),
|
||||
"run directory should be deleted even without sandbox.json"
|
||||
"run directory should be deleted"
|
||||
);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1333,13 +1333,11 @@ fn detach_creates_run_dir_with_detach_log() {
|
|||
serde_json::json!({
|
||||
"run_dir": run_dir,
|
||||
"launcher_log_exists": context.storage_dir.join("launchers").join(format!("{run_id}.log")).exists(),
|
||||
"detach_log_exists": run_dir.join("detach.log").exists(),
|
||||
}),
|
||||
@r#"
|
||||
{
|
||||
"run_dir": "[RUN_DIR]",
|
||||
"launcher_log_exists": false,
|
||||
"detach_log_exists": false
|
||||
"launcher_log_exists": false
|
||||
}
|
||||
"#
|
||||
);
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ use fabro_types::{EventBody, RunEvent};
|
|||
|
||||
use super::support::{run_events, run_state, server_target};
|
||||
use crate::support::{fabro_json_snapshot, unique_run_id};
|
||||
use fabro_config::RunScratch;
|
||||
|
||||
const SHARED_DAEMON_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
|
||||
|
||||
|
|
@ -446,9 +445,6 @@ fn detached_run_answers_pending_question_without_interview_scratch_files() {
|
|||
);
|
||||
|
||||
let run_dir = context.find_run_dir(&run_id);
|
||||
let scratch = RunScratch::new(&run_dir);
|
||||
let legacy_interview_request = scratch.runtime_dir().join("interview_request.json");
|
||||
let legacy_interview_response = scratch.runtime_dir().join("interview_response.json");
|
||||
let runtime = tokio::runtime::Runtime::new().expect("test runtime should build");
|
||||
let question_id = runtime.block_on(async {
|
||||
let (client, base_url) = server_endpoint(&context.storage_dir);
|
||||
|
|
@ -459,14 +455,6 @@ fn detached_run_answers_pending_question_without_interview_scratch_files() {
|
|||
.to_string();
|
||||
|
||||
assert_eq!(question["stage"], "approve");
|
||||
assert!(
|
||||
!legacy_interview_request.exists(),
|
||||
"worker should not create interview_request.json"
|
||||
);
|
||||
assert!(
|
||||
!legacy_interview_response.exists(),
|
||||
"worker should not create interview_response.json"
|
||||
);
|
||||
|
||||
let response = client
|
||||
.post(format!(
|
||||
|
|
|
|||
|
|
@ -73,7 +73,6 @@ fn sandbox_cp_downloads_file_from_run() {
|
|||
fn sandbox_cp_downloads_file_from_store_without_sandbox_json() {
|
||||
let context = test_context!();
|
||||
let setup = setup_local_sandbox_run(&context);
|
||||
let _ = std::fs::remove_file(setup.run.run_dir.join("sandbox.json"));
|
||||
let dest = context.temp_dir.join("downloaded-from-store.txt");
|
||||
let mut cmd = context.cp();
|
||||
cmd.args([
|
||||
|
|
|
|||
|
|
@ -102,9 +102,6 @@ fn start_by_run_id_starts_created_run_without_run_json_or_status_json() {
|
|||
.assert()
|
||||
.success();
|
||||
|
||||
let run_dir = context.find_run_dir(&run_id);
|
||||
let _ = std::fs::remove_file(run_dir.join("run.json"));
|
||||
|
||||
context
|
||||
.command()
|
||||
.args(["start", &run_id])
|
||||
|
|
|
|||
|
|
@ -579,9 +579,6 @@ pub(crate) fn find_run_dir(storage_dir: &Path, run_id: &str) -> Option<PathBuf>
|
|||
}
|
||||
|
||||
fn infer_run_id(run_dir: &Path) -> String {
|
||||
if let Ok(id) = std::fs::read_to_string(run_dir.join("id.txt")) {
|
||||
return id.trim().to_string();
|
||||
}
|
||||
run_dir
|
||||
.file_name()
|
||||
.map(|name| name.to_string_lossy().to_string())
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ fn help() {
|
|||
--workflow <WORKFLOW> Filter by workflow name (substring match)
|
||||
--label <KEY=VALUE> Filter by label (KEY=VALUE, repeatable, AND semantics)
|
||||
--verbose Enable verbose output [env: FABRO_VERBOSE=]
|
||||
--orphans Include orphan directories (no run.json)
|
||||
--orphans Include orphan directories (no matching durable run)
|
||||
--older-than <DURATION> Only prune runs older than this duration (e.g. 24h, 7d). Default: 24h when no explicit filters are set
|
||||
--yes Actually delete (default is dry-run)
|
||||
-h, --help Print help
|
||||
|
|
|
|||
|
|
@ -57,7 +57,6 @@ fn wait_completed_run_prints_success_summary() {
|
|||
fn wait_completed_run_reads_store_without_status_or_conclusion_files() {
|
||||
let context = test_context!();
|
||||
let run = setup_completed_fast_dry_run(&context);
|
||||
let _ = std::fs::remove_file(run.run_dir.join("conclusion.json"));
|
||||
let mut filters = context.filters();
|
||||
filters.push((
|
||||
r"\b\d+(\.\d+)?(ms|s)\b".to_string(),
|
||||
|
|
|
|||
|
|
@ -134,16 +134,17 @@ fn dry_run_create_start_attach_works_with_default_run_lookup() {
|
|||
.success();
|
||||
|
||||
let run_dir = context.find_run_dir(&run_id);
|
||||
let state = run_state(&run_dir);
|
||||
fabro_json_snapshot!(
|
||||
context,
|
||||
serde_json::json!({
|
||||
"run_json_exists": run_dir.join("run.json").exists(),
|
||||
"conclusion_json_exists": run_dir.join("conclusion.json").exists(),
|
||||
"status": state.status.map(|status| status.status),
|
||||
"has_conclusion": state.conclusion.is_some(),
|
||||
}),
|
||||
@r#"
|
||||
{
|
||||
"run_json_exists": false,
|
||||
"conclusion_json_exists": false
|
||||
"status": "succeeded",
|
||||
"has_conclusion": true
|
||||
}
|
||||
"#
|
||||
);
|
||||
|
|
@ -176,16 +177,17 @@ fn dry_run_detach_attach_works_with_default_run_lookup() {
|
|||
.success();
|
||||
|
||||
let run_dir = context.find_run_dir(&run_id);
|
||||
let state = run_state(&run_dir);
|
||||
fabro_json_snapshot!(
|
||||
context,
|
||||
serde_json::json!({
|
||||
"run_dir": run_dir,
|
||||
"conclusion_json_exists": run_dir.join("conclusion.json").exists(),
|
||||
"has_conclusion": state.conclusion.is_some(),
|
||||
}),
|
||||
@r#"
|
||||
{
|
||||
"run_dir": "[RUN_DIR]",
|
||||
"conclusion_json_exists": false
|
||||
"has_conclusion": true
|
||||
}
|
||||
"#
|
||||
);
|
||||
|
|
@ -247,16 +249,17 @@ digraph BarBaz {
|
|||
.success();
|
||||
|
||||
let run_dir = context.find_run_dir(&run_id);
|
||||
let state = run_state(&run_dir);
|
||||
fabro_json_snapshot!(
|
||||
context,
|
||||
serde_json::json!({
|
||||
"run_dir_exists": run_dir.exists(),
|
||||
"conclusion_json_exists": run_dir.join("conclusion.json").exists(),
|
||||
"has_conclusion": state.conclusion.is_some(),
|
||||
}),
|
||||
@r#"
|
||||
{
|
||||
"run_dir_exists": true,
|
||||
"conclusion_json_exists": false
|
||||
"has_conclusion": true
|
||||
}
|
||||
"#
|
||||
);
|
||||
|
|
|
|||
|
|
@ -27,15 +27,10 @@ fn block_on<T>(future: impl std::future::Future<Output = T>) -> T {
|
|||
}
|
||||
|
||||
fn infer_run_id(run_dir: &Path) -> String {
|
||||
std::fs::read_to_string(run_dir.join("id.txt"))
|
||||
.ok()
|
||||
.map(|id| id.trim().to_string())
|
||||
.or_else(|| {
|
||||
run_dir
|
||||
.file_name()
|
||||
.map(|name| name.to_string_lossy().to_string())
|
||||
.and_then(|name| name.rsplit('-').next().map(ToOwned::to_owned))
|
||||
})
|
||||
run_dir
|
||||
.file_name()
|
||||
.map(|name| name.to_string_lossy().to_string())
|
||||
.and_then(|name| name.rsplit('-').next().map(ToOwned::to_owned))
|
||||
.expect("run dir should contain resolvable run id")
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
use fabro_test::test_context;
|
||||
|
||||
use super::{
|
||||
completed_nodes, find_run_dir, fixture, read_conclusion, sandbox_tests, store_dump_export,
|
||||
timeout_for,
|
||||
completed_nodes, find_run_dir, fixture, read_conclusion, run_id_for, sandbox_tests,
|
||||
store_dump_export, timeout_for,
|
||||
};
|
||||
|
||||
sandbox_tests!(command_agent_mixed, keys = ["ANTHROPIC_API_KEY"]);
|
||||
|
|
@ -43,7 +43,7 @@ fn scenario_command_agent_mixed(sandbox: &str) {
|
|||
"verify should be completed"
|
||||
);
|
||||
|
||||
let export_dir = store_dump_export(&context, &run_dir.file_name().unwrap().to_string_lossy());
|
||||
let export_dir = store_dump_export(&context, &run_id_for(&run_dir));
|
||||
let stdout = std::fs::read_to_string(export_dir.join("nodes/verify/visit-1/stdout.log"))
|
||||
.expect("verify stdout.log should exist");
|
||||
assert!(
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
use fabro_test::test_context;
|
||||
|
||||
use super::{
|
||||
completed_nodes, find_run_dir, fixture, read_conclusion, sandbox_tests, store_dump_export,
|
||||
timeout_for,
|
||||
completed_nodes, find_run_dir, fixture, read_conclusion, run_id_for, sandbox_tests,
|
||||
store_dump_export, timeout_for,
|
||||
};
|
||||
|
||||
sandbox_tests!(command_pipeline);
|
||||
|
|
@ -42,7 +42,7 @@ fn scenario_command_pipeline(sandbox: &str) {
|
|||
"step2 should be completed"
|
||||
);
|
||||
|
||||
let export_dir = store_dump_export(&context, &run_dir.file_name().unwrap().to_string_lossy());
|
||||
let export_dir = store_dump_export(&context, &run_id_for(&run_dir));
|
||||
let stdout1 = std::fs::read_to_string(export_dir.join("nodes/step1/visit-1/stdout.log"))
|
||||
.expect("step1 stdout.log should exist");
|
||||
assert!(
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
use fabro_test::test_context;
|
||||
|
||||
use super::{
|
||||
completed_nodes, find_run_dir, fixture, has_event, read_conclusion, read_json, sandbox_tests,
|
||||
store_dump_export, timeout_for,
|
||||
completed_nodes, find_run_dir, fixture, has_event, read_conclusion, read_run_record,
|
||||
run_id_for, sandbox_tests, store_dump_export, timeout_for,
|
||||
};
|
||||
|
||||
sandbox_tests!(full_stack, keys = ["ANTHROPIC_API_KEY"]);
|
||||
|
|
@ -38,7 +38,7 @@ fn scenario_full_stack(sandbox: &str) {
|
|||
);
|
||||
|
||||
// RunRecord should have key fields
|
||||
let run_record = read_json(&run_dir.join("run.json"));
|
||||
let run_record = read_run_record(&run_dir);
|
||||
assert!(
|
||||
run_record["run_id"].as_str().is_some(),
|
||||
"run record should have run_id"
|
||||
|
|
@ -68,7 +68,7 @@ fn scenario_full_stack(sandbox: &str) {
|
|||
}
|
||||
|
||||
// Verify node stdout should contain PASS
|
||||
let export_dir = store_dump_export(&context, &run_dir.file_name().unwrap().to_string_lossy());
|
||||
let export_dir = store_dump_export(&context, &run_id_for(&run_dir));
|
||||
let stdout = std::fs::read_to_string(export_dir.join("nodes/verify/visit-1/stdout.log"))
|
||||
.expect("verify stdout.log should exist");
|
||||
assert!(
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ use std::time::Duration;
|
|||
use crate::cmd::support::RunProjection;
|
||||
use fabro_config::Storage;
|
||||
use fabro_server::bind::Bind;
|
||||
use fabro_store::EventEnvelope;
|
||||
use fabro_test::TestContext;
|
||||
use serde_json::Value;
|
||||
|
||||
|
|
@ -25,15 +26,22 @@ pub(super) fn fixture(name: &str) -> PathBuf {
|
|||
.join(name)
|
||||
}
|
||||
|
||||
pub(super) fn read_json(path: &Path) -> Value {
|
||||
let content = std::fs::read_to_string(path)
|
||||
.unwrap_or_else(|e| panic!("failed to read {}: {e}", path.display()));
|
||||
serde_json::from_str(&content)
|
||||
.unwrap_or_else(|e| panic!("failed to parse {}: {e}", path.display()))
|
||||
pub(super) fn read_conclusion(run_dir: &Path) -> Value {
|
||||
serde_json::to_value(
|
||||
run_state(run_dir)
|
||||
.conclusion
|
||||
.expect("run store conclusion should exist"),
|
||||
)
|
||||
.expect("conclusion should serialize")
|
||||
}
|
||||
|
||||
pub(super) fn read_conclusion(run_dir: &Path) -> Value {
|
||||
read_json(&run_dir.join("conclusion.json"))
|
||||
pub(super) fn read_run_record(run_dir: &Path) -> Value {
|
||||
serde_json::to_value(
|
||||
run_state(run_dir)
|
||||
.run
|
||||
.expect("run store run record should exist"),
|
||||
)
|
||||
.expect("run record should serialize")
|
||||
}
|
||||
|
||||
pub(super) fn completed_nodes(run_dir: &Path) -> Vec<String> {
|
||||
|
|
@ -44,15 +52,13 @@ pub(super) fn completed_nodes(run_dir: &Path) -> Vec<String> {
|
|||
}
|
||||
|
||||
pub(super) fn has_event(run_dir: &Path, event_name: &str) -> bool {
|
||||
let path = run_dir.join("progress.jsonl");
|
||||
let content = std::fs::read_to_string(&path)
|
||||
.unwrap_or_else(|e| panic!("failed to read progress.jsonl: {e}"));
|
||||
content.lines().any(|line| {
|
||||
if let Ok(v) = serde_json::from_str::<Value>(line) {
|
||||
v["event"].as_str() == Some(event_name)
|
||||
} else {
|
||||
false
|
||||
}
|
||||
run_events(run_dir).into_iter().any(|event| {
|
||||
event
|
||||
.payload
|
||||
.as_value()
|
||||
.get("event")
|
||||
.and_then(Value::as_str)
|
||||
== Some(event_name)
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -77,10 +83,11 @@ pub(super) fn find_run_dir(context: &TestContext) -> PathBuf {
|
|||
context.single_run_dir()
|
||||
}
|
||||
|
||||
pub(super) fn run_id_for(run_dir: &Path) -> String {
|
||||
infer_run_id(run_dir)
|
||||
}
|
||||
|
||||
fn infer_run_id(run_dir: &Path) -> String {
|
||||
if let Ok(id) = std::fs::read_to_string(run_dir.join("id.txt")) {
|
||||
return id.trim().to_string();
|
||||
}
|
||||
run_dir
|
||||
.file_name()
|
||||
.map(|name| name.to_string_lossy().to_string())
|
||||
|
|
@ -158,6 +165,17 @@ fn run_state(run_dir: &Path) -> RunProjection {
|
|||
))
|
||||
}
|
||||
|
||||
fn run_events(run_dir: &Path) -> Vec<EventEnvelope> {
|
||||
let run_id = infer_run_id(run_dir);
|
||||
let runs_dir = run_dir.parent().expect("run dir should have parent");
|
||||
let storage_dir = runs_dir.parent().expect("runs dir should have parent");
|
||||
let response: serde_json::Value = block_on(get_server_json_for_storage(
|
||||
storage_dir,
|
||||
&format!("/api/v1/runs/{run_id}/events"),
|
||||
));
|
||||
serde_json::from_value(response["data"].clone()).expect("event list should parse")
|
||||
}
|
||||
|
||||
macro_rules! sandbox_tests {
|
||||
($name:ident) => {
|
||||
sandbox_tests!($name, keys = []);
|
||||
|
|
|
|||
|
|
@ -1122,29 +1122,29 @@ impl TestContext {
|
|||
|
||||
/// Return the only run directory currently present under storage.
|
||||
pub fn single_run_dir(&self) -> PathBuf {
|
||||
let scratch_dir = self.storage_dir.join("scratch");
|
||||
let entries: Vec<_> = std::fs::read_dir(&scratch_dir)
|
||||
.expect("scratch directory should exist")
|
||||
.flatten()
|
||||
.map(|entry| entry.path())
|
||||
.filter(|path| path.is_dir())
|
||||
.filter(|path| {
|
||||
let Ok(contents) = std::fs::read_to_string(path.join("run.json")) else {
|
||||
return false;
|
||||
};
|
||||
serde_json::from_str::<Value>(&contents)
|
||||
.ok()
|
||||
.and_then(|value| {
|
||||
value
|
||||
.get("labels")
|
||||
.and_then(Value::as_object)
|
||||
.and_then(|labels| labels.get("fabro_test_case"))
|
||||
.and_then(Value::as_str)
|
||||
.map(|value| value == self.test_case_id())
|
||||
})
|
||||
.unwrap_or(false)
|
||||
let output = self
|
||||
.ps()
|
||||
.args(["-a", "--json", "--label", &self.test_case_label()])
|
||||
.output()
|
||||
.expect("ps should execute");
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"ps should succeed\nstdout:\n{}\nstderr:\n{}",
|
||||
String::from_utf8_lossy(&output.stdout),
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
let runs: Vec<Value> =
|
||||
serde_json::from_slice(&output.stdout).expect("ps JSON should parse");
|
||||
let entries: Vec<_> = runs
|
||||
.into_iter()
|
||||
.filter_map(|run| {
|
||||
run.get("run_id")
|
||||
.and_then(Value::as_str)
|
||||
.map(ToOwned::to_owned)
|
||||
})
|
||||
.map(|run_id| self.find_run_dir(&run_id))
|
||||
.collect();
|
||||
let scratch_dir = self.storage_dir.join("scratch");
|
||||
assert_eq!(
|
||||
entries.len(),
|
||||
1,
|
||||
|
|
|
|||
|
|
@ -19,9 +19,7 @@ pub fn parse_legacy_blob_file_ref(value: &str) -> Option<RunBlobId> {
|
|||
let path = value.strip_prefix("file://")?;
|
||||
let blob_id = parse_blob_file_name(path)?;
|
||||
|
||||
if has_path_suffix(path, &["cache", "artifacts", "values"])
|
||||
|| has_path_suffix(path, &[".fabro", "artifacts"])
|
||||
{
|
||||
if has_path_suffix(path, &[".fabro", "artifacts"]) {
|
||||
Some(blob_id)
|
||||
} else {
|
||||
None
|
||||
|
|
@ -72,14 +70,6 @@ mod tests {
|
|||
assert_eq!(parse_blob_ref(&formatted), Some(blob_id));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_local_blob_file_ref_is_recognized() {
|
||||
let blob_id = RunBlobId::new(b"hello");
|
||||
let value = format!("file:///tmp/run/cache/artifacts/values/{blob_id}.json");
|
||||
|
||||
assert_eq!(parse_legacy_blob_file_ref(&value), Some(blob_id));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_remote_blob_file_ref_is_recognized() {
|
||||
let blob_id = RunBlobId::new(b"hello");
|
||||
|
|
|
|||
|
|
@ -74,8 +74,8 @@ pub fn is_artifact_pointer(value: &Value) -> bool {
|
|||
|
||||
/// Resolve an artifact pointer to the base name displayed in preamble rendering.
|
||||
///
|
||||
/// Given `"file:///tmp/logs/cache/artifacts/values/response.plan.json"`, returns
|
||||
/// `"See: /tmp/logs/cache/artifacts/values/response.plan.json"`.
|
||||
/// Given `"file:///tmp/logs/runtime/blobs/response.plan.json"`, returns
|
||||
/// `"See: /tmp/logs/runtime/blobs/response.plan.json"`.
|
||||
#[must_use]
|
||||
pub fn format_artifact_reference(path: &str) -> String {
|
||||
format!("See: {path}")
|
||||
|
|
@ -432,10 +432,10 @@ mod tests {
|
|||
|
||||
#[test]
|
||||
fn artifact_path_extracts_path_from_pointer() {
|
||||
let value = serde_json::json!("file:///tmp/logs/cache/artifacts/values/response.plan.json");
|
||||
let value = serde_json::json!("file:///tmp/logs/runtime/blobs/response.plan.json");
|
||||
assert_eq!(
|
||||
artifact_path(&value),
|
||||
Some("/tmp/logs/cache/artifacts/values/response.plan.json")
|
||||
Some("/tmp/logs/runtime/blobs/response.plan.json")
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -494,9 +494,7 @@ mod tests {
|
|||
),
|
||||
(
|
||||
"response.work".to_string(),
|
||||
serde_json::json!(format!(
|
||||
"file:///tmp/run/cache/artifacts/values/{blob_id}.json"
|
||||
)),
|
||||
serde_json::json!(format!("file:///sandbox/.fabro/artifacts/{blob_id}.json")),
|
||||
),
|
||||
]),
|
||||
node_outcomes: HashMap::from([(
|
||||
|
|
|
|||
|
|
@ -803,7 +803,7 @@ mod tests {
|
|||
crate::run_status::RunStatus::Submitted
|
||||
);
|
||||
assert_eq!(created.run_dir, default_run_dir(&fixtures::RUN_1));
|
||||
assert!(!created.run_dir.join("id.txt").exists());
|
||||
assert!(created.run_dir.is_dir());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
|
|
|||
|
|
@ -50,5 +50,5 @@ pub async fn resume(run_dir: &Path, services: StartServices) -> Result<Started,
|
|||
}
|
||||
|
||||
fn cleanup_resume_artifacts(run_dir: &Path) {
|
||||
let _ = std::fs::remove_file(run_dir.join("detached_failure.json"));
|
||||
let _ = run_dir;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1099,12 +1099,6 @@ mod tests {
|
|||
HashMap::new(),
|
||||
HashMap::new(),
|
||||
);
|
||||
std::fs::write(
|
||||
run_dir.join("checkpoint.json"),
|
||||
serde_json::to_string_pretty(&checkpoint).unwrap(),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
let conclusion = crate::records::Conclusion {
|
||||
timestamp: Utc::now(),
|
||||
status: StageStatus::Success,
|
||||
|
|
@ -1115,10 +1109,51 @@ mod tests {
|
|||
billing: None,
|
||||
total_retries: 0,
|
||||
};
|
||||
std::fs::write(
|
||||
run_dir.join("conclusion.json"),
|
||||
serde_json::to_string_pretty(&conclusion).unwrap(),
|
||||
let run_store = store.open_run(&fixtures::RUN_1).await.unwrap();
|
||||
crate::event::append_event(
|
||||
&run_store,
|
||||
&fixtures::RUN_1,
|
||||
&Event::CheckpointCompleted {
|
||||
node_id: checkpoint.current_node.clone(),
|
||||
status: "success".to_string(),
|
||||
current_node: checkpoint.current_node.clone(),
|
||||
completed_nodes: checkpoint.completed_nodes.clone(),
|
||||
node_retries: checkpoint.node_retries.clone().into_iter().collect(),
|
||||
context_values: checkpoint.context_values.clone().into_iter().collect(),
|
||||
node_outcomes: checkpoint.node_outcomes.clone().into_iter().collect(),
|
||||
next_node_id: checkpoint.next_node_id.clone(),
|
||||
git_commit_sha: checkpoint.git_commit_sha.clone(),
|
||||
loop_failure_signatures: checkpoint
|
||||
.loop_failure_signatures
|
||||
.iter()
|
||||
.map(|(sig, count)| (sig.to_string(), *count))
|
||||
.collect(),
|
||||
restart_failure_signatures: checkpoint
|
||||
.restart_failure_signatures
|
||||
.iter()
|
||||
.map(|(sig, count)| (sig.to_string(), *count))
|
||||
.collect(),
|
||||
node_visits: checkpoint.node_visits.clone().into_iter().collect(),
|
||||
diff: None,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
crate::event::append_event(
|
||||
&run_store,
|
||||
&fixtures::RUN_1,
|
||||
&Event::WorkflowRunCompleted {
|
||||
duration_ms: conclusion.duration_ms,
|
||||
artifact_count: 0,
|
||||
status: "success".to_string(),
|
||||
reason: None,
|
||||
total_usd_micros: None,
|
||||
final_git_commit_sha: None,
|
||||
final_patch: None,
|
||||
billing: None,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let result = resume(
|
||||
|
|
|
|||
|
|
@ -182,8 +182,10 @@ mod tests {
|
|||
.unwrap();
|
||||
|
||||
assert!(run_dir.is_dir());
|
||||
assert!(!run_dir.join("workflow.fabro").exists());
|
||||
assert!(!run_dir.join("run.json").exists());
|
||||
assert!(
|
||||
std::fs::read_dir(&run_dir).unwrap().next().is_none(),
|
||||
"persist should not project files into the scratch dir"
|
||||
);
|
||||
assert_eq!(persisted.run_dir(), run_dir.as_path());
|
||||
assert_eq!(
|
||||
serde_json::to_value(persisted.run_record().graph.clone()).unwrap(),
|
||||
|
|
|
|||
|
|
@ -115,7 +115,8 @@ pub(crate) struct PersistOptions {
|
|||
pub run_record: RunRecord,
|
||||
}
|
||||
|
||||
/// Output of the PERSIST phase. Run directory created, run.json and workflow.fabro written.
|
||||
/// Output of the PERSIST phase. Run directory created and the validated workflow
|
||||
/// is persisted into the durable run record.
|
||||
#[derive(Debug)]
|
||||
#[non_exhaustive]
|
||||
pub struct Persisted {
|
||||
|
|
|
|||
|
|
@ -160,20 +160,16 @@ fn scan_orphan_runs(base: &Path) -> Result<Vec<RunInfo>> {
|
|||
}
|
||||
|
||||
let dir_name = entry.file_name().to_string_lossy().to_string();
|
||||
if parse_run_id(&dir_name).is_none() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let mtime_dt = entry
|
||||
.metadata()
|
||||
.ok()
|
||||
.and_then(|m| m.modified().ok())
|
||||
.map(|time| -> DateTime<Utc> { time.into() });
|
||||
|
||||
let run_id = std::fs::read_to_string(path.join("id.txt"))
|
||||
.ok()
|
||||
.and_then(|s| parse_run_id(&s))
|
||||
.or_else(|| parse_run_id(&dir_name));
|
||||
if run_id.is_none() {
|
||||
continue;
|
||||
}
|
||||
|
||||
runs.push(RunInfo::new(
|
||||
None,
|
||||
RunLocalState {
|
||||
|
|
@ -437,7 +433,6 @@ mod tests {
|
|||
let temp = tempfile::tempdir().unwrap();
|
||||
let run_dir = make_run_dir(temp.path(), &fixtures::RUN_1);
|
||||
std::fs::create_dir_all(&run_dir).unwrap();
|
||||
std::fs::write(run_dir.join("id.txt"), format!("{}\n", fixtures::RUN_1)).unwrap();
|
||||
|
||||
let store = memory_store();
|
||||
let run_record = sample_run_record();
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::hash::{Hash, Hasher};
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
|
@ -17,6 +18,15 @@ use crate::pipeline::types::Initialized;
|
|||
use crate::records::Checkpoint;
|
||||
use crate::run_options::RunOptions;
|
||||
|
||||
pub fn test_store_dir(run_dir: &std::path::Path) -> PathBuf {
|
||||
let mut hasher = std::collections::hash_map::DefaultHasher::new();
|
||||
std::process::id().hash(&mut hasher);
|
||||
run_dir.hash(&mut hasher);
|
||||
std::env::temp_dir()
|
||||
.join("fabro-test-run-stores")
|
||||
.join(format!("{:016x}", hasher.finish()))
|
||||
}
|
||||
|
||||
struct InitializedOptions {
|
||||
hook_runner: Option<Arc<fabro_hooks::HookRunner>>,
|
||||
env: HashMap<String, String>,
|
||||
|
|
@ -44,16 +54,12 @@ async fn initialized(
|
|||
options: InitializedOptions,
|
||||
) -> InitializedState {
|
||||
std::fs::create_dir_all(&run_options.run_dir).expect("failed to create run dir");
|
||||
std::fs::create_dir_all(run_options.run_dir.join("store"))
|
||||
.expect("failed to create local test run store dir");
|
||||
std::fs::write(
|
||||
run_options.run_dir.join("id.txt"),
|
||||
run_options.run_id.to_string(),
|
||||
)
|
||||
.expect("failed to write run id marker");
|
||||
let store_dir = test_store_dir(&run_options.run_dir);
|
||||
let _ = std::fs::remove_dir_all(&store_dir);
|
||||
std::fs::create_dir_all(&store_dir).expect("failed to create local test run store dir");
|
||||
let store = Arc::new(Database::new(
|
||||
Arc::new(
|
||||
LocalFileSystem::new_with_prefix(run_options.run_dir.join("store"))
|
||||
LocalFileSystem::new_with_prefix(&store_dir)
|
||||
.expect("failed to create local test run store"),
|
||||
),
|
||||
"",
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ use fabro_workflow::handler::{Handler, HandlerRegistry};
|
|||
use fabro_workflow::outcome::{Outcome, OutcomeExt, StageStatus};
|
||||
use fabro_workflow::records::Checkpoint;
|
||||
use fabro_workflow::run_options::{GitCheckpointOptions, RunOptions};
|
||||
use fabro_workflow::test_support::WorkflowRunner;
|
||||
use fabro_workflow::test_support::{WorkflowRunner, test_store_dir};
|
||||
use object_store::local::LocalFileSystem;
|
||||
use ulid::Ulid;
|
||||
|
||||
|
|
@ -44,83 +44,101 @@ fn test_run_id(label: &str) -> RunId {
|
|||
RunId::from(Ulid(u128::from(hasher.finish())))
|
||||
}
|
||||
|
||||
fn load_checkpoint(path: &Path) -> Result<Checkpoint, Box<dyn std::error::Error>> {
|
||||
if !path.exists()
|
||||
&& path
|
||||
.file_name()
|
||||
.is_some_and(|name| name == "checkpoint.json")
|
||||
{
|
||||
let run_dir = path
|
||||
.parent()
|
||||
.ok_or("checkpoint path should have a parent")?;
|
||||
let local_store_dir = run_dir.join("store");
|
||||
let (store_dir, run_id) =
|
||||
if let Ok(run_id_text) = std::fs::read_to_string(run_dir.join("id.txt")) {
|
||||
(local_store_dir, run_id_text.trim().parse()?)
|
||||
} else {
|
||||
let runs_dir = run_dir.parent().ok_or("run dir should have parent")?;
|
||||
let storage_dir = runs_dir.parent().ok_or("runs dir should have parent")?;
|
||||
let run_id: RunId = run_dir
|
||||
.file_name()
|
||||
.ok_or("run dir should have file name")?
|
||||
.to_string_lossy()
|
||||
.rsplit('-')
|
||||
.next()
|
||||
.ok_or("run dir should contain run id suffix")?
|
||||
.parse()?;
|
||||
(storage_dir.join("store"), run_id)
|
||||
};
|
||||
let object_store = Arc::new(LocalFileSystem::new_with_prefix(store_dir)?);
|
||||
let store = Arc::new(Database::new(
|
||||
object_store,
|
||||
"",
|
||||
std::time::Duration::from_millis(1),
|
||||
));
|
||||
let state = if tokio::runtime::Handle::try_current().is_ok() {
|
||||
std::thread::spawn(
|
||||
move || -> Result<_, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let run = runtime.block_on(store.open_run_reader(&run_id))?;
|
||||
let state = runtime.block_on(async {
|
||||
for attempt in 0..20 {
|
||||
let state = run.state().await?;
|
||||
if state.checkpoint.is_some() || attempt == 19 {
|
||||
return Ok::<_, fabro_store::StoreError>(state);
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
|
||||
fn load_run_checkpoint(run_dir: &Path) -> Result<Checkpoint, Box<dyn std::error::Error>> {
|
||||
let run_dir = run_dir.to_path_buf();
|
||||
let uses_shared_store = run_dir
|
||||
.parent()
|
||||
.and_then(Path::file_name)
|
||||
.is_some_and(|name| name == "scratch");
|
||||
let store_dir = if uses_shared_store {
|
||||
let runs_dir = run_dir.parent().ok_or("run dir should have parent")?;
|
||||
let storage_dir = runs_dir.parent().ok_or("runs dir should have parent")?;
|
||||
storage_dir.join("store")
|
||||
} else {
|
||||
test_store_dir(&run_dir)
|
||||
};
|
||||
let object_store = Arc::new(LocalFileSystem::new_with_prefix(store_dir)?);
|
||||
let store = Arc::new(Database::new(
|
||||
object_store,
|
||||
"",
|
||||
std::time::Duration::from_millis(1),
|
||||
));
|
||||
let state = if tokio::runtime::Handle::try_current().is_ok() {
|
||||
std::thread::spawn(
|
||||
move || -> Result<_, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let run_id = if uses_shared_store {
|
||||
run_dir
|
||||
.file_name()
|
||||
.ok_or("run dir should have file name")?
|
||||
.to_string_lossy()
|
||||
.rsplit('-')
|
||||
.next()
|
||||
.ok_or("run dir should contain run id suffix")?
|
||||
.parse()?
|
||||
} else {
|
||||
runtime
|
||||
.block_on(store.list_runs(&fabro_store::ListRunsQuery::default()))?
|
||||
.into_iter()
|
||||
.next()
|
||||
.ok_or("test store should contain one run")?
|
||||
.run_id
|
||||
};
|
||||
let run = runtime.block_on(store.open_run_reader(&run_id))?;
|
||||
let state = runtime.block_on(async {
|
||||
for attempt in 0..20 {
|
||||
let state = run.state().await?;
|
||||
if state.checkpoint.is_some() || attempt == 19 {
|
||||
return Ok::<_, fabro_store::StoreError>(state);
|
||||
}
|
||||
unreachable!()
|
||||
})?;
|
||||
Ok(state)
|
||||
},
|
||||
)
|
||||
.join()
|
||||
.map_err(|_| "checkpoint loader thread panicked")?
|
||||
.map_err(|err| err.to_string())?
|
||||
} else {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let run = runtime.block_on(store.open_run_reader(&run_id))?;
|
||||
runtime.block_on(async {
|
||||
for attempt in 0..20 {
|
||||
let state = run.state().await?;
|
||||
if state.checkpoint.is_some() || attempt == 19 {
|
||||
return Ok::<_, fabro_store::StoreError>(state);
|
||||
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
|
||||
}
|
||||
unreachable!()
|
||||
})?
|
||||
unreachable!()
|
||||
})?;
|
||||
Ok(state)
|
||||
},
|
||||
)
|
||||
.join()
|
||||
.map_err(|_| "checkpoint loader thread panicked")?
|
||||
.map_err(|err| err.to_string())?
|
||||
} else {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let run_id = if uses_shared_store {
|
||||
run_dir
|
||||
.file_name()
|
||||
.ok_or("run dir should have file name")?
|
||||
.to_string_lossy()
|
||||
.rsplit('-')
|
||||
.next()
|
||||
.ok_or("run dir should contain run id suffix")?
|
||||
.parse()?
|
||||
} else {
|
||||
runtime
|
||||
.block_on(store.list_runs(&fabro_store::ListRunsQuery::default()))?
|
||||
.into_iter()
|
||||
.next()
|
||||
.ok_or("test store should contain one run")?
|
||||
.run_id
|
||||
};
|
||||
return state
|
||||
.checkpoint
|
||||
.ok_or_else(|| "checkpoint should exist in run store".into());
|
||||
}
|
||||
let data = std::fs::read_to_string(path)?;
|
||||
Ok(serde_json::from_str(&data)?)
|
||||
let run = runtime.block_on(store.open_run_reader(&run_id))?;
|
||||
runtime.block_on(async {
|
||||
for attempt in 0..20 {
|
||||
let state = run.state().await?;
|
||||
if state.checkpoint.is_some() || attempt == 19 {
|
||||
return Ok::<_, fabro_store::StoreError>(state);
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
|
||||
}
|
||||
unreachable!()
|
||||
})?
|
||||
};
|
||||
return state
|
||||
.checkpoint
|
||||
.ok_or_else(|| "checkpoint should exist in run store".into());
|
||||
}
|
||||
|
||||
async fn create_env() -> DaytonaSandbox {
|
||||
|
|
@ -490,8 +508,7 @@ async fn daytona_pipeline_artifact_offload_and_sync() {
|
|||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
// Checkpoint should persist a durable blob ref.
|
||||
let checkpoint =
|
||||
load_checkpoint(&dir.path().join("checkpoint.json")).expect("checkpoint should load");
|
||||
let checkpoint = load_run_checkpoint(dir.path()).expect("checkpoint should load");
|
||||
let pointer_value = checkpoint
|
||||
.context_values
|
||||
.get("response.big_output")
|
||||
|
|
@ -705,8 +722,7 @@ async fn daytona_git_checkpoint_remote_emits_events() {
|
|||
}
|
||||
|
||||
// Verify checkpoint.json has git_commit_sha
|
||||
let checkpoint =
|
||||
load_checkpoint(&dir.path().join("checkpoint.json")).expect("checkpoint should load");
|
||||
let checkpoint = load_run_checkpoint(dir.path()).expect("checkpoint should load");
|
||||
assert!(
|
||||
checkpoint.git_commit_sha.is_some(),
|
||||
"checkpoint should have git_commit_sha"
|
||||
|
|
@ -856,8 +872,7 @@ async fn daytona_parallel_git_branching_e2e() {
|
|||
);
|
||||
|
||||
// Verify parallel.results has head_sha for each branch
|
||||
let checkpoint =
|
||||
load_checkpoint(&run_tmp.path().join("checkpoint.json")).expect("checkpoint should load");
|
||||
let checkpoint = load_run_checkpoint(run_tmp.path()).expect("checkpoint should load");
|
||||
let parallel_results = checkpoint
|
||||
.context_values
|
||||
.get("parallel.results")
|
||||
|
|
|
|||
|
|
@ -44,7 +44,7 @@ use fabro_workflow::handler::{Handler, HandlerRegistry};
|
|||
use fabro_workflow::outcome::{Outcome, OutcomeExt, StageStatus};
|
||||
use fabro_workflow::records::{Checkpoint, CheckpointExt};
|
||||
use fabro_workflow::run_options::{GitCheckpointOptions, RunOptions};
|
||||
use fabro_workflow::test_support::{WorkflowRunner, run_graph_with_hooks};
|
||||
use fabro_workflow::test_support::{WorkflowRunner, run_graph_with_hooks, test_store_dir};
|
||||
use fabro_workflow::transforms::stylesheet::{apply_stylesheet, parse_stylesheet};
|
||||
use fabro_workflow::transforms::{
|
||||
StylesheetApplicationTransform, Transform, VariableExpansionTransform,
|
||||
|
|
@ -73,72 +73,105 @@ fn load_checkpoint(path: &Path) -> Result<Checkpoint, Box<dyn std::error::Error>
|
|||
let run_dir = path
|
||||
.parent()
|
||||
.ok_or("checkpoint path should have a parent")?;
|
||||
let local_store_dir = run_dir.join("store");
|
||||
let (store_dir, run_id) =
|
||||
if let Ok(run_id_text) = std::fs::read_to_string(run_dir.join("id.txt")) {
|
||||
(local_store_dir, run_id_text.trim().parse()?)
|
||||
} else {
|
||||
let runs_dir = run_dir.parent().ok_or("run dir should have parent")?;
|
||||
let storage_dir = runs_dir.parent().ok_or("runs dir should have parent")?;
|
||||
let run_id: RunId = run_dir
|
||||
.file_name()
|
||||
.ok_or("run dir should have file name")?
|
||||
.to_string_lossy()
|
||||
.rsplit('-')
|
||||
.next()
|
||||
.ok_or("run dir should contain run id suffix")?
|
||||
.parse()?;
|
||||
(storage_dir.join("store"), run_id)
|
||||
};
|
||||
let object_store = Arc::new(LocalFileSystem::new_with_prefix(store_dir)?);
|
||||
let store = Arc::new(Database::new(object_store, "", Duration::from_millis(1)));
|
||||
let state = if tokio::runtime::Handle::try_current().is_ok() {
|
||||
std::thread::spawn(
|
||||
move || -> Result<_, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let run = runtime.block_on(store.open_run_reader(&run_id))?;
|
||||
let state = runtime.block_on(async {
|
||||
for attempt in 0..20 {
|
||||
let state = run.state().await?;
|
||||
if state.checkpoint.is_some() || attempt == 19 {
|
||||
return Ok::<_, fabro_store::StoreError>(state);
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
}
|
||||
unreachable!()
|
||||
})?;
|
||||
Ok(state)
|
||||
},
|
||||
)
|
||||
.join()
|
||||
.map_err(|_| "checkpoint loader thread panicked")?
|
||||
.map_err(|err| err.to_string())?
|
||||
} else {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let run = runtime.block_on(store.open_run_reader(&run_id))?;
|
||||
runtime.block_on(async {
|
||||
for attempt in 0..20 {
|
||||
let state = run.state().await?;
|
||||
if state.checkpoint.is_some() || attempt == 19 {
|
||||
return Ok::<_, fabro_store::StoreError>(state);
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
}
|
||||
unreachable!()
|
||||
})?
|
||||
};
|
||||
return state
|
||||
.checkpoint
|
||||
.ok_or_else(|| "checkpoint should exist in run store".into());
|
||||
return load_run_checkpoint(run_dir);
|
||||
}
|
||||
let data = std::fs::read_to_string(path)?;
|
||||
Ok(serde_json::from_str(&data)?)
|
||||
}
|
||||
|
||||
fn load_run_checkpoint(run_dir: &Path) -> Result<Checkpoint, Box<dyn std::error::Error>> {
|
||||
let run_dir = run_dir.to_path_buf();
|
||||
let uses_shared_store = run_dir
|
||||
.parent()
|
||||
.and_then(Path::file_name)
|
||||
.is_some_and(|name| name == "scratch");
|
||||
let store_dir = if uses_shared_store {
|
||||
let runs_dir = run_dir.parent().ok_or("run dir should have parent")?;
|
||||
let storage_dir = runs_dir.parent().ok_or("runs dir should have parent")?;
|
||||
storage_dir.join("store")
|
||||
} else {
|
||||
test_store_dir(&run_dir)
|
||||
};
|
||||
let object_store = Arc::new(LocalFileSystem::new_with_prefix(store_dir)?);
|
||||
let store = Arc::new(Database::new(object_store, "", Duration::from_millis(1)));
|
||||
let state = if tokio::runtime::Handle::try_current().is_ok() {
|
||||
std::thread::spawn(
|
||||
move || -> Result<_, Box<dyn std::error::Error + Send + Sync>> {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let run_id = if uses_shared_store {
|
||||
run_dir
|
||||
.file_name()
|
||||
.ok_or("run dir should have file name")?
|
||||
.to_string_lossy()
|
||||
.rsplit('-')
|
||||
.next()
|
||||
.ok_or("run dir should contain run id suffix")?
|
||||
.parse()?
|
||||
} else {
|
||||
runtime
|
||||
.block_on(store.list_runs(&fabro_store::ListRunsQuery::default()))?
|
||||
.into_iter()
|
||||
.next()
|
||||
.ok_or("test store should contain one run")?
|
||||
.run_id
|
||||
};
|
||||
let run = runtime.block_on(store.open_run_reader(&run_id))?;
|
||||
let state = runtime.block_on(async {
|
||||
for attempt in 0..20 {
|
||||
let state = run.state().await?;
|
||||
if state.checkpoint.is_some() || attempt == 19 {
|
||||
return Ok::<_, fabro_store::StoreError>(state);
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
}
|
||||
unreachable!()
|
||||
})?;
|
||||
Ok(state)
|
||||
},
|
||||
)
|
||||
.join()
|
||||
.map_err(|_| "checkpoint loader thread panicked")?
|
||||
.map_err(|err| err.to_string())?
|
||||
} else {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()?;
|
||||
let run_id = if uses_shared_store {
|
||||
run_dir
|
||||
.file_name()
|
||||
.ok_or("run dir should have file name")?
|
||||
.to_string_lossy()
|
||||
.rsplit('-')
|
||||
.next()
|
||||
.ok_or("run dir should contain run id suffix")?
|
||||
.parse()?
|
||||
} else {
|
||||
runtime
|
||||
.block_on(store.list_runs(&fabro_store::ListRunsQuery::default()))?
|
||||
.into_iter()
|
||||
.next()
|
||||
.ok_or("test store should contain one run")?
|
||||
.run_id
|
||||
};
|
||||
let run = runtime.block_on(store.open_run_reader(&run_id))?;
|
||||
runtime.block_on(async {
|
||||
for attempt in 0..20 {
|
||||
let state = run.state().await?;
|
||||
if state.checkpoint.is_some() || attempt == 19 {
|
||||
return Ok::<_, fabro_store::StoreError>(state);
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(10)).await;
|
||||
}
|
||||
unreachable!()
|
||||
})?
|
||||
};
|
||||
return state
|
||||
.checkpoint
|
||||
.ok_or_else(|| "checkpoint should exist in run store".into());
|
||||
}
|
||||
|
||||
fn save_checkpoint(path: &Path, checkpoint: &Checkpoint) {
|
||||
std::fs::write(path, serde_json::to_string_pretty(checkpoint).unwrap()).unwrap();
|
||||
}
|
||||
|
|
@ -313,8 +346,7 @@ async fn end_to_end_linear_pipeline() {
|
|||
.expect("run should succeed");
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let checkpoint =
|
||||
load_checkpoint(&dir.path().join("checkpoint.json")).expect("checkpoint should load");
|
||||
let checkpoint = load_run_checkpoint(dir.path()).expect("checkpoint should load");
|
||||
assert!(checkpoint.completed_nodes.contains(&"start".to_string()));
|
||||
assert!(
|
||||
checkpoint
|
||||
|
|
@ -1729,7 +1761,7 @@ async fn smoke_test_with_mock_codergen_backend() {
|
|||
.expect("smoke test should succeed");
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let checkpoint = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let checkpoint = load_run_checkpoint(dir.path()).unwrap();
|
||||
assert!(
|
||||
checkpoint.completed_nodes.contains(&"plan".to_string()),
|
||||
"plan should have executed"
|
||||
|
|
@ -2254,7 +2286,7 @@ async fn tool_handler_e2e() {
|
|||
.expect("run");
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let cp = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let cp = load_run_checkpoint(dir.path()).unwrap();
|
||||
let command_output = cp
|
||||
.context_values
|
||||
.get("command.output")
|
||||
|
|
@ -2328,7 +2360,7 @@ async fn auto_approve_interviewer_e2e() {
|
|||
.expect("run");
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let cp = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let cp = load_run_checkpoint(dir.path()).unwrap();
|
||||
assert!(cp.completed_nodes.contains(&"approve".to_string()));
|
||||
assert!(!cp.completed_nodes.contains(&"reject".to_string()));
|
||||
}
|
||||
|
|
@ -2471,7 +2503,7 @@ async fn branching_loop_back_on_failure() {
|
|||
.expect("run");
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let cp = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let cp = load_run_checkpoint(dir.path()).unwrap();
|
||||
let implement_count = cp
|
||||
.completed_nodes
|
||||
.iter()
|
||||
|
|
@ -2556,7 +2588,7 @@ async fn human_gate_loops_back() {
|
|||
.expect("run");
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let cp = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let cp = load_run_checkpoint(dir.path()).unwrap();
|
||||
let gate_count = cp.completed_nodes.iter().filter(|n| *n == "gate").count();
|
||||
assert!(
|
||||
gate_count >= 2,
|
||||
|
|
@ -2616,7 +2648,7 @@ async fn scenario_ship_a_feature() {
|
|||
.expect("run");
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let cp = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let cp = load_run_checkpoint(dir.path()).unwrap();
|
||||
let command_output = cp
|
||||
.context_values
|
||||
.get("command.output")
|
||||
|
|
@ -3686,7 +3718,7 @@ async fn integration_smoke_plan_implement_review_done() {
|
|||
.expect("run");
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let cp = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let cp = load_run_checkpoint(dir.path()).unwrap();
|
||||
assert!(cp.completed_nodes.contains(&"plan".to_string()));
|
||||
assert!(cp.completed_nodes.contains(&"implement".to_string()));
|
||||
assert!(cp.completed_nodes.contains(&"review".to_string()));
|
||||
|
|
@ -6096,7 +6128,7 @@ mod real_llm {
|
|||
})
|
||||
}
|
||||
|
||||
use super::{load_checkpoint, local_env, test_run_id};
|
||||
use super::{load_run_checkpoint, local_env, test_run_id};
|
||||
use fabro_graphviz::graph::{AttrValue, Edge, Graph};
|
||||
use fabro_interview::AutoApproveInterviewer;
|
||||
use fabro_workflow::event::Emitter;
|
||||
|
|
@ -6193,7 +6225,7 @@ mod real_llm {
|
|||
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let checkpoint = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let checkpoint = load_run_checkpoint(dir.path()).unwrap();
|
||||
assert!(checkpoint.completed_nodes.contains(&"plan".to_string()));
|
||||
assert!(checkpoint.completed_nodes.contains(&"review".to_string()));
|
||||
|
||||
|
|
@ -6301,7 +6333,7 @@ mod real_llm {
|
|||
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let checkpoint = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let checkpoint = load_run_checkpoint(dir.path()).unwrap();
|
||||
let last_stage = checkpoint
|
||||
.context_values
|
||||
.get("last_stage")
|
||||
|
|
@ -6433,7 +6465,7 @@ mod real_llm {
|
|||
|
||||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
let checkpoint = load_checkpoint(&dir.path().join("checkpoint.json")).unwrap();
|
||||
let checkpoint = load_run_checkpoint(dir.path()).unwrap();
|
||||
assert!(
|
||||
checkpoint.completed_nodes.contains(&"write".to_string()),
|
||||
"write should be completed"
|
||||
|
|
@ -8616,8 +8648,7 @@ async fn large_context_values_are_offloaded_to_artifact_store() {
|
|||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
// The checkpoint context should contain a durable blob ref, not the full value.
|
||||
let checkpoint =
|
||||
load_checkpoint(&dir.path().join("checkpoint.json")).expect("checkpoint should load");
|
||||
let checkpoint = load_run_checkpoint(dir.path()).expect("checkpoint should load");
|
||||
let pointer_value = checkpoint
|
||||
.context_values
|
||||
.get("response.big_output")
|
||||
|
|
@ -8828,8 +8859,7 @@ async fn artifact_pointers_rewritten_for_remote_sandbox() {
|
|||
assert_eq!(outcome.status, StageStatus::Success);
|
||||
|
||||
// The checkpoint context should contain a durable blob ref.
|
||||
let checkpoint =
|
||||
load_checkpoint(&dir.path().join("checkpoint.json")).expect("checkpoint should load");
|
||||
let checkpoint = load_run_checkpoint(dir.path()).expect("checkpoint should load");
|
||||
let pointer_value = checkpoint
|
||||
.context_values
|
||||
.get("response.big_output")
|
||||
|
|
@ -10348,8 +10378,7 @@ async fn git_checkpoint_host_emits_events_and_diff_patch() {
|
|||
);
|
||||
|
||||
// 7. Verify checkpoint has git_commit_sha
|
||||
let checkpoint =
|
||||
load_checkpoint(&run_dir.path().join("checkpoint.json")).expect("checkpoint should load");
|
||||
let checkpoint = load_run_checkpoint(run_dir.path()).expect("checkpoint should load");
|
||||
assert!(
|
||||
checkpoint.git_commit_sha.is_some(),
|
||||
"checkpoint should have git_commit_sha"
|
||||
|
|
@ -10693,8 +10722,7 @@ async fn parallel_git_branching_host_e2e() {
|
|||
);
|
||||
|
||||
// 6. Verify parallel.results has head_sha for each branch
|
||||
let checkpoint =
|
||||
load_checkpoint(&run_dir.path().join("checkpoint.json")).expect("checkpoint should load");
|
||||
let checkpoint = load_run_checkpoint(run_dir.path()).expect("checkpoint should load");
|
||||
let parallel_results = checkpoint
|
||||
.context_values
|
||||
.get("parallel.results")
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue