scriptoriumadmin/fabro
1
0
Fork 0
mirror of https://github.com/fabro-sh/fabro.git synced 2026-09-07 08:27:12 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge remote-tracking branch 'origin/main'

Browse source 
# Conflicts:
#	lib/crates/fabro-cli/tests/it/cmd/attach.rs
#	lib/crates/fabro-cli/tests/it/cmd/create.rs
#	lib/crates/fabro-cli/tests/it/cmd/resume.rs
#	lib/crates/fabro-cli/tests/it/cmd/start.rs
This commit is contained in:
Bryan Helmkamp 2026-03-30 18:26:57 -04:00
commit 0313a1f4ee
No known key found for this signature in database
108 changed files with 7695 additions and 7991 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

23
AGENTS.md
View file

@ -26,7 +26,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
- `cd apps/marketing && bunx vercel --prod` — deploy to Vercel (project: website, domain: fabro.sh)
### Dev servers
1. `fabro serve` — starts the Rust API server (demo mode is per-request via `X-Fabro-Demo: 1` header)
1. `fabro server start` — starts the Rust API server (demo mode is per-request via `X-Fabro-Demo: 1` header)
2. `cd apps/fabro-web && bun run dev` — starts the React dev server
3. Mintlify docs dev server (requires Docker — `mintlify dev` needs Node LTS which may not match the host):
```
@ -50,7 +50,7 @@ The OpenAPI spec at `docs/api-reference/fabro-api.yaml` is the source of truth f
Fabro is an AI-powered workflow orchestration platform. Workflows are defined as Graphviz graphs, where each node is a stage (agent, prompt, command, conditional, human, parallel, etc.) executed by the workflow engine.
### Rust crates (`lib/crates/`)
- **fabro-cli** — CLI entry point. Commands: `run`, `exec`, `serve`, `validate`, `parse`, `cp`, `model`, `doctor`, `init`, `install`, `ps`, `system prune`, `llm`
- **fabro-cli** — CLI entry point. Commands: `run`, `exec`, `serve`, `validate`, `parse`, `cp`, `model`, `doctor`, `install`, `ps`, `system prune`, `llm`
- **fabro-workflow** — Core workflow engine. Parses Graphviz graphs, runs stages, manages checkpoints/resume, hooks, retros, and human-in-the-loop interactions
- **fabro-agent** — AI coding agent with tool use (Bash, Read, Write, Edit, Glob, Grep, WebFetch). `Sandbox` trait abstracts execution environments
- **fabro-server** — Axum HTTP server. Routes for runs, sessions, models, completions, usage. SSE event streaming. Demo mode via header
@ -61,7 +61,7 @@ Fabro is an AI-powered workflow orchestration platform. Workflows are defined as
- **fabro-mcp** — Model Context Protocol client/server
- **fabro-slack** — Slack integration (socket mode, blocks API)
- **fabro-devcontainer** — Parses `.devcontainer/devcontainer.json` for container setup
- **fabro-git-storage** — Git-based storage with branch store and snapshots
- **fabro-checkpoint** — Git-based checkpoint storage with branch store and metadata branches
- **fabro-telemetry** — CLI analytics (Segment) and crash reporting (Sentry), with anonymous IDs, command sanitization, and detached subprocess delivery
- **fabro-util** — Shared utilities (redaction, terminal formatting)
@ -75,12 +75,13 @@ Fabro is an AI-powered workflow orchestration platform. Workflows are defined as
- **OpenAPI-first** — `fabro-api.yaml` drives both Rust type generation (typify) and TypeScript client generation (openapi-generator)
- **Checkpoint/resume** — Workflows can be paused, checkpointed, and resumed
## Logging and events
## Strategy docs
When working on Rust crates, read the relevant strategy doc **before** making changes:
- **`files-internal/logging-strategy.md`** — read when adding `tracing` calls (`info!`, `debug!`, `warn!`, `error!`), working on error handling paths, or adding new operations that should be observable
- **`files-internal/events-strategy.md`** — read when adding or modifying `WorkflowRunEvent` variants, touching `EventEmitter`/`emit()`, changing `progress.jsonl` output, or adding new workflow stage types
- **`docs-internal/logging-strategy.md`** — read when adding `tracing` calls (`info!`, `debug!`, `warn!`, `error!`), working on error handling paths, or adding new operations that should be observable
- **`docs-internal/events-strategy.md`** — read when adding or modifying `WorkflowRunEvent` variants, touching `EventEmitter`/`emit()`, changing `progress.jsonl` output, or adding new workflow stage types
- **`files-internal/testing-strategy.md`** — read when adding or reorganizing tests, choosing between unit vs `tests/it`, deciding whether a test belongs in `cmd` vs `workflow` vs `scenario`, or deciding how to structure snapshots and fixtures
## Shell quoting in sandbox code
@ -92,6 +93,16 @@ When interpolating values into shell command strings (in `fabro-workflow`), alwa
- **Functions**: import the parent module, call as `module::function()` — `use fabro_workflow::operations; operations::create(...)`
- **No glob imports** in production code (`use foo::*`). Globs are acceptable in test modules and preludes. Enforced by clippy `wildcard_imports` lint.
## Snapshot tests (insta)
Many CLI tests use `insta` inline snapshots. When a snapshot needs updating:
1. Run `cargo insta pending-snapshots` to list what changed
2. Verify each pending snapshot is expected
3. Run `cargo insta accept` to accept all, or `cargo insta accept --snapshot <path>` for a specific one
Never run `cargo insta accept` without first checking what's pending — it accepts *all* pending snapshots, which may include unrelated changes.
## Testing workflows
- `fabro run <name>` — run a workflow by name (resolves `fabro/workflows/<name>/workflow.toml`), e.g. `fabro run repl`

39
Cargo.lock generated
View file

@ -628,6 +628,15 @@ dependencies = [
"strsim 0.11.1",
]
[[package]]
name = "clap_complete"
version = "4.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19c9f1dde76b736e3681f28cec9d5a61299cbaae0fce80a68e43724ad56031eb"
dependencies = [
"clap",
]
[[package]]
name = "clap_derive"
version = "4.5.55"
@ -1437,6 +1446,20 @@ dependencies = [
"uuid",
]
[[package]]
name = "fabro-checkpoint"
version = "0.176.2"
dependencies = [
"chrono",
"fabro-types",
"git2",
"serde",
"serde_json",
"tempfile",
"thiserror 2.0.18",
"tracing",
]
[[package]]
name = "fabro-cli"
version = "0.176.2"
@ -1448,6 +1471,7 @@ dependencies = [
"base64",
"chrono",
"clap",
"clap_complete",
"cli-table",
"console 0.15.11",
"core-foundation 0.9.4",
@ -1456,9 +1480,9 @@ dependencies = [
"dirs",
"dotenvy",
"fabro-agent",
"fabro-checkpoint",
"fabro-config",
"fabro-devcontainer",
"fabro-git-storage",
"fabro-github",
"fabro-graphviz",
"fabro-hooks",
@ -1571,17 +1595,6 @@ dependencies = [
"tracing",
]
[[package]]
name = "fabro-git-storage"
version = "0.176.2"
dependencies = [
"git2",
"tempfile",
"thiserror 2.0.18",
"tracing",
"walkdir",
]
[[package]]
name = "fabro-github"
version = "0.176.2"
@ -1983,10 +1996,10 @@ dependencies = [
"dirs",
"dotenvy",
"fabro-agent",
"fabro-checkpoint",
"fabro-config",
"fabro-core",
"fabro-devcontainer",
"fabro-git-storage",
"fabro-github",
"fabro-graphviz",
"fabro-hooks",

3
Cargo.toml
View file

@ -16,7 +16,7 @@ serde_json = { version = "1", features = ["preserve_order"] }
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", default-features = false, features = ["json", "stream", "rustls-tls"] }
ulid = "1"
uuid = { version = "1", features = ["v4"] }
uuid = { version = "1", features = ["v4", "v7"] }
rand = "0.8"
dotenvy = "0.15"
futures = "0.3"
@ -26,6 +26,7 @@ base64 = "0.22"
bytes = "1"
tokio-util = "0.7"
clap = { version = "4", features = ["derive", "env"] }
clap_complete = "4"
jsonschema = { version = "0.42", default-features = false }
chrono = { version = "0.4", features = ["clock"] }
bollard = "0.18"

2
README.md
View file

@ -124,7 +124,7 @@ Then initialize Fabro in your project:
fabro install # one-time setup
cd my-project
fabro init # per project
fabro repo init # per project
```
---

438
docs-internal/events-strategy.md
View file

@ -1,312 +1,164 @@
# Fabro Events Strategy
Fabro emits structured **workflow run events** during execution for observability. Events write to `progress.jsonl` (one JSON object per line) and `live.json` (latest event snapshot) inside the run's log directory. Events are the primary record of what happened during a run — they feed the retro system, CLI verbose output, `fabro attach`, `fabro logs`, and live monitoring.
Fabro emits structured **workflow run events** during execution for observability. Events are the durable audit trail for a run: they drive `progress.jsonl`, `live.json`, the run store, SSE streaming, CLI progress rendering, and retro analysis.
Events are distinct from tracing logs (see `logging-strategy.md`). Tracing is developer diagnostics; events are the structured audit trail consumed by tooling.
Events are distinct from tracing logs. Tracing is developer diagnostics; events are product-facing state transitions and activity records that other systems consume.
Detached runs rely on this distinction. If a warning or error needs to be visible to users after reattach, emit a `WorkflowRunEvent` rather than only writing to stderr/`detach.log`.
Detached runs rely on this distinction. If something needs to be visible after reattach, emit a `WorkflowRunEvent` rather than only logging to stderr or `detach.log`.
## Architecture
```
Engine/Handler → WorkflowRunEvent enum → EventEmitter
├─ .trace() → tracing log line (automatic)
├─ flatten_event() → progress.jsonl + live.json
└─ on_event() callbacks → CLI output, cost tracking, etc.
```text
Engine/Handler -> WorkflowRunEvent -> EventEmitter::emit()
|- trace(raw event)
|- canonicalize -> RunEventEnvelope
`- on_event(&RunEventEnvelope)
|- progress.jsonl + live.json
|- run store
|- SSE
`- CLI / tests / metrics listeners
```
**Three layers:**
The canonical envelope is built exactly once in `fabro-workflow/src/event.rs`.
1. **Rust enum** (`WorkflowRunEvent` in `event.rs`) — the source of truth for event structure. Variants use Rust naming and types. `AgentEvent` and `SandboxEvent` from `fabro-agent` are wrapped as `Agent { stage, event }` and `Sandbox { event }`.
- `WorkflowRunEvent` remains the internal typed source of truth.
- `EventEmitter` owns an immutable `run_id` and converts typed events into `RunEventEnvelope`.
- Every listener receives `&RunEventEnvelope`, not `&WorkflowRunEvent`.
- Bypass paths that cannot go through the emitter must call `canonicalize_event()` once and reuse the same envelope for every sink.
2. **Flattening** (`flatten_event()` in `event.rs`) — serializes the enum via serde, then restructures nested/tagged variants into `(event_name, flat_fields_map)`. Wrapper variants use dot notation: `Agent.ToolCallStarted`, `Sandbox.Initializing`.
## Canonical Envelope
3. **Field renaming** (`rename_fields()` in `event.rs`) — post-processes the flat fields to give them self-describing names for JSONL output. This avoids changing the Rust enum while making the external format unambiguous.
## JSONL Envelope
Every line in `progress.jsonl` has three envelope fields, then the event's own fields merged at the top level:
Each line in `progress.jsonl` is a `RunEventEnvelope`:
```json
{"ts":"2025-06-15T12:00:00.123Z","run_id":"01J...","event":"StageCompleted","node_id":"plan","node_label":"Plan","stage_index":0,"duration_ms":5000,"status":"success",...}
```
| Envelope field | Type | Description |
|---|---|---|
| `ts` | ISO 8601 string | UTC timestamp with millisecond precision |
| `run_id` | string | ULID for this workflow run |
| `event` | string | Event name (matches Rust variant, dot-separated for wrapped types) |
The envelope is built in `fabro-workflow/src/event.rs` by `build_event_envelope()`, and file logging is handled by `ProgressLogger`. Field names from the event that collide with envelope keys (`ts`, `run_id`, `event`) are dropped — the `run_id` from `WorkflowRunStarted` populates the envelope itself.
## Run Completion Contract
`status.json` is the authoritative completion signal for detached runs. Write a terminal `RunStatus` only after all post-run work is complete, including retro generation, finalize-commit work, pull request creation, and sandbox cleanup. `conclusion.json` should be written at that same final point so `attach`, `wait`, and summary rendering all agree on when the run is actually finished.
## Node Terminology
- **`node_id`** — programmatic identifier (the id from the DOT graph). Stable, used for matching.
- **`node_label`** — display name (from the DOT `label` attribute, defaults to `node_id`). Human-readable.
Every event that references a graph node should include both. Stage events carry both from the Rust enum. Events that only have an id (Agent, ParallelBranch, etc.) get `node_label` defaulted to `node_id` by `rename_fields()`.
## Field Naming Conventions
### Rules
1. **Self-describing** — a field name should be unambiguous without knowing the event type. Use `node_id` not `name`, `stage_index` not `index`, `sandbox_provider` not `provider`.
2. **`_id` suffix** for identifiers — `node_id`, `from_node_id`, `to_node_id`, `start_node_id`, `tool_call_id`, `agent_id`.
3. **`_ms` suffix** for durations — `duration_ms`, `delay_ms`. Always milliseconds.
4. **`_count` suffix** for counts — `branch_count`, `command_count`, `tool_call_count`.
5. **No prefix** for fields that are already unambiguous — `error`, `status`, `command`, `question`, `answer`, `model`.
6. **`snake_case`** for all field names.
### Rename Table (Rust enum → JSONL)
The Rust enum uses short field names for ergonomics. `rename_fields()` transforms them for the JSONL output:
| Rust field | JSONL field | Events | Reason |
|---|---|---|---|
| `name` | `workflow_name` | WorkflowRunStarted | Disambiguate |
| `name` | `node_label` | Stage* | Display name |
| `name` | `snapshot_name` | Sandbox.Snapshot* | Disambiguate |
| `index` | `stage_index` | Stage* | Disambiguate |
| `index` | `branch_index` | ParallelBranch* | Disambiguate |
| `index` | `command_index` | SetupCommand*, SetupFailed, DevcontainerLifecycleCommand*, DevcontainerLifecycleFailed | Disambiguate |
| `stage` | `node_id` | Agent.*, Interview*, Prompt | Unify terminology |
| `branch` | `node_id` | ParallelBranch* | Consistent |
| `node` | `node_id` | StallWatchdogTimeout | Consistent |
| `from_node` | `from_node_id` | EdgeSelected, LoopRestart | `_id` suffix |
| `to_node` | `to_node_id` | EdgeSelected, LoopRestart | `_id` suffix |
| `start_node` | `start_node_id` | SubgraphStarted | `_id` suffix |
| `provider` | `sandbox_provider` | Sandbox.* | Disambiguate |
| `text` | `prompt_text` | Prompt | Disambiguate |
| _(inserted)_ | `node_label` | Agent.*, ParallelBranch*, etc. | Defaults to `node_id` |
Fields not in this table pass through unchanged.
## Adding a New Event
### Step 1: Add to the Rust enum
Add a variant to `WorkflowRunEvent` in `event.rs`. Use the short Rust field names (they'll be renamed in step 3).
```rust
MyNewEvent {
node_id: String, // if it references a graph node
name: String, // if it has a display label (will become node_label)
duration_ms: u64,
// ...
},
```
For events that wrap `AgentEvent` or `SandboxEvent`, add the variant to those enums in `fabro-agent` instead — they're automatically wrapped by the existing `Agent { stage, event }` and `Sandbox { event }` variants.
### Step 2: Add a trace() match arm
Add a match arm in `WorkflowRunEvent::trace()`. Choose the tracing level per `logging-strategy.md`:
- INFO for lifecycle boundaries (started/completed at the workflow level)
- DEBUG for individual steps (stage started, tool call, etc.)
- WARN for retries and degraded behavior
- ERROR for terminal failures
```rust
Self::MyNewEvent { node_id, duration_ms, .. } => {
debug!(node_id, duration_ms, "My new event happened");
{
"id": "01960d0c-5d16-7d6e-8f61-9fd6f4a532b5",
"ts": "2026-03-30T12:00:01.000Z",
"run_id": "01JQ...",
"event": "agent.tool.started",
"session_id": "ses_child",
"parent_session_id": "ses_parent",
"node_id": "code",
"node_label": "Code",
"properties": {
"tool_name": "read_file",
"tool_call_id": "call_1",
"arguments": {"path": "src/main.rs"}
}
}
```
### Step 3: Add rename rules (if needed)
Always-present fields:
If your event has fields that need renaming (ambiguous `name`, `index`, `stage`, etc.), add a branch in `rename_fields()` in `event.rs`. If your event references a graph node and only has `node_id`, call `default_node_label(fields)` to insert `node_label`.
### Step 4: Emit from engine or handler
Emit via the `EventEmitter`:
```rust
self.services.emitter.emit(&WorkflowRunEvent::MyNewEvent {
node_id: node.id.clone(),
name: node.label().to_string(),
duration_ms: elapsed,
});
```
### Step 5: Update format_event_summary
Add a match arm in `format_event_summary()` in `cli/mod.rs` for `-v` verbose output:
```rust
WorkflowRunEvent::MyNewEvent { node_id, duration_ms, .. } => {
format!("[MY_NEW_EVENT] node_id={node_id} duration={duration_ms}ms")
}
```
### Step 6: Update tests
- Add a serialization test in `event.rs` (serde round-trip)
- Add a `rename_fields` test if you added rename rules
- Update integration test patterns in `integration.rs` if matching on the new event
## Complete Event Reference
### Workflow lifecycle
| Event | JSONL fields |
|---|---|
| `WorkflowRunStarted` | `workflow_name`, `run_id`, `base_branch`?, `base_sha`?, `run_branch`?, `worktree_dir`? |
| `WorkflowRunCompleted` | `duration_ms`, `artifact_count`, `total_cost`?, `final_git_commit_sha`? |
| `WorkflowRunFailed` | `error`, `duration_ms`, `git_commit_sha`? |
| `RunNotice` | `level`, `code`, `message` |
### Stage execution
| Event | JSONL fields |
|---|---|
| `StageStarted` | `node_id`, `node_label`, `stage_index`, `handler_type`?, `attempt`, `max_attempts` |
| `StageCompleted` | `node_id`, `node_label`, `stage_index`, `duration_ms`, `status`, `preferred_label`?, `suggested_next_ids`, `usage`?, `failure_reason`?, `notes`?, `files_touched`, `attempt`, `max_attempts`, `failure_class`? |
| `StageFailed` | `node_id`, `node_label`, `stage_index`, `error`, `will_retry`, `failure_reason`?, `failure_class`? |
| `StageRetrying` | `node_id`, `node_label`, `stage_index`, `attempt`, `max_attempts`, `delay_ms` |
### Parallel execution
| Event | JSONL fields |
|---|---|
| `ParallelStarted` | `branch_count`, `join_policy` |
| `ParallelBranchStarted` | `node_id`, `node_label`, `branch_index` |
| `ParallelBranchCompleted` | `node_id`, `node_label`, `branch_index`, `duration_ms`, `status` |
| `ParallelCompleted` | `duration_ms`, `success_count`, `failure_count` |
### Graph navigation
| Event | JSONL fields |
|---|---|
| `EdgeSelected` | `from_node_id`, `to_node_id`, `label`?, `condition`? |
| `LoopRestart` | `from_node_id`, `to_node_id` |
| `SubgraphStarted` | `node_id`, `node_label`, `start_node_id` |
| `SubgraphCompleted` | `node_id`, `node_label`, `steps_executed`, `status`, `duration_ms` |
### Checkpoints and git
| Event | JSONL fields |
|---|---|
| `CheckpointCompleted` | `node_id`, `node_label`, `status`, `git_commit_sha` (optional) |
| `CheckpointFailed` | `node_id`, `node_label`, `error` |
| `GitCommit` | `node_id` (optional), `node_label` (optional), `sha` |
| `GitPush` | `branch`, `success` |
| `GitBranch` | `branch`, `sha` |
| `GitWorktreeAdd` | `path`, `branch` |
| `GitWorktreeRemove` | `path` |
| `GitFetch` | `branch`, `success` |
| `GitReset` | `sha` |
### Human interaction
| Event | JSONL fields |
|---|---|
| `InterviewStarted` | `question`, `node_id`, `node_label`, `question_type` |
| `InterviewCompleted` | `question`, `answer`, `duration_ms` |
| `InterviewTimeout` | `question`, `node_id`, `node_label`, `duration_ms` |
| `Prompt` | `node_id`, `node_label`, `prompt_text` |
### Setup
| Event | JSONL fields |
|---|---|
| `SetupStarted` | `command_count` |
| `SetupCommandStarted` | `command`, `command_index` |
| `SetupCommandCompleted` | `command`, `command_index`, `exit_code`, `duration_ms` |
| `SetupCompleted` | `duration_ms` |
| `SetupFailed` | `command`, `command_index`, `exit_code`, `stderr` |
### Devcontainer
| Event | JSONL fields |
|---|---|
| `DevcontainerResolved` | `dockerfile_lines`, `environment_count`, `lifecycle_command_count`, `workspace_folder` |
| `DevcontainerLifecycleStarted` | `phase`, `command_count` |
| `DevcontainerLifecycleCommandStarted` | `phase`, `command`, `command_index` |
| `DevcontainerLifecycleCommandCompleted` | `phase`, `command`, `command_index`, `exit_code`, `duration_ms` |
| `DevcontainerLifecycleCompleted` | `phase`, `duration_ms` |
| `DevcontainerLifecycleFailed` | `phase`, `command`, `command_index`, `exit_code`, `stderr` |
### Stall detection
| Event | JSONL fields |
|---|---|
| `StallWatchdogTimeout` | `node_id`, `node_label`, `idle_seconds` |
### Agent events (prefixed `Agent.`)
All agent events include `node_id` and `node_label`.
| Event | Additional JSONL fields |
|---|---|
| `Agent.SessionStarted` | _(none)_ |
| `Agent.SessionEnded` | _(none)_ |
| `Agent.UserInput` | `text` |
| `Agent.AssistantTextStart` | _(none)_ |
| `Agent.AssistantMessage` | `text`, `model`, `usage` (object), `tool_call_count` |
| `Agent.TextDelta` | `delta` |
| `Agent.ToolCallStarted` | `tool_name`, `tool_call_id`, `arguments` |
| `Agent.ToolCallOutputDelta` | `delta` |
| `Agent.ToolCallCompleted` | `tool_name`, `tool_call_id`, `output`, `is_error` |
| `Agent.Error` | `error` |
| `Agent.Warning` | `kind`, `message`, `details` |
| `Agent.LoopDetected` | _(none)_ |
| `Agent.TurnLimitReached` | `max_turns` |
| `Agent.SkillExpanded` | `skill_name` |
| `Agent.SteeringInjected` | `text` |
| `Agent.CompactionStarted` | `estimated_tokens`, `context_window_size` |
| `Agent.CompactionCompleted` | `original_turn_count`, `preserved_turn_count`, `summary_token_estimate`, `tracked_file_count` |
| `Agent.LlmRetry` | `provider`, `model`, `attempt`, `delay_secs`, `error` |
| `Agent.SubAgentSpawned` | `agent_id`, `depth`, `task` |
| `Agent.SubAgentCompleted` | `agent_id`, `depth`, `success`, `turns_used` |
| `Agent.SubAgentFailed` | `agent_id`, `depth`, `error` |
| `Agent.SubAgentClosed` | `agent_id`, `depth` |
| `Agent.SubAgentEvent.*` | `agent_id`, `depth`, `nested_event` (JSON) |
| `Agent.McpServerReady` | `server_name`, `tool_count` |
| `Agent.McpServerFailed` | `server_name`, `error` |
### Sandbox events (prefixed `Sandbox.`)
| Event | JSONL fields |
|---|---|
| `Sandbox.Initializing` | `sandbox_provider` |
| `Sandbox.Ready` | `sandbox_provider`, `duration_ms` |
| `Sandbox.InitializeFailed` | `sandbox_provider`, `error`, `duration_ms` |
| `Sandbox.CleanupStarted` | `sandbox_provider` |
| `Sandbox.CleanupCompleted` | `sandbox_provider`, `duration_ms` |
| `Sandbox.CleanupFailed` | `sandbox_provider`, `error` |
| `Sandbox.SnapshotPulling` | `snapshot_name` |
| `Sandbox.SnapshotPulled` | `snapshot_name`, `duration_ms` |
| `Sandbox.SnapshotEnsuring` | `snapshot_name` |
| `Sandbox.SnapshotCreating` | `snapshot_name` |
| `Sandbox.SnapshotReady` | `snapshot_name`, `duration_ms` |
| `Sandbox.SnapshotFailed` | `snapshot_name`, `error` |
| `Sandbox.GitCloneStarted` | `url`, `branch` |
| `Sandbox.GitCloneCompleted` | `url`, `duration_ms` |
| `Sandbox.GitCloneFailed` | `url`, `error` |
## Error Fields
Error information is stored as plain strings. The `error` field contains the human-readable message; `failure_class` contains the machine-readable classification.
| Field | Type | Events | Purpose |
|---|---|---|---|
| `error` | string | StageFailed, WorkflowRunFailed, Agent.Error, Agent.LlmRetry, Sandbox.*Failed, etc. | Human-readable error message |
| `failure_reason` | string? | StageFailed, StageCompleted | Outcome-level failure description |
| `failure_class` | string? | StageFailed, StageCompleted | Machine classification: `transient_infra`, `deterministic`, `budget_exhausted`, `compilation_loop`, `canceled`, `structural` |
`failure_class` is derived from `FabroError::failure_class()` for handler errors, or from handler hints in `context_updates["failure_class"]` for outcome-based failures. See `error.rs` for the classification logic.
## Consumers
| Consumer | Reads | Purpose |
| Field | Type | Notes |
|---|---|---|
| `retro.rs` `extract_stage_durations()` | `node_label`, `duration_ms` from `StageCompleted` | Build retro report |
| `cli/run.rs` non-verbose listener | `name`, `duration_ms`, `status`, `usage` from `StageCompleted/Failed` | CLI progress output |
| `cli/mod.rs` `format_event_summary()` | All events | `-v` verbose output |
| `cli/run.rs` cost accumulator | `usage` from `StageCompleted` | Total cost tracking |
| `cli/run.rs` git SHA tracker | `git_commit_sha` from `CheckpointCompleted` | Final SHA for `conclusion.json` |
| External tooling | `progress.jsonl` | Live monitoring, dashboards |
| `id` | string | UUIDv7 event id |
| `ts` | string | UTC timestamp with millisecond precision |
| `run_id` | string | Workflow run id |
| `event` | string | Lowercase dot-notation event name |
Optional top-level fields:
| Field | When present |
|---|---|
| `session_id` | Agent/session events |
| `parent_session_id` | Forwarded child-session events |
| `node_id` | Events tied to a graph node or branch |
| `node_label` | Display label for `node_id`; omitted when not applicable |
Everything else lives inside `properties`.
Important rules:
- Optional envelope fields are omitted, not serialized as `null`.
- Event-specific fields do not get flattened into the top level.
- `EventPayload` validation requires `id`, `ts`, `run_id`, and `event`.
## Naming
The external event name is lowercase dot notation, for example:
- `run.started`
- `stage.completed`
- `agent.tool.started`
- `sandbox.ready`
- `parallel.branch.completed`
`event_name()` in `event.rs` is exhaustive. Do not use wildcard fallthroughs when adding new variants.
## Node And Session Metadata
`node_id` is the stable graph identifier. `node_label` is the human-facing display name. Stage events should surface both through the envelope when applicable.
Agent events now use explicit session links:
- `session_id` identifies the session that originally emitted the event.
- `parent_session_id` identifies the immediate parent session for forwarded child events.
- Nested sub-agents preserve immediate parentage across boundaries.
`AgentEvent::SubAgentEvent` no longer exists. Child activity is forwarded as normal agent events with session linkage in the envelope.
## Direct-Write Paths
Most events flow through `EventEmitter::emit()`. The remaining direct-write paths must use:
1. `canonicalize_event(run_id, event)`
2. Serialize and redact once
3. Reuse that exact serialized envelope for every sink
Never canonicalize the same logical event twice if multiple sinks receive it.
## Adding A New Event
### 1. Add the typed event
Add a variant to `WorkflowRunEvent`, `AgentEvent`, or `SandboxEvent` as appropriate.
### 2. Add tracing
Extend `WorkflowRunEvent::trace()` so the raw event is observable in tracing output.
### 3. Add an external name
Extend `event_name()` with the new lowercase dot-notation string.
### 4. Map envelope fields
Update `extract_envelope_fields()`:
- Move `node_id`, `node_label`, `session_id`, and `parent_session_id` into the envelope when appropriate.
- Keep event-specific data in `properties`.
- Flatten structured failure details into explicit property keys when needed.
### 5. Emit it
Prefer `EventEmitter::emit(&WorkflowRunEvent::...)`.
Use `canonicalize_event()` only for true bypass paths.
### 6. Update consumers
Check:
- CLI progress parsing
- `fabro logs`
- retro duration extraction
- store validation
- tests or fixtures that inspect event names or fields
## Consumer Guidance
When writing listeners:
- Match on `envelope.event`, not Rust variant names.
- Read event payload from `envelope.properties`.
- Read stage/branch identity from `node_id` and `node_label`.
- Read agent hierarchy from `session_id` and `parent_session_id`.
Do not rebuild or mutate the envelope in downstream listeners.
## Bypass And Persistence Guarantees
`progress.jsonl`, the run store, and SSE should reflect the same canonical envelope bytes after redaction.
`status.json` remains the authoritative completion signal for detached runs. Terminal run status should only be written after all post-run work is finished.

2
docs-internal/product/current-state.md
View file

@ -7,7 +7,7 @@ This snapshot is intentionally brief and omits volatile counts, benchmarks, and
Fabro currently presents as:
- a CLI for defining and running workflows
- an API server via `fabro serve`
- an API server via `fabro server start`
- a React web app for monitoring runs and inspecting workflows
- a docs site and example workflows

2
docs-internal/product/personas.md
View file

@ -6,7 +6,7 @@ Fabro is for expert engineers who want to encode software processes as workflows
- **Solo expert engineer**: runs workflows locally to automate planning, implementation, testing, and review loops.
- **Tech lead or platform engineer**: defines shared workflows, model routing rules, and quality gates for a team.
- **Infrastructure engineer**: deploys `fabro serve`, configures sandboxes and auth, and keeps the system reliable.
- **Infrastructure engineer**: deploys `fabro server start`, configures sandboxes and auth, and keeps the system reliable.
## Good fits

8
docs/administration/deploy-server.mdx
View file

@ -7,7 +7,7 @@ description: "Run Fabro as an API server with a web UI, concurrent runs, and tea
Server mode is in private early access. Contact [bryan@qlty.sh](mailto:bryan@qlty.sh) if you're interested in trying it.
</Warning>
Fabro has two modes: **standalone** and **server**. Standalone mode (`fabro run`) executes a single workflow synchronously in your terminal. Server mode (`fabro serve`) starts an HTTP API that queues runs, streams events, and serves a web UI — so you can close your laptop and let workflows run.
Fabro has two modes: **standalone** and **server**. Standalone mode (`fabro run`) executes a single workflow synchronously in your terminal. Server mode (`fabro server start`) starts an HTTP API that queues runs, streams events, and serves a web UI — so you can close your laptop and let workflows run.
Both modes use the same workflow engine, the same Graphviz files, and the same sandbox providers. The difference is how you interact with them.
@ -15,7 +15,7 @@ Both modes use the same workflow engine, the same Graphviz files, and the same s
| | Standalone | Server |
|---|---|---|
| **Command** | `fabro run workflow.fabro` | `fabro serve` |
| **Command** | `fabro run workflow.fabro` | `fabro server start` |
| **Best for** | Local development, one-off runs, CI/CD | Production, team use, running at scale |
| **Execution** | Synchronous, one run per process | Asynchronous, queued with configurable concurrency |
| **Human-in-the-loop** | Terminal prompts | Web UI or HTTP endpoints |
@ -27,13 +27,13 @@ Both modes use the same workflow engine, the same Graphviz files, and the same s
## Starting the server
```bash
fabro serve
fabro server start
```
This starts the API on `127.0.0.1:3000` by default. To also run the web UI:
```bash
fabro serve # API on port 3000
fabro server start # API on port 3000
cd apps/fabro-web && bun run dev # Web UI on port 5173
```

6
docs/administration/server-configuration.mdx
View file

@ -5,7 +5,7 @@ description: "Server config file, CLI overrides, and environment variables"
## Config file
The server config file at `~/.fabro/server.toml` controls how `fabro serve` behaves — API binding, authentication, run defaults, and more. The [Quick Start](/getting-started/quick-start) doesn't require one, but production deployments should configure it explicitly.
The server config file at `~/.fabro/server.toml` controls how `fabro server start` behaves — API binding, authentication, run defaults, and more. The [Quick Start](/getting-started/quick-start) doesn't require one, but production deployments should configure it explicitly.
### Full reference
@ -83,7 +83,7 @@ default_branch = "main"
### CLI overrides
Several `server.toml` settings can be overridden via `fabro serve` flags:
Several `server.toml` settings can be overridden via `fabro server start` flags:
| Flag | Default | Description |
|---|---|---|
@ -123,7 +123,7 @@ The CLI can also set `[git.author]` in `user.toml` to override the server defaul
### `[git.webhooks]` section
Enable automatic GitHub webhook delivery via Tailscale funnel. When configured, `fabro serve` binds a local HTTP listener, exposes it through `tailscale funnel`, and updates the GitHub App's webhook URL on startup. Incoming webhooks are verified with HMAC-SHA256.
Enable automatic GitHub webhook delivery via Tailscale funnel. When configured, `fabro server start` binds a local HTTP listener, exposes it through `tailscale funnel`, and updates the GitHub App's webhook URL on startup. Incoming webhooks are verified with HMAC-SHA256.
| Key | Description | Values |
|---|---|---|

2
docs/administration/troubleshooting.mdx
View file

@ -28,7 +28,7 @@ It checks:
**Sandbox creation failures** — For Docker: ensure the Docker daemon is running and the configured image exists. For Daytona: verify `DAYTONA_API_KEY` is set and the `gh` CLI is authenticated. For Exe: verify your SSH keys are configured for `exe.dev` and that `ssh exe.dev` connects successfully.
**Port already in use** — Change the port with `fabro serve --port 3001` or stop the conflicting process.
**Port already in use** — Change the port with `fabro server start --port 3001` or stop the conflicting process.
**SSE streams disconnecting** — If using a reverse proxy, ensure buffering is disabled and the connection timeout is long enough for workflow runs. See the [reverse proxy example](/administration/deployment#binding-and-tls).

140
docs/agents/subagents.mdx
View file

@ -3,7 +3,7 @@ title: "Sub-agents"
description: "Delegate subtasks to child agent sessions"
---
An agent can spawn **sub-agents** to delegate work to independent child sessions. Each sub-agent gets its own LLM session and tool access, runs concurrently with the parent, and returns its result when finished. This is useful for parallelizing research, isolating risky operations, or breaking complex tasks into focused pieces.
An agent can spawn **sub-agents** to delegate work to independent child sessions. Each sub-agent gets its own LLM session and tool access, runs concurrently with the parent, and returns its result when finished.
<Note>
Sub-agents are only available with the [API backend](/core-concepts/agents#api-backend-default) (the default). Agents using the [CLI backend](/core-concepts/agents#cli-backend) cannot spawn sub-agents.
@ -16,124 +16,84 @@ Sub-agent management is exposed through four built-in tools:
| Tool | Description |
|---|---|
| `spawn_agent` | Create a new sub-agent with a task prompt |
| `send_input` | Send a follow-up message to a running sub-agent |
| `send_input` | Send follow-up input to a running sub-agent |
| `wait` | Block until a sub-agent completes and return its result |
| `close_agent` | Cancel and remove a running sub-agent |
These tools are registered automatically when the session starts. They inherit the parent's permissions -- no additional approval is needed for sub-agent tool calls.
### spawn_agent
Creates a new sub-agent session and starts it working on the given task.
| Parameter | Type | Required | Description |
|---|---|---|---|
| `task` | string | yes | The task description for the sub-agent |
| `working_dir` | string | no | Working directory for the sub-agent |
| `model` | string | no | Model to use for the sub-agent |
| `max_turns` | integer | no | Maximum number of turns (default: unlimited) |
Returns the `agent_id` string used to reference this sub-agent in subsequent calls.
### send_input
Sends a follow-up message to a running sub-agent. The message is queued and processed by the sub-agent on its next turn.
| Parameter | Type | Required | Description |
|---|---|---|---|
| `agent_id` | string | yes | The ID returned by `spawn_agent` |
| `message` | string | yes | The message to send |
### wait
Blocks until the specified sub-agent completes, then returns its result. The result includes whether the sub-agent succeeded, how many turns it used, and the final text output.
| Parameter | Type | Required | Description |
|---|---|---|---|
| `agent_id` | string | yes | The ID returned by `spawn_agent` |
Returns a formatted string:
```
Agent completed (success: true, turns: 12)
<final assistant message text>
```
### close_agent
Cancels a running sub-agent and removes it. Use this to clean up sub-agents that are no longer needed.
| Parameter | Type | Required | Description |
|---|---|---|---|
| `agent_id` | string | yes | The ID returned by `spawn_agent` |
These tools are registered automatically when the session starts. They inherit the parent's permissions.
## Session isolation
Each sub-agent runs in its own independent session:
Each sub-agent runs in its own session:
- **Own LLM conversation** -- the sub-agent has a fresh conversation history starting from the task prompt. It does not see the parent's prior conversation.
- **Own tool access** -- the sub-agent has the same tools as the parent (shell, file editing, grep, etc.) and can use them independently.
- **Concurrent execution** -- after `spawn_agent`, the parent continues working immediately. The sub-agent runs in a background task. Use `wait` to collect the result when needed.
- **Own conversation** -- the child starts with a fresh LLM history
- **Own tool access** -- the child can use the same tools as the parent
- **Concurrent execution** -- the parent can continue working before calling `wait`
The parent can spawn multiple sub-agents and let them work in parallel, then `wait` on each to collect results:
```
1. spawn_agent("Research authentication patterns") -> agent_a
2. spawn_agent("Analyze the test suite structure") -> agent_b
3. ... parent does its own work ...
4. wait(agent_a) -> result_a
5. wait(agent_b) -> result_b
6. Synthesize both results
```
The parent can spawn multiple sub-agents and synchronize with them later.
## Depth limits
Sub-agents can themselves spawn sub-agents, creating a hierarchy. The **maximum nesting depth** prevents runaway chains. By default, the depth limit is **1** -- a parent agent can spawn sub-agents, but those sub-agents cannot spawn their own children.
Sub-agents can themselves spawn sub-agents, creating a hierarchy. `max_subagent_depth` limits how deep that tree can grow. By default the depth limit is `1`.
If a sub-agent tries to spawn beyond the depth limit, the `spawn_agent` call returns an error:
```
Maximum subagent depth (1) reached
```
The depth limit is set via the `max_subagent_depth` field in the session configuration.
If a child tries to exceed the limit, `spawn_agent` returns an error immediately.
## Error handling
When a sub-agent fails, the error is captured and returned to the parent via the `wait` tool — the parent stage does **not** automatically fail. The parent LLM sees the error message and decides how to respond: retry, try a different approach, or report the failure.
Sub-agent failures do not automatically fail the parent stage. The parent receives the failure through `wait` and decides how to respond.
Specific failure modes:
Common cases:
- **Sub-agent hits `max_turns`** — The sub-agent stops naturally and returns its last output as a successful result. The parent sees a normal completion with the final assistant message. By default, there is no turn limit — sub-agents run until they complete their task or hit the context window.
- **Sub-agent panics or errors** — The error is captured and returned through `wait` as a failure result. The parent can inspect the error and decide what to do.
- **`spawn_agent` fails** (e.g. depth limit exceeded) — The error is returned immediately as a tool result. The parent can adjust its approach without waiting.
- **Hits `max_turns`** -- returns normally with its last output
- **Panics or errors** -- returned as a failed `wait` result
- **`spawn_agent` fails** -- returned immediately as a tool result
## Event forwarding
Sub-agent events (tool calls, assistant messages, errors) are forwarded to the parent session's event stream as `SubAgentEvent` wrappers. This means the parent's progress log captures the full activity of all children, giving you visibility into what sub-agents are doing.
Sub-agent observability now uses **session linkage**, not wrapper events.
Key lifecycle events:
Parent-owned lifecycle events:
| Event | When |
|---|---|
| `SubAgentSpawned` | A sub-agent was created |
| `SubAgentCompleted` | A sub-agent finished successfully |
| `SubAgentFailed` | A sub-agent encountered an error |
| `SubAgentClosed` | A sub-agent was cancelled |
| `agent.sub.spawned` | A sub-agent was created |
| `agent.sub.completed` | A sub-agent finished successfully |
| `agent.sub.failed` | A sub-agent failed |
| `agent.sub.closed` | A sub-agent was cancelled |
Streaming events (text deltas, tool output deltas) from sub-agents are filtered out to reduce noise.
Forwarded child activity:
- Child tool calls, assistant messages, warnings, and other non-noisy session events are forwarded into the parent run's event stream as normal agent events.
- The forwarded event keeps the child's `session_id`.
- `parent_session_id` points to the child's immediate parent session.
Example envelope:
```json
{
"event": "agent.tool.started",
"session_id": "ses_child",
"parent_session_id": "ses_parent",
"node_id": "code",
"properties": {
"tool_name": "read_file"
}
}
```
Nested sub-agents preserve the immediate parent-child relationship. A grandchild forwarded through multiple parents still carries its own `session_id`, and `parent_session_id` remains set to the grandchild's direct parent rather than the root.
High-volume streaming events such as text deltas and tool output deltas are filtered out of the forwarded stream to reduce noise.
## When to use sub-agents
Sub-agents are most useful when:
Sub-agents are most useful for:
- **Parallel research** -- gather information from multiple parts of a codebase simultaneously
- **Isolation** -- let a sub-agent try a risky approach without polluting the parent's context
- **Divide and conquer** -- break a large task into independent pieces that can be worked on concurrently
- **Context management** -- offload work to a sub-agent when the parent's context window is getting full
- **Parallel research** -- inspect multiple code paths at once
- **Isolation** -- try a risky approach without polluting the parent's context
- **Divide and conquer** -- split an independent task into smaller pieces
- **Context management** -- offload work when the parent session is getting crowded
<Note>
Sub-agents run with no turn limit by default — they continue until the task is complete or the context window is exhausted. Pass `max_turns` when spawning to cap execution for predictable cost control. The parent agent is blocked during a `wait` call, so spawn sub-agents before starting the wait to maximize concurrency.
Sub-agents run with no turn limit by default. Pass `max_turns` when you want predictable cost or time bounds. All active sub-agents are cleaned up automatically when the parent session closes.
</Note>
All active sub-agents are automatically cleaned up when the parent session closes — you don't need to explicitly `close_agent` on every sub-agent.

2
docs/api-reference/overview.mdx
View file

@ -11,7 +11,7 @@ The Fabro API is a REST API for managing workflow runs, interactive sessions, an
## Base URL
The versioned API is served by `fabro serve`, which defaults to:
The versioned API is served by `fabro server start`, which defaults to:
```
http://localhost:3000/api/v1

2
docs/core-concepts/how-fabro-works.mdx
View file

@ -14,7 +14,7 @@ Fabro is a workflow engine that reads a graph definition, executes nodes one at
Fabro has two interfaces, both backed by the same workflow engine:
- **Standalone mode** (`fabro run`) — Run a single workflow synchronously in your terminal. Best for local development, one-off runs, and CI/CD.
- **Server mode** (`fabro serve`) — Start an HTTP API server with a web UI, concurrent run scheduling, and team access. Best for production use and running at scale.
- **Server mode** (`fabro server start`) — Start an HTTP API server with a web UI, concurrent run scheduling, and team access. Best for production use and running at scale.
Both modes parse the same Graphviz files, use the same execution engine, and support the same sandbox providers. See [Server Mode](/administration/deploy-server) for a detailed comparison and setup guide, or [Architecture](/reference/architecture) for internals.

1
docs/docs.json
View file

@ -103,6 +103,7 @@
"pages": [
"reference/dot-language",
"reference/cli",
"reference/shell-completions",
"reference/user-configuration",
"reference/run-directory",
"reference/sdk",

288
docs/execution/observability.mdx
View file

@ -3,171 +3,119 @@ title: "Observability"
description: "How to monitor, inspect, and analyze workflow runs"
---
Fabro captures a structured event for every significant action during a workflow run — stage starts and completions, agent tool calls, edge selections, retries, failovers, sandbox lifecycle, and more. These events power real-time monitoring, post-run analysis, and cross-run analytics.
Fabro captures a structured event for every significant action during a workflow run. These events cover stage execution, agent tool calls, retries, routing, sandbox lifecycle, git checkpoints, retro generation, and more.
## Event stream
The event stream is the foundation of Fabro's observability. Every workflow run emits a sequence of `WorkflowRunEvent` records that are:
Every workflow run emits a sequence of canonical **run event envelopes** that are:
- **Written to `progress.jsonl`** in the run's directory (one JSON object per line)
- **Broadcast via SSE** to connected API clients in real time
- **Logged via `tracing`** to the daily log file at `~/.fabro/logs/`
- Written to `progress.jsonl` in the run directory
- Broadcast over SSE to connected API clients
- Stored for later analysis and retro generation
- Rendered by CLI progress and log tooling
### Event types
### Event names
Events fall into several categories:
Event names use lowercase dot notation, for example:
**Run lifecycle** — bookend events for the entire run:
- `run.started`
- `stage.started`
- `stage.completed`
- `agent.tool.started`
- `agent.tool.completed`
- `sandbox.ready`
- `parallel.branch.completed`
| Event | Key fields | Description |
|---|---|---|
| `WorkflowRunStarted` | `name`, `run_id`, `base_sha`, `run_branch`, `goal` | Run begins |
| `WorkflowRunCompleted` | `duration_ms`, `artifact_count`, `total_cost`, `status`, `usage` | Run finishes successfully |
| `WorkflowRunFailed` | `error`, `duration_ms` | Run terminates with an error |
### Envelope format
**Stage lifecycle** — events for each node execution:
| Event | Key fields | Description |
|---|---|---|
| `StageStarted` | `node_id`, `name`, `handler_type`, `attempt`, `max_attempts` | Node begins executing |
| `StageCompleted` | `node_id`, `duration_ms`, `status`, `usage`, `files_touched` | Node finishes |
| `StageFailed` | `node_id`, `failure`, `will_retry` | Node fails (may or may not retry) |
| `StageRetrying` | `node_id`, `attempt`, `max_attempts`, `delay_ms` | Retry scheduled after failure |
**Agent activity** — forwarded from the LLM agent session, prefixed with `Agent.`:
| Event | Key fields | Description |
|---|---|---|
| `Agent.SessionStarted` | `stage` | Agent session begins |
| `Agent.ToolCallStarted` | `stage`, `tool_name`, `arguments` | Agent invokes a tool |
| `Agent.ToolCallCompleted` | `stage`, `tool_name`, `output`, `is_error` | Tool returns a result |
| `Agent.AssistantMessage` | `stage`, `text`, `model`, `usage` | LLM responds with text |
| `Agent.Error` | `stage`, `error` | Agent-level error |
| `Agent.LoopDetected` | `stage` | Repeated tool call pattern detected |
| `Agent.SteeringInjected` | `stage`, `text` | Human steering message injected |
| `Agent.Warning` | `stage`, `kind`, `message`, `details` | Non-fatal warning (e.g. context window usage) |
| `Agent.CompactionStarted` | `stage`, `estimated_tokens`, `context_window_size` | Context compaction triggered |
| `Agent.CompactionCompleted` | `stage`, `original_turn_count`, `preserved_turn_count`, `summary_token_estimate`, `tracked_file_count` | Context compaction finished |
| `Agent.LlmRetry` | `stage`, `provider`, `model`, `attempt`, `delay_secs` | LLM API call retried |
| `Agent.SubAgentSpawned` | `stage`, `agent_id`, `task` | Sub-agent launched |
| `Agent.SubAgentCompleted` | `stage`, `agent_id`, `success`, `turns_used` | Sub-agent finished |
**Routing and control flow:**
| Event | Key fields | Description |
|---|---|---|
| `EdgeSelected` | `from_node`, `to_node`, `label`, `condition`, `reason`, `stage_status` | Transition between nodes |
| `LoopRestart` | `from_node`, `to_node` | Loop restart edge taken |
| `CheckpointCompleted` | `node_id`, `git_commit_sha` (optional) | Checkpoint saved (with git SHA when git is enabled) |
| `GitCommit` | `node_id`, `sha` | Git commit created |
| `GitPush` | `branch`, `success` | Git push attempted |
| `GitBranch` | `branch`, `sha` | Git branch created |
| `GitWorktreeAdd` | `path`, `branch` | Git worktree added |
| `GitWorktreeRemove` | `path` | Git worktree removed |
| `GitFetch` | `branch`, `success` | Git fetch attempted |
| `GitReset` | `sha` | Git reset executed |
| `Failover` | `stage`, `from_provider`, `to_provider`, `error` | LLM provider failover |
| `RetroStarted` | — | Retrospective generation begins |
| `RetroCompleted` | `duration_ms` | Retrospective generation finished |
| `RetroFailed` | `error`, `duration_ms` | Retrospective generation failed |
**Parallel execution:**
| Event | Key fields | Description |
|---|---|---|
| `ParallelStarted` | `branch_count`, `join_policy` | Fan-out begins |
| `ParallelBranchStarted` | `branch`, `index` | Individual branch begins |
| `ParallelBranchCompleted` | `branch`, `duration_ms`, `status` | Branch finishes |
| `ParallelCompleted` | `duration_ms`, `success_count`, `failure_count` | All branches done |
**Human-in-the-loop:**
| Event | Key fields | Description |
|---|---|---|
| `InterviewStarted` | `question`, `stage`, `question_type` | Human input requested |
| `InterviewCompleted` | `question`, `answer`, `duration_ms` | Human responded |
| `InterviewTimeout` | `question`, `stage`, `duration_ms` | Human didn't respond in time |
**Sandbox and setup:**
| Event | Key fields | Description |
|---|---|---|
| `Sandbox.Initializing` | `provider` | Sandbox creation started |
| `Sandbox.Ready` | `provider`, `duration_ms`, `cpu`, `memory` | Sandbox ready |
| `SetupStarted` | `command_count` | Setup commands beginning |
| `SetupCommandCompleted` | `command`, `exit_code`, `duration_ms` | Single setup command finished |
| `SetupFailed` | `command`, `exit_code`, `stderr` | Setup command failed |
| `SshAccessReady` | `ssh_command` | SSH connection command printed |
| `StallWatchdogTimeout` | `node`, `idle_seconds` | No activity for too long |
### Event envelope format
Each line in `progress.jsonl` is a JSON object with a standard envelope:
Each line in `progress.jsonl` is a JSON object with a stable envelope:
```json
{
"ts": "2026-03-05T14:30:01.234Z",
"run_id": "01JKXYZ...",
"event": "Agent.ToolCallStarted",
"stage": "implement",
"tool_name": "shell",
"arguments": {"command": "cargo test"}
"id": "01960d0c-5d16-7d6e-8f61-9fd6f4a532b5",
"ts": "2026-03-30T12:00:01.000Z",
"run_id": "01JQ...",
"event": "agent.tool.started",
"session_id": "ses_child",
"parent_session_id": "ses_parent",
"node_id": "implement",
"node_label": "Implement",
"properties": {
"tool_name": "shell",
"tool_call_id": "call_1",
"arguments": {"command": "cargo test"}
}
}
```
The `ts`, `run_id`, and `event` fields are always present. The remaining fields vary by event type. Events are flattened — nested variants like agent and sandbox events use dot notation (e.g. `Agent.ToolCallStarted`, `Sandbox.Ready`).
Envelope fields:
## Log files
| Field | Description |
|---|---|
| `id` | Unique event id |
| `ts` | UTC timestamp |
| `run_id` | Workflow run id |
| `event` | Event name |
| `session_id` | Session that emitted the event, when applicable |
| `parent_session_id` | Immediate parent session for forwarded child events |
| `node_id` | Node or branch id, when applicable |
| `node_label` | Human-facing label for `node_id`, when applicable |
| `properties` | Event-specific payload |
Fabro writes two kinds of logs:
Only `id`, `ts`, `run_id`, and `event` are always present. Optional fields are omitted when they do not apply.
### Run logs (`progress.jsonl`)
## Reading `progress.jsonl`
Every run writes its event stream to `{run_dir}/progress.jsonl`. This is the primary data source for post-run analysis — [retros](/execution/retros) read it, and you can query it directly with standard tools:
Because event payload lives in `properties`, most shell queries should look there.
```bash
# Count tool calls in a run
grep "ToolCallStarted" ~/.fabro/runs/01JKXYZ.../progress.jsonl | wc -l
jq -r 'select(.event == "agent.tool.started") | .properties.tool_name' \
~/.fabro/runs/01JKXYZ.../progress.jsonl | wc -l
# Find all failures
grep -E "StageFailed|WorkflowRunFailed" ~/.fabro/runs/01JKXYZ.../progress.jsonl | jq .
# Find stage failures
jq 'select(.event == "stage.failed")' \
~/.fabro/runs/01JKXYZ.../progress.jsonl
# See which edges were taken
grep "EdgeSelected" ~/.fabro/runs/01JKXYZ.../progress.jsonl | jq '{from: .from_node, to: .to_node}'
jq '{from: .properties.from_node, to: .properties.to_node, label: .properties.label}' \
~/.fabro/runs/01JKXYZ.../progress.jsonl | head
```
### Live snapshot (`live.json`)
`live.json` is still a pretty-printed copy of the most recent event envelope.
During execution, Fabro also writes `live.json` — a pretty-printed copy of the most recent event. This is useful for quick status checks while a run is in progress:
## Event categories
```bash
cat ~/.fabro/runs/01JKXYZ.../live.json
```
Common categories include:
### Application logs
Fabro uses the `tracing` crate to write structured logs to `~/.fabro/logs/YYYY-MM-DD.log`. Control the log level with the `FABRO_LOG` environment variable:
```bash
FABRO_LOG=debug fabro run workflow.fabro
```
| Level | What's logged |
| Category | Example events |
|---|---|
| `error` | Run failures, stage failures that won't retry |
| `warn` | Retries, timeouts, interview timeouts, failovers, early terminations |
| `info` | Run start/complete, SSH access ready |
| `debug` | Stage start/complete, edge selections, checkpoints, parallel branches, tool calls |
| Run lifecycle | `run.started`, `run.completed`, `run.failed`, `run.notice` |
| Stage lifecycle | `stage.started`, `stage.completed`, `stage.failed`, `stage.retrying` |
| Agent activity | `agent.message`, `agent.tool.started`, `agent.warning`, `agent.sub.spawned` |
| Routing | `edge.selected`, `loop.restart`, `parallel.started` |
| Git and checkpoints | `checkpoint.completed`, `git.commit`, `git.push` |
| Setup and sandbox | `sandbox.initializing`, `sandbox.ready`, `setup.started` |
| Retro | `retro.started`, `retro.completed`, `retro.failed` |
## Sub-agent visibility
Sub-agent activity now appears as normal agent events with session linkage:
- `session_id` identifies the child session
- `parent_session_id` identifies its immediate parent
Lifecycle events such as `agent.sub.spawned` and `agent.sub.completed` are emitted by the parent session. Tool calls and other child activity are forwarded with their original `session_id`.
## Real-time monitoring
### API: Server-Sent Events
When running workflows through the API server, subscribe to a live event stream via the [run events endpoint](/api-reference/runs/stream-run-events). Each event is a JSON-serialized `WorkflowRunEvent`. The stream stays open until the run completes.
When running workflows through the API server, subscribe to the [run events endpoint](/api-reference/runs/stream-run-events). Each SSE payload is a serialized run event envelope in the same shape used by `progress.jsonl`.
### Web UI
The web frontend connects to the SSE stream automatically and displays run progress in real time — stage transitions, agent tool calls, and human gate prompts are all visible as they happen.
The web frontend consumes the SSE stream automatically and shows stage progress, tool calls, and human interaction as they happen.
<Frame caption="The Stages tab shows the full agent conversation including tool calls and responses.">
<img src="/images/web/run-stages.png" alt="Fabro web UI run stages showing agent conversation with tool calls" />
@ -175,90 +123,20 @@ The web frontend connects to the SSE stream automatically and displays run progr
### CLI progress
The CLI displays a live progress bar during execution with per-stage status, duration, and cost tracking. This is rendered to stderr so it doesn't interfere with output piping.
The CLI renders live progress from the same envelope format. This is written to stderr so stdout remains pipe-friendly.
## Post-run analysis
### Listing runs
Browse your run history with `fabro ps`:
```bash
fabro ps
fabro ps --workflow PlanImplement
fabro ps --before 2026-03-01
fabro ps --label team=platform
fabro ps --json
```
This scans `~/.fabro/runs/` and displays each run's ID, workflow name, status, and start time. Use `--json` for machine-readable output.
### Run artifacts
Each run's directory contains a standard set of files:
Run artifacts still include:
| File | Description |
|---|---|
| `run.json` | Run record — ID, config, graph, workflow slug, labels |
| `start.json` | Start record — run ID, start time, run branch, base SHA |
| `progress.jsonl` | Full event stream |
| `live.json` | Last event snapshot (overwritten during run) |
| `run.json` | Run metadata and graph |
| `start.json` | Start record |
| `progress.jsonl` | Full event envelope stream |
| `live.json` | Latest event envelope |
| `checkpoint.json` | Final execution state |
| `retro.json` | Retrospective (if retro generation is enabled) |
| `conclusion.json` | Terminal status (`completed`, `failed`, `canceled`) |
| `retro.json` | Retrospective, when enabled |
| `conclusion.json` | Terminal summary |
### Inspecting stages and turns
The API provides endpoints for drilling into individual stages and the agent turns within them. See the [stages](/api-reference/run-internals/list-run-stages) and [turns](/api-reference/run-internals/list-stage-turns) API reference pages.
## Insights (SQL analytics)
The Insights feature lets you run SQL queries across your run data using DuckDB. This is useful for aggregate analysis — finding slow workflows, tracking failure rates, comparing model costs, and spotting trends.
Insights is managed through the [saved queries](/api-reference/insights/list-saved-queries) and [execute query](/api-reference/insights/execute-query) API reference pages. You can save, update, and execute queries programmatically.
### Example queries
**Average run duration by workflow:**
```sql
SELECT workflow_name, AVG(duration_seconds) as avg_duration,
COUNT(*) as run_count
FROM runs
GROUP BY workflow_name
ORDER BY avg_duration DESC
LIMIT 20
```
**Daily failure rate:**
```sql
SELECT date_trunc('day', created_at) as day,
COUNT(*) FILTER (WHERE status = 'failed') as failures,
COUNT(*) as total
FROM runs
GROUP BY 1
ORDER BY 1 DESC
LIMIT 30
```
**Top repositories by activity:**
```sql
SELECT repo, COUNT(*) as runs
FROM runs
GROUP BY repo
ORDER BY runs DESC
```
## Aggregate usage
The API server tracks aggregate usage counters across all runs — total run count, total runtime, and per-model breakdowns of token usage and cost. See the [usage endpoint](/api-reference/usage/aggregate-usage) in the API reference. Counters reset on server restart.
<Frame caption="The Usage tab breaks down token counts and costs by stage and by model.">
<img src="/images/web/run-usage.png" alt="Fabro web UI run usage showing per-stage and per-model token and cost breakdown" />
</Frame>
## Credential redaction
All event output — `progress.jsonl`, SSE streams, and log files — is automatically redacted before being written. Fabro detects and replaces patterns that look like API keys, AWS credentials, bearer tokens, and other secrets with `REDACTED`. This ensures sensitive values that appear in tool call arguments or command output are never persisted to disk.
See [retros](/execution/retros), [stages](/api-reference/run-internals/list-run-stages), and [turns](/api-reference/run-internals/list-stage-turns) for higher-level analysis views built on top of this event stream.

2
docs/execution/run-configuration.mdx
View file

@ -423,7 +423,7 @@ Project defaults are merged with run config values using the same rules as serve
### Server defaults
When running via `fabro serve`, the server config at `~/.fabro/server.toml` can set default values for `[llm]`, `[setup]`, `[sandbox]`, and `[vars]`. These defaults are applied to every run unless the run config overrides them.
When running via `fabro server start`, the server config at `~/.fabro/server.toml` can set default values for `[llm]`, `[setup]`, `[sandbox]`, and `[vars]`. These defaults are applied to every run unless the run config overrides them.
For variables, defaults and run config are **merged** — the run config wins on key collisions:

2
docs/getting-started/quick-start.mdx
View file

@ -34,7 +34,7 @@ Fabro has two modes:
```bash
cd my-repo/
fabro init
fabro repo init
```
This creates a default workflow and configuration in your project directory.

2
docs/integrations/daytona.mdx
View file

@ -156,7 +156,7 @@ auto_stop_interval = 30
## Server defaults
When running via `fabro serve`, the server config at `~/.fabro/server.toml` can set default Daytona settings for all runs. Run config TOML values override server defaults. Labels are **merged** — run config labels win on key collisions. The `network` setting uses simple override (run config replaces the server default entirely).
When running via `fabro server start`, the server config at `~/.fabro/server.toml` can set default Daytona settings for all runs. Run config TOML values override server defaults. Labels are **merged** — run config labels win on key collisions. The `network` setting uses simple override (run config replaces the server default entirely).
See [Server Configuration](/administration/server-configuration) for details.

8
docs/reference/architecture.mdx
View file

@ -25,10 +25,10 @@ CLI mode is ideal for:
## API mode
```bash
fabro serve
fabro server start
```
`fabro serve` starts an HTTP server (default `127.0.0.1:3000`) backed by SQLite for run persistence. Runs are submitted via the REST API and executed asynchronously.
`fabro server start` starts an HTTP server (default `127.0.0.1:3000`) backed by SQLite for run persistence. Runs are submitted via the REST API and executed asynchronously.
### Configuration
@ -72,10 +72,10 @@ Demo mode is per-request: send the `X-Fabro-Demo: 1` HTTP header to get static m
## Web UI
The web UI is a React app (`apps/fabro-web`) that connects to the API server. Start it alongside `fabro serve`:
The web UI is a React app (`apps/fabro-web`) that connects to the API server. Start it alongside `fabro server start`:
```bash
fabro serve # API on port 3000
fabro server start # API on port 3000
cd apps/fabro-web && bun run dev # Web UI on port 5173
```

30
docs/reference/cli.mdx
View file

@ -327,14 +327,14 @@ fabro model test -m claude-sonnet-4-5
---
## `fabro serve`
## `fabro server start`
Start the HTTP API server that exposes the [REST API](/api-reference) for launching and managing workflow runs.
```bash
fabro serve
fabro serve --port 8080 --host 0.0.0.0
fabro serve --sandbox daytona --max-concurrent-runs 4
fabro server start
fabro server start --port 8080 --host 0.0.0.0
fabro server start --sandbox daytona --max-concurrent-runs 4
```
| Flag | Description | Default |
@ -648,10 +648,6 @@ The command must be run inside a git repository. It creates:
After creating files, it checks whether the GitHub App is installed for the repository. If the app is not installed and the repository owner differs from the app owner, it warns that the app may need to be [made public](/integrations/github#github-app-is-private-but-this-repo-belongs-to-a-different-owner) first.
<Note>
The old `fabro init` still works but prints a deprecation warning. Use `fabro repo init` instead.
</Note>
## `fabro repo deinit`
Remove Fabro from a project by deleting `fabro.toml` and the `fabro/` directory. Fails with an error if the project is not initialized.
@ -888,6 +884,24 @@ fabro store dump abc123 -o ./debug-output
---
## `fabro completion`
Generate shell completion scripts for bash, zsh, fish, elvish, and PowerShell.
```bash
fabro completion bash
fabro completion zsh
fabro completion fish
```
| Argument | Description |
|---|---|
| `<SHELL>` | Shell to generate completions for: `bash`, `zsh`, `fish`, `elvish`, `powershell` (required) |
See [Shell Completions](/reference/shell-completions) for installation instructions for each shell.
---
## `fabro docs`
Open the Fabro documentation website in your default browser.

70
docs/reference/shell-completions.mdx Normal file
View file

@ -0,0 +1,70 @@
---
title: "Shell Completions"
description: "Set up tab completion for the fabro CLI in your shell"
---
The `fabro completion` command generates shell completion scripts for tab-completing commands, flags, and arguments.
## Bash
Add to your `~/.bashrc`:
```bash
eval "$(fabro completion bash)"
```
Or generate a file and source it:
```bash
fabro completion bash > ~/.local/share/bash-completion/completions/fabro
```
## Zsh
Add to your `~/.zshrc` (before `compinit`):
```bash
eval "$(fabro completion zsh)"
```
Or generate a file:
```bash
fabro completion zsh > "${fpath[1]}/_fabro"
```
You may need to run `compinit` or start a new shell session for changes to take effect.
## Fish
```bash
fabro completion fish | source
```
Or persist to the completions directory:
```bash
fabro completion fish > ~/.config/fish/completions/fabro.fish
```
## PowerShell
Add to your PowerShell profile:
```powershell
fabro completion powershell | Out-String | Invoke-Expression
```
## Elvish
```bash
eval (fabro completion elvish | slurp)
```
## Supported shells
Run `fabro completion --help` to see all supported shells:
```bash
fabro completion --help
```

318
files-internal/testing-strategy.md Normal file
View file

@ -0,0 +1,318 @@
# Testing Strategy
This document defines the default testing rules for this repository, with extra emphasis on CLI integration tests.
The goal is to make the correct test shape obvious:
- put each test in the right layer
- create state through public interfaces
- prefer stable black-box assertions
- avoid brittle tests that mirror implementation details
## Core principles
- Test the public contract of the layer you are in.
- Prefer command-driven or API-driven setup over manually fabricating internal state.
- Prefer snapshots over ad hoc string matching.
- Prefer structured snapshots over parsing JSON and checking one field.
- If a test is only practical by writing internal runtime files directly, it probably belongs in a lower-level test.
## Test layers
Use the narrowest layer that can express the behavior cleanly.
### Unit and crate-level integration tests
Use unit tests or crate-local integration tests when the behavior under test is implementation-facing rather than CLI-facing.
This is the right place for:
- helper logic
- parsing and normalization
- rendering internals
- interview file claim/response mechanics
- retry bookkeeping
- asset manifest parsing
- event formatting
If the setup requires direct writes to internal run files or runtime directories, prefer this layer over `fabro-cli/tests/it`.
### `lib/crates/fabro-cli/tests/it/cmd/*.rs`
`cmd/*` tests are command-owned tests.
Each file should focus on one command's public contract. Setup may use other commands for convenience, but the final assertion should still be about the command under test.
Examples:
- `cmd/run.rs` tests `fabro run`
- `cmd/create.rs` tests `fabro create`
- `cmd/start.rs` tests `fabro start`
- `cmd/attach.rs` tests `fabro attach`
Good command-test assertions:
- help and clap behavior
- required-argument failures
- command-owned persisted state
- command-owned selection or lookup behavior
- user-visible output and lifecycle behavior owned by that command
Bad command-test assertions:
- long multi-command narratives where no single command is the subject
- behavior primarily owned by another command
- runtime internals that only exist because the test planted them by hand
### `lib/crates/fabro-cli/tests/it/workflow/*.rs`
`workflow/*` tests are black-box workflow-behavior tests.
Use this layer when the workflow content is the thing under test, even if the harness command is `fabro run`.
Examples:
- branching behavior
- conditional routing
- parallel execution shape
- representative fixture workflows
These tests should focus on the workflow's observed behavior, not on CLI help text or command argument validation.
### `lib/crates/fabro-cli/tests/it/scenario/*.rs`
`scenario/*` tests are cross-command lifecycle tests.
Use this layer when the point of the test is the interaction among commands or command families.
Examples:
- create -> start -> attach flows
- detached run -> attach flows
- rewind / fork recovery flows
- lookup behavior that spans several commands
Scenario tests are allowed to be broader, but they should still stay command-driven and black-box.
## Placement rules
When choosing where a test belongs, ask: "What is the main contract I am trying to prove?"
- If the answer is a single command, use `cmd/*`.
- If the answer is a workflow fixture or workflow shape, use `workflow/*`.
- If the answer is a multi-command flow, use `scenario/*`.
- If the answer is an implementation detail, use a lower-level test near the code.
If a test starts in `cmd/*` and grows into a workflow or lifecycle narrative, move it.
## State setup rules
Integration tests should create state through public interfaces.
Allowed setup:
- checked-in workflow fixtures
- temp `.fabro` workflow files
- temp `workflow.toml` and `fabro.toml`
- temp git repositories
- temp user config and environment variables
- invoking commands to create runs, checkpoints, branches, and persisted state
Disallowed setup in `fabro-cli/tests/it`:
- writing `run.json` directly
- writing `status.json` directly
- writing `progress.jsonl` directly
- writing `conclusion.json` directly
- writing runtime interview files directly
- writing cached workflow files into run dirs directly
- writing asset manifests directly
- planting files into run internals solely to simulate engine output
The rule is simple: do not hand-author run-directory internals in CLI integration tests.
### Exceptions
Exceptions should be rare.
Only keep a direct internal-state setup when all of the following are true:
- the behavior cannot be reproduced through public commands at reasonable cost
- the behavior is still best validated at the CLI integration layer
- the test clearly documents why the exception exists
- no cleaner lower-level test would cover the behavior better
If those conditions are not met, move the test down a layer.
## Assertion rules
Default to snapshot-first assertions.
### Use transcript snapshots for CLI behavior
Use `fabro_snapshot!` for:
- `--help`
- clap errors
- normal CLI stderr/stdout transcripts
- detached/start/attach lifecycle output
Do not replace full-output snapshots with a handful of `contains()` checks unless the output is intentionally partial and a full snapshot would be noisy or unstable.
### Use structured snapshots for persisted state
When verifying JSON or JSONL:
1. parse it
2. normalize or compact it if needed
3. snapshot the parsed structure
Use `insta` directly or shared helpers such as `fabro_json_snapshot!`.
Good structured snapshot targets:
- `run.json`
- `status.json`
- `inspect` output
- `live.json`
- compacted `progress.jsonl` event sequences
- workflow conclusions and checkpoint summaries
### Keep direct assertions for relational invariants
Use direct assertions when the point is an exact relationship rather than a representation.
Examples:
- exact selected run id
- equality before and after a rejected mutation attempt
- `live.json` equals the last progress event
- exact SHA lineage across rewind/fork
- exact file existence semantics
## Snapshot rules
- Prefer inline snapshots unless the payload is too large to read comfortably.
- Normalize unstable values: timestamps, durations, ULIDs, temp paths, storage paths, run dirs, and SHAs.
- Never accept snapshot churn blindly.
- Review pending snapshots before accepting them.
For CLI snapshot updates:
1. run `cargo insta pending-snapshots`
2. inspect each pending change
3. accept only the intended updates
## Helpers and fixtures
Use the test helpers that reinforce the rules above.
### `TestContext`
Use `TestContext` for CLI integration tests so each test gets isolated home, storage, and temp directories.
Prefer helpers like:
- `context.command()`
- `context.run_cmd()`
- `context.find_run_dir(...)`
- `context.single_run_dir()`
### Shared `tests/it/support`
Shared integration-test helpers may:
- locate fixtures
- read and parse JSON / JSONL
- normalize output
- compact structured events
- poll for stable command-created conditions
Shared integration-test helpers should not:
- fabricate run internals
- write runtime files the engine is supposed to own
- hide broad scenario setup behind opaque helper functions
### Fixtures
Prefer checked-in fixtures when they express a reusable workflow or scenario shape.
Use temporary inline fixtures when the test needs a small one-off input and a checked-in fixture would add noise.
Keep fixtures user-facing:
- workflow sources
- config files
- repo contents
Do not turn fixtures into prebuilt run directories.
## Determinism rules
Tests should be stable on any developer machine.
- Use fixed run ids when practical.
- Prefer dry-run where it still exercises the intended public behavior.
- Use local temp directories, never ambient user state.
- Mark tests that require real providers, real sandboxes, or external services with `#[ignore]` and a clear reason.
- Filter or normalize machine-specific output in snapshots.
If a test depends on `.env` or real credentials, it must be clearly marked and opt-in.
## Naming rules
Name tests after the contract they prove.
Prefer:
- `start_by_workflow_name_prefers_newly_created_submitted_run`
- `detached_uses_cached_graph_after_source_deleted`
- `attach_requires_run_arg`
Avoid:
- `bug4_test`
- `regression_123`
- names that describe setup rather than behavior
If a test exists because of a regression, mention the bug number in a comment or commit message, not in the primary test name unless the bug id is itself part of the contract.
## Review checklist
Before merging a test change, check:
- Is the test in the correct layer: unit, `cmd`, `workflow`, or `scenario`?
- Is the state created through public commands or public inputs?
- Does the test avoid hand-writing run-directory internals?
- Does the assertion use snapshots where snapshots are the better tool?
- Is JSON / JSONL asserted structurally rather than via substring matching?
- Are unstable values normalized?
- Is the test name describing the contract?
- Would a lower-level test be cleaner and less brittle?
## Anti-patterns
Avoid these patterns in CLI integration tests:
- manually creating fake run directories
- writing `progress.jsonl` lines by hand
- writing runtime interview files by hand
- writing asset manifests by hand
- scattering the same workflow setup across many files instead of using fixtures
- asserting one field from a parsed JSON payload when the full structure is the behavior
- using many `contains()` checks for output that should be snapshot-tested
- keeping scenario tests in `cmd/*`
- keeping workflow-shape tests in `cmd/*`
## Defaults for new tests
When adding a new CLI integration test, the default choice should be:
1. decide the layer (`cmd`, `workflow`, `scenario`, or lower-level test)
2. create input state through public files and commands
3. use `TestContext`
4. assert with snapshots
5. keep the test focused on one contract
If you need to break one of these defaults, document why in the test itself.

36
lib/crates/fabro-agent/src/cli.rs
View file

@ -486,7 +486,7 @@ pub async fn run_with_args_and_client(
manager_for_callback
.lock()
.await
.set_event_callback(session.event_callback());
.set_event_callback(session.sub_agent_event_callback());
// SIGINT handler
let cancel_token = session.cancel_token();
@ -522,6 +522,11 @@ pub async fn run_with_args_and_client(
OutputFormat::Text => {
let s = styles;
while let Ok(event) = rx.recv().await {
let child_prefix = if event.parent_session_id.is_some() {
format!("[child {}] ", event.session_id)
} else {
String::new()
};
match &event.event {
AgentEvent::ToolCallStarted {
tool_name,
@ -531,7 +536,7 @@ pub async fn run_with_args_and_client(
eprintln!(
" {} {}{}",
s.dim.apply_to("\u{25cf}"),
s.bold_cyan.apply_to(tool_name),
s.bold_cyan.apply_to(format!("{child_prefix}{tool_name}")),
s.dim.apply_to(format!(
"({})",
format_tool_args(arguments, &cwd_str)
@ -551,13 +556,17 @@ pub async fn run_with_args_and_client(
};
eprintln!(
" {}\n{}",
s.dim.apply_to(format!("[{label}] {tool_name}:")),
s.dim
.apply_to(format!("[{label}] {child_prefix}{tool_name}:")),
serde_json::to_string_pretty(output)
.unwrap_or_else(|_| output.to_string()),
);
}
AgentEvent::Error { error } => {
eprintln!(" {}", s.red.apply_to(format!("\u{2717} {error}")),);
eprintln!(
" {}",
s.red.apply_to(format!("\u{2717} {child_prefix}{error}")),
);
}
AgentEvent::SubAgentSpawned {
agent_id,
@ -573,7 +582,7 @@ pub async fn run_with_args_and_client(
eprintln!(
" {}",
s.dim.apply_to(format!(
"\u{25b6} subagent {agent_id} spawned (depth={depth}) task={task_preview:?}"
"{child_prefix}\u{25b6} subagent {agent_id} spawned (depth={depth}) task={task_preview:?}"
)),
);
}
@ -586,7 +595,7 @@ pub async fn run_with_args_and_client(
eprintln!(
" {}",
s.dim.apply_to(format!(
"\u{25a0} subagent {agent_id} completed (depth={depth}, success={success}, turns={turns_used})"
"{child_prefix}\u{25a0} subagent {agent_id} completed (depth={depth}, success={success}, turns={turns_used})"
)),
);
}
@ -598,7 +607,7 @@ pub async fn run_with_args_and_client(
eprintln!(
" {}",
s.red.apply_to(format!(
"\u{2717} subagent {agent_id} failed (depth={depth}): {error}"
"{child_prefix}\u{2717} subagent {agent_id} failed (depth={depth}): {error}"
)),
);
}
@ -606,21 +615,10 @@ pub async fn run_with_args_and_client(
eprintln!(
" {}",
s.dim.apply_to(format!(
"\u{25a0} subagent {agent_id} closed (depth={depth})"
"{child_prefix}\u{25a0} subagent {agent_id} closed (depth={depth})"
)),
);
}
AgentEvent::SubAgentEvent {
agent_id,
event: child_event,
..
} if verbose => {
eprintln!(
" {}",
s.dim
.apply_to(format!("[subagent {agent_id}] {child_event:?}")),
);
}
_ => {}
}
}

27
lib/crates/fabro-agent/src/event.rs
View file

@ -20,11 +20,16 @@ impl EventEmitter {
event,
timestamp: SystemTime::now(),
session_id,
parent_session_id: None,
};
// Ignore send error (no receivers)
let _ = self.sender.send(wrapped);
}
pub fn forward(&self, event: SessionEvent) {
let _ = self.sender.send(event);
}
#[must_use]
pub fn subscribe(&self) -> broadcast::Receiver<SessionEvent> {
self.sender.subscribe()
@ -52,6 +57,7 @@ mod tests {
let event = receiver.recv().await.unwrap();
assert!(matches!(event.event, AgentEvent::SessionStarted));
assert_eq!(event.session_id, "sess-1");
assert_eq!(event.parent_session_id, None);
}
#[tokio::test]
@ -70,6 +76,7 @@ mod tests {
assert!(
matches!(&event.event, AgentEvent::Error { error } if error.to_string().contains("something went wrong"))
);
assert_eq!(event.parent_session_id, None);
}
#[tokio::test]
@ -86,6 +93,8 @@ mod tests {
assert!(matches!(e2.event, AgentEvent::SessionEnded));
assert_eq!(e1.session_id, "sess-3");
assert_eq!(e2.session_id, "sess-3");
assert_eq!(e1.parent_session_id, None);
assert_eq!(e2.parent_session_id, None);
}
#[test]
@ -104,4 +113,22 @@ mod tests {
let emitter = EventEmitter::default();
let _rx = emitter.subscribe();
}
#[tokio::test]
async fn forward_preserves_session_ids() {
let emitter = EventEmitter::new();
let mut receiver = emitter.subscribe();
emitter.forward(SessionEvent {
event: AgentEvent::SessionStarted,
timestamp: SystemTime::now(),
session_id: "child".into(),
parent_session_id: Some("parent".into()),
});
let event = receiver.recv().await.unwrap();
assert_eq!(event.session_id, "child");
assert_eq!(event.parent_session_id.as_deref(), Some("parent"));
assert!(matches!(event.event, AgentEvent::SessionStarted));
}
}

28
lib/crates/fabro-agent/src/session.rs
View file

@ -13,7 +13,7 @@ use crate::sandbox::Sandbox;
use crate::skills::{
ExpandedInput, Skill, default_skill_dirs, discover_skills, expand_skill, make_use_skill_tool,
};
use crate::subagent::{SubAgentEventCallback, SubAgentManager};
use crate::subagent::{SubAgentCallbackEvent, SubAgentEventCallback, SubAgentManager};
use crate::tool_execution::execute_tool_calls;
use crate::types::{AgentEvent, SessionEvent, SessionState, Turn};
use fabro_llm::client::Client;
@ -93,6 +93,11 @@ impl Session {
self.tool_env = Some(env);
}
#[must_use]
pub fn id(&self) -> &str {
&self.id
}
/// Initialize session by discovering project docs and capturing environment context.
/// Call before `process_input`.
pub async fn initialize(&mut self) {
@ -453,13 +458,22 @@ impl Session {
self.cancel_token.clone()
}
/// Build a callback that forwards `AgentEvent`s through this session's emitter.
/// Build a callback that forwards sub-agent lifecycle and child session events
/// through this session's emitter.
#[must_use]
pub fn event_callback(&self) -> SubAgentEventCallback {
pub fn sub_agent_event_callback(&self) -> SubAgentEventCallback {
let emitter = self.event_emitter.clone();
let session_id = self.id.clone();
Arc::new(move |event| {
emitter.emit(session_id.clone(), event);
let parent_session_id = self.id.clone();
Arc::new(move |event| match event {
SubAgentCallbackEvent::Lifecycle(event) => {
emitter.emit(parent_session_id.clone(), event);
}
SubAgentCallbackEvent::Forwarded(mut event) => {
if event.parent_session_id.is_none() {
event.parent_session_id = Some(parent_session_id.clone());
}
emitter.forward(event);
}
})
}
@ -2720,7 +2734,7 @@ mod tests {
manager
.lock()
.await
.set_event_callback(session.event_callback());
.set_event_callback(session.sub_agent_event_callback());
// Spawn a subagent
let child = make_session(vec![text_response("child done")]).await;

79
lib/crates/fabro-agent/src/subagent.rs
View file

@ -2,7 +2,7 @@ use crate::error::AgentError;
use crate::session::Session;
use crate::tool_registry::RegisteredTool;
use crate::tools::required_str;
use crate::types::{AgentEvent, Turn};
use crate::types::{AgentEvent, SessionEvent, Turn};
use fabro_llm::types::ToolDefinition;
use std::collections::{HashMap, VecDeque};
use std::sync::{Arc, Mutex};
@ -11,7 +11,14 @@ use tokio::task::JoinHandle;
use tokio_util::sync::CancellationToken;
pub type SessionFactory = Arc<dyn Fn() -> Session + Send + Sync>;
pub type SubAgentEventCallback = Arc<dyn Fn(AgentEvent) + Send + Sync>;
#[derive(Debug, Clone)]
pub enum SubAgentCallbackEvent {
Lifecycle(AgentEvent),
Forwarded(SessionEvent),
}
pub type SubAgentEventCallback = Arc<dyn Fn(SubAgentCallbackEvent) + Send + Sync>;
#[derive(Debug, Clone)]
pub struct SubAgentResult {
@ -57,7 +64,7 @@ impl SubAgentManager {
fn emit_event(&self, event: AgentEvent) {
if let Some(ref cb) = self.event_callback {
cb(event);
cb(SubAgentCallbackEvent::Lifecycle(event));
}
}
@ -82,8 +89,6 @@ impl SubAgentManager {
if let Some(ref cb) = self.event_callback {
let mut rx = session.subscribe();
let cb = cb.clone();
let fwd_agent_id = agent_id.clone();
let child_depth = depth + 1;
tokio::spawn(async move {
while let Ok(event) = rx.recv().await {
// Skip streaming / noise events
@ -101,11 +106,7 @@ impl SubAgentManager {
) {
continue;
}
cb(AgentEvent::SubAgentEvent {
agent_id: fwd_agent_id.clone(),
depth: child_depth,
event: Box::new(event.event),
});
cb(SubAgentCallbackEvent::Forwarded(event));
}
});
}
@ -620,8 +621,11 @@ mod tests {
assert!(close_required.contains(&serde_json::json!("agent_id")));
}
fn captured_events() -> (SubAgentEventCallback, Arc<Mutex<Vec<AgentEvent>>>) {
let events: Arc<Mutex<Vec<AgentEvent>>> = Arc::new(Mutex::new(Vec::new()));
fn captured_events() -> (
SubAgentEventCallback,
Arc<Mutex<Vec<SubAgentCallbackEvent>>>,
) {
let events: Arc<Mutex<Vec<SubAgentCallbackEvent>>> = Arc::new(Mutex::new(Vec::new()));
let events_clone = events.clone();
let cb: SubAgentEventCallback = Arc::new(move |event| {
events_clone.lock().unwrap().push(event);
@ -640,9 +644,11 @@ mod tests {
let captured = events.lock().unwrap();
assert_eq!(captured.len(), 1);
assert!(
matches!(&captured[0], AgentEvent::SubAgentSpawned { depth: 1, task, .. } if task == "test task")
);
assert!(matches!(
&captured[0],
SubAgentCallbackEvent::Lifecycle(AgentEvent::SubAgentSpawned { depth: 1, task, .. })
if task == "test task"
));
}
#[tokio::test]
@ -658,11 +664,11 @@ mod tests {
let captured = events.lock().unwrap();
assert!(captured.iter().any(|e| matches!(
e,
AgentEvent::SubAgentCompleted {
SubAgentCallbackEvent::Lifecycle(AgentEvent::SubAgentCompleted {
success: true,
depth: 1,
..
}
})
)));
}
@ -677,11 +683,10 @@ mod tests {
manager.close(&agent_id).unwrap();
let captured = events.lock().unwrap();
assert!(
captured
.iter()
.any(|e| matches!(e, AgentEvent::SubAgentClosed { depth: 2, .. }))
);
assert!(captured.iter().any(|e| matches!(
e,
SubAgentCallbackEvent::Lifecycle(AgentEvent::SubAgentClosed { depth: 2, .. })
)));
}
#[tokio::test]
@ -702,15 +707,41 @@ mod tests {
let captured = events.lock().unwrap();
let forwarded_count = captured
.iter()
.filter(|e| matches!(e, AgentEvent::SubAgentEvent { .. }))
.filter(|e| matches!(e, SubAgentCallbackEvent::Forwarded(_)))
.count();
// Child session emits at least UserInput and AssistantMessage (filtered from SessionStarted/SessionEnded/etc)
assert!(
forwarded_count > 0,
"expected at least one forwarded child event, got {forwarded_count}"
);
}
#[tokio::test]
async fn session_callback_stamps_parent_only_once() {
let parent = make_session(vec![text_response("parent")]).await;
let callback = parent.sub_agent_event_callback();
let mut rx = parent.subscribe();
callback(SubAgentCallbackEvent::Forwarded(SessionEvent {
event: AgentEvent::SessionStarted,
timestamp: std::time::SystemTime::now(),
session_id: "child".into(),
parent_session_id: None,
}));
callback(SubAgentCallbackEvent::Forwarded(SessionEvent {
event: AgentEvent::SessionStarted,
timestamp: std::time::SystemTime::now(),
session_id: "grandchild".into(),
parent_session_id: Some("child".into()),
}));
let child = rx.recv().await.unwrap();
let grandchild = rx.recv().await.unwrap();
assert_eq!(child.session_id, "child");
assert_eq!(child.parent_session_id.as_deref(), Some(parent.id()));
assert_eq!(grandchild.session_id, "grandchild");
assert_eq!(grandchild.parent_session_id.as_deref(), Some("child"));
}
#[test]
fn no_callback_does_not_panic() {
// Manager without callback should not panic on emit

63
lib/crates/fabro-agent/src/types.rs
View file

@ -185,11 +185,6 @@ pub enum AgentEvent {
agent_id: String,
depth: usize,
},
SubAgentEvent {
agent_id: String,
depth: usize,
event: Box<Self>,
},
McpServerReady {
server_name: String,
tool_count: usize,
@ -237,8 +232,7 @@ impl AgentEvent {
Self::TextDelta { .. }
| Self::ReasoningDelta { .. }
| Self::AssistantOutputReplace { .. }
| Self::ToolCallOutputDelta { .. }
| Self::SubAgentEvent { .. } => {}
| Self::ToolCallOutputDelta { .. } => {}
Self::ToolCallStarted {
tool_name,
tool_call_id,
@ -392,6 +386,8 @@ pub struct SessionEvent {
#[serde(with = "system_time_iso8601")]
pub timestamp: SystemTime,
pub session_id: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub parent_session_id: Option<String>,
}
#[cfg(test)]
@ -404,9 +400,11 @@ mod tests {
event: AgentEvent::SessionStarted,
timestamp: SystemTime::now(),
session_id: "sess_1".into(),
parent_session_id: None,
};
assert!(matches!(event.event, AgentEvent::SessionStarted));
assert_eq!(event.session_id, "sess_1");
assert_eq!(event.parent_session_id, None);
}
#[test]
@ -498,21 +496,6 @@ mod tests {
assert!(matches!(event, AgentEvent::SubAgentClosed { depth: 2, .. }));
}
#[test]
fn subagent_event_wraps_child_event() {
let child = AgentEvent::ToolCallStarted {
tool_name: "read_file".into(),
tool_call_id: "tc-1".into(),
arguments: serde_json::json!({}),
};
let event = AgentEvent::SubAgentEvent {
agent_id: "sa-1".into(),
depth: 1,
event: Box::new(child),
};
assert!(matches!(event, AgentEvent::SubAgentEvent { depth: 1, .. }));
}
#[test]
fn subagent_events_serde_round_trip() {
let events = vec![
@ -536,39 +519,53 @@ mod tests {
agent_id: "sa-1".into(),
depth: 0,
},
AgentEvent::SubAgentEvent {
agent_id: "sa-1".into(),
depth: 1,
event: Box::new(AgentEvent::SessionStarted),
},
];
let json = serde_json::to_string(&events).unwrap();
let deserialized: Vec<AgentEvent> = serde_json::from_str(&json).unwrap();
assert_eq!(deserialized.len(), 5);
assert!(
matches!(&deserialized[4], AgentEvent::SubAgentEvent { event, .. } if matches!(event.as_ref(), AgentEvent::SessionStarted))
);
assert_eq!(deserialized.len(), 4);
}
#[test]
fn session_event_serde_round_trip() {
fn session_event_serde_round_trip_without_parent_session_id() {
let event = SessionEvent {
event: AgentEvent::SessionStarted,
timestamp: SystemTime::now(),
session_id: "sess_42".into(),
parent_session_id: None,
};
let json = serde_json::to_string(&event).unwrap();
assert!(json.contains("sess_42"));
assert!(json.contains("SessionStarted"));
// Timestamp should be ISO-8601
assert!(!json.contains("parent_session_id"));
assert!(json.contains('T'));
assert!(json.contains('Z'));
let deserialized: SessionEvent = serde_json::from_str(&json).unwrap();
assert_eq!(deserialized.session_id, "sess_42");
assert_eq!(deserialized.parent_session_id, None);
assert!(matches!(deserialized.event, AgentEvent::SessionStarted));
}
#[test]
fn session_event_serde_round_trip_with_parent_session_id() {
let event = SessionEvent {
event: AgentEvent::SessionStarted,
timestamp: SystemTime::now(),
session_id: "sess_child".into(),
parent_session_id: Some("sess_parent".into()),
};
let json = serde_json::to_string(&event).unwrap();
assert!(json.contains("sess_child"));
assert!(json.contains("sess_parent"));
let deserialized: SessionEvent = serde_json::from_str(&json).unwrap();
assert_eq!(deserialized.session_id, "sess_child");
assert_eq!(
deserialized.parent_session_id.as_deref(),
Some("sess_parent")
);
}
#[test]
fn mcp_server_ready_constructible() {
let event = AgentEvent::McpServerReady {

9
lib/crates/fabro-git-storage/Cargo.toml → lib/crates/fabro-checkpoint/Cargo.toml
View file

@ -1,10 +1,10 @@
[package]
name = "fabro-git-storage"
name = "fabro-checkpoint"
edition.workspace = true
version.workspace = true
publish = false
license.workspace = true
description = "Store structured data in git without touching the working directory"
description = "Git-backed checkpoint storage for Fabro workflows"
repository = "https://github.com/brynary/arc"
[lib]
@ -14,10 +14,13 @@ doctest = false
workspace = true
[dependencies]
fabro-types = { path = "../fabro-types" }
git2.workspace = true
serde.workspace = true
serde_json.workspace = true
thiserror.workspace = true
tracing.workspace = true
walkdir.workspace = true
[dev-dependencies]
chrono.workspace = true
tempfile = "3"

56
lib/crates/fabro-checkpoint/src/author.rs Normal file
View file

@ -0,0 +1,56 @@
use std::fmt::Write;
use fabro_types::settings::server::GitAuthorSettings;
/// Resolved git author identity for checkpoint commits.
#[derive(Debug, Clone, PartialEq)]
pub struct GitAuthor {
pub name: String,
pub email: String,
}
impl Default for GitAuthor {
fn default() -> Self {
Self {
name: "Fabro".into(),
email: "noreply@fabro.sh".into(),
}
}
}
impl GitAuthor {
/// Create a `GitAuthor` from optional name/email, falling back to defaults.
pub fn from_options(name: Option<String>, email: Option<String>) -> Self {
let defaults = Self::default();
Self {
name: name.unwrap_or(defaults.name),
email: email.unwrap_or(defaults.email),
}
}
/// Returns true when this identity matches the default Fabro identity.
pub fn is_default(&self) -> bool {
let defaults = Self::default();
self.name == defaults.name && self.email == defaults.email
}
/// Append the Fabro footer (and Co-Authored-By when the author is not the
/// default identity) to a commit message.
pub fn append_footer(&self, message: &mut String) {
message.push_str("\n\u{2692}\u{fe0f} Generated with [Fabro](https://fabro.sh)\n");
if !self.is_default() {
let defaults = Self::default();
let _ = write!(
message,
"\nCo-Authored-By: {} <{}>\n",
defaults.name, defaults.email
);
}
}
}
impl From<&GitAuthorSettings> for GitAuthor {
fn from(value: &GitAuthorSettings) -> Self {
Self::from_options(value.name.clone(), value.email.clone())
}
}

8
lib/crates/fabro-git-storage/src/branchstore.rs → lib/crates/fabro-checkpoint/src/branch.rs
View file

@ -2,7 +2,7 @@ use git2::{Oid, Signature};
use tracing::{debug, warn};
use crate::Result;
use crate::gitobj::{FileMode, Store, TreeEntries};
use crate::git::{FileMode, Store, TreeEntries};
/// Metadata about a commit, returned by `log`.
#[derive(Debug)]
@ -47,7 +47,7 @@ impl<'a> BranchStore<'a> {
self.objects
.write_commit(empty_tree, &[], "initialize branch", &self.author)?;
self.objects.update_ref(&self.branch, commit_oid)?;
debug!(branch = %self.branch, "Created git storage branch");
debug!(branch = %self.branch, "Created checkpoint branch");
Ok(())
}
@ -74,7 +74,7 @@ impl<'a> BranchStore<'a> {
self.objects
.write_commit(new_tree, &[parent_oid], message, &self.author)?;
self.objects.update_ref(&self.branch, commit_oid)?;
debug!(branch = %self.branch, commit = %commit_oid, "Wrote git storage commit");
debug!(branch = %self.branch, commit = %commit_oid, "Wrote checkpoint commit");
Ok(commit_oid)
}
@ -219,7 +219,7 @@ pub fn sharded_path(id: &str, prefix_len: usize) -> String {
#[cfg(test)]
mod tests {
use super::*;
use crate::gitobj::FileMode;
use crate::git::FileMode;
use git2::Repository;
fn temp_repo() -> (tempfile::TempDir, Store) {

14
lib/crates/fabro-git-storage/src/error.rs → lib/crates/fabro-checkpoint/src/error.rs
View file

@ -16,3 +16,17 @@ pub enum Error {
#[error("branch {branch} not found")]
BranchNotFound { branch: String },
}
#[derive(Debug, thiserror::Error)]
pub enum MetadataError {
#[error(transparent)]
Storage(#[from] Error),
#[error("deserialize {entity} on branch {branch}: {source}")]
Deserialize {
entity: &'static str,
branch: String,
#[source]
source: serde_json::Error,
},
}

4
lib/crates/fabro-git-storage/src/gitobj.rs → lib/crates/fabro-checkpoint/src/git.rs
View file

@ -565,7 +565,7 @@ mod tests {
fn read_blob_at_returns_content() {
let (_dir, store) = temp_repo();
let sig = Signature::now("Test", "test@example.com").unwrap();
let bs = crate::branchstore::BranchStore::new(&store, "test/data", &sig);
let bs = crate::branch::BranchStore::new(&store, "test/data", &sig);
bs.ensure_branch().unwrap();
bs.write_entry("hello.txt", b"world", "add hello").unwrap();
@ -580,7 +580,7 @@ mod tests {
fn read_blob_at_returns_none_for_missing_path() {
let (_dir, store) = temp_repo();
let sig = Signature::now("Test", "test@example.com").unwrap();
let bs = crate::branchstore::BranchStore::new(&store, "test/data", &sig);
let bs = crate::branch::BranchStore::new(&store, "test/data", &sig);
bs.ensure_branch().unwrap();
bs.write_entry("hello.txt", b"world", "add hello").unwrap();

10
lib/crates/fabro-checkpoint/src/lib.rs Normal file
View file

@ -0,0 +1,10 @@
pub mod author;
pub mod branch;
pub mod error;
pub mod git;
pub mod metadata;
pub mod trailer;
pub const META_BRANCH_PREFIX: &str = "fabro/meta/";
pub use error::{Error, MetadataError, Result};

436
lib/crates/fabro-checkpoint/src/metadata.rs Normal file
View file

@ -0,0 +1,436 @@
use std::path::{Path, PathBuf};
use fabro_types::{Checkpoint, RunRecord, StartRecord};
use git2::{Repository, Signature};
use crate::META_BRANCH_PREFIX;
use crate::author::GitAuthor;
use crate::branch::BranchStore;
use crate::error::{Error, MetadataError};
use crate::git::Store;
/// Git-native metadata storage for pipeline runs.
///
/// Stores checkpoint data, run records, and metadata on an orphan branch
/// (`fabro/meta/{run_id}`) so that runs can be resumed from git alone.
pub struct MetadataStore {
repo_path: PathBuf,
author: GitAuthor,
}
impl MetadataStore {
pub fn new(repo_path: impl Into<PathBuf>, author: &GitAuthor) -> Self {
Self {
repo_path: repo_path.into(),
author: author.clone(),
}
}
/// Returns the branch name for a run: `fabro/meta/{run_id}`.
pub fn branch_name(run_id: &str) -> String {
format!("{META_BRANCH_PREFIX}{run_id}")
}
/// Format a commit message with the standard Fabro footer appended.
fn commit_message(&self, subject: &str) -> String {
let mut msg = format!("{subject}\n");
self.author.append_footer(&mut msg);
msg
}
fn open_store(&self) -> Result<(Store, Signature<'static>), MetadataError> {
let repo = Repository::discover(&self.repo_path).map_err(Error::from)?;
let store = Store::new(repo);
let sig = Signature::now(&self.author.name, &self.author.email).map_err(Error::from)?;
Ok((store, sig))
}
/// Initialize a run's metadata branch with the given files.
///
/// Callers pass all files (run.json, start.json, sandbox.json, etc.)
/// via the `files` slice.
pub fn init_run(&self, run_id: &str, files: &[(&str, &[u8])]) -> Result<(), MetadataError> {
let (store, sig) = self.open_store()?;
let branch = Self::branch_name(run_id);
let branch_store = BranchStore::new(&store, &branch, &sig);
branch_store.ensure_branch()?;
let message = self.commit_message("init run");
branch_store.write_entries(files, &message)?;
Ok(())
}
/// Write arbitrary files to the metadata branch without overwriting checkpoint.json.
pub fn write_files(
&self,
run_id: &str,
entries: &[(&str, &[u8])],
message: &str,
) -> Result<(), MetadataError> {
let (store, sig) = self.open_store()?;
let branch = Self::branch_name(run_id);
let branch_store = BranchStore::new(&store, &branch, &sig);
let message = self.commit_message(message);
branch_store.write_entries(entries, &message)?;
Ok(())
}
/// Write checkpoint data (and optional artifacts) to the metadata branch.
/// Returns the SHA of the new commit on the shadow branch.
pub fn write_checkpoint(
&self,
run_id: &str,
checkpoint_json: &[u8],
artifacts: &[(&str, &[u8])],
) -> Result<String, MetadataError> {
let (store, sig) = self.open_store()?;
let branch = Self::branch_name(run_id);
let branch_store = BranchStore::new(&store, &branch, &sig);
let mut entries: Vec<(&str, &[u8])> = vec![("checkpoint.json", checkpoint_json)];
entries.extend_from_slice(artifacts);
let message = self.commit_message("checkpoint");
let oid = branch_store.write_entries(&entries, &message)?;
Ok(oid.to_string())
}
/// Read a single file from the metadata branch. Returns `None` if branch or path doesn't exist.
fn read_file(
repo_path: &Path,
run_id: &str,
path: &str,
) -> Result<Option<Vec<u8>>, MetadataError> {
let Ok(repo) = Repository::discover(repo_path) else {
return Ok(None);
};
let store = Store::new(repo);
let sig = Signature::now("Fabro", "noreply@fabro.sh").map_err(Error::from)?;
let branch = Self::branch_name(run_id);
let branch_store = BranchStore::new(&store, &branch, &sig);
Ok(branch_store.read_entry(path)?)
}
/// Read a checkpoint from the metadata branch. Returns `None` if branch or file doesn't exist.
pub fn read_checkpoint(
repo_path: &Path,
run_id: &str,
) -> Result<Option<Checkpoint>, MetadataError> {
let branch = Self::branch_name(run_id);
match Self::read_file(repo_path, run_id, "checkpoint.json")? {
Some(bytes) => serde_json::from_slice(&bytes).map(Some).map_err(|source| {
MetadataError::Deserialize {
entity: "checkpoint",
branch,
source,
}
}),
None => Ok(None),
}
}
/// Read the run record from the metadata branch. Returns `None` if not found.
pub fn read_run_record(
repo_path: &Path,
run_id: &str,
) -> Result<Option<RunRecord>, MetadataError> {
let branch = Self::branch_name(run_id);
match Self::read_file(repo_path, run_id, "run.json")? {
Some(bytes) => serde_json::from_slice(&bytes).map(Some).map_err(|source| {
MetadataError::Deserialize {
entity: "run record",
branch,
source,
}
}),
None => Ok(None),
}
}
/// Read the start record from the metadata branch. Returns `None` if not found.
pub fn read_start_record(
repo_path: &Path,
run_id: &str,
) -> Result<Option<StartRecord>, MetadataError> {
let branch = Self::branch_name(run_id);
match Self::read_file(repo_path, run_id, "start.json")? {
Some(bytes) => serde_json::from_slice(&bytes).map(Some).map_err(|source| {
MetadataError::Deserialize {
entity: "start record",
branch,
source,
}
}),
None => Ok(None),
}
}
/// Read an artifact from the metadata branch. Returns `None` if not found.
pub fn read_artifact(
repo_path: &Path,
run_id: &str,
key: &str,
) -> Result<Option<Vec<u8>>, MetadataError> {
Self::read_file(repo_path, run_id, &format!("artifacts/{key}.json"))
}
}
#[cfg(test)]
mod tests {
use std::collections::HashMap;
use super::*;
use chrono::{TimeZone, Utc};
use fabro_types::{FabroSettings, Graph, fixtures};
/// Create a temporary git repo with an initial commit.
fn init_repo(dir: &Path) {
std::process::Command::new("git")
.args(["init"])
.current_dir(dir)
.output()
.unwrap();
std::process::Command::new("git")
.args([
"-c",
"user.name=test",
"-c",
"user.email=test@test",
"commit",
"--allow-empty",
"-m",
"init",
])
.current_dir(dir)
.output()
.unwrap();
}
fn test_run_record(run_id: fabro_types::RunId) -> RunRecord {
RunRecord {
run_id,
created_at: Utc.with_ymd_and_hms(2025, 1, 1, 0, 0, 0).single().unwrap(),
settings: FabroSettings::default(),
graph: Graph::new("test"),
workflow_slug: None,
working_directory: PathBuf::from("/tmp"),
host_repo_path: None,
base_branch: None,
labels: HashMap::new(),
}
}
fn test_checkpoint(
current_node: &str,
completed_nodes: Vec<String>,
next_node_id: Option<String>,
) -> Checkpoint {
Checkpoint {
timestamp: Utc.with_ymd_and_hms(2025, 1, 1, 0, 0, 0).single().unwrap(),
current_node: current_node.to_string(),
completed_nodes,
node_retries: HashMap::new(),
context_values: HashMap::new(),
node_outcomes: HashMap::new(),
next_node_id,
git_commit_sha: None,
loop_failure_signatures: HashMap::new(),
restart_failure_signatures: HashMap::new(),
node_visits: HashMap::new(),
}
}
fn branch_entry(repo_dir: &Path, run_id: &str, path: &str) -> Vec<u8> {
let repo = Repository::discover(repo_dir).unwrap();
let store = Store::new(repo);
let sig = Signature::now("Test", "test@example.com").unwrap();
let branch = MetadataStore::branch_name(run_id);
let branch_store = BranchStore::new(&store, &branch, &sig);
branch_store.read_entry(path).unwrap().unwrap()
}
#[test]
fn metadata_store_init_run_and_read() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
let run_id = fixtures::RUN_1.to_string();
let run_record = serde_json::to_vec_pretty(&test_run_record(fixtures::RUN_1)).unwrap();
store
.init_run(&run_id, &[("run.json", &run_record)])
.unwrap();
let read_record = MetadataStore::read_run_record(dir.path(), &run_id)
.unwrap()
.unwrap();
assert_eq!(read_record.run_id, fixtures::RUN_1);
assert_eq!(read_record.graph.name, "test");
}
#[test]
fn metadata_store_write_and_read_checkpoint() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let run_id = fixtures::RUN_2.to_string();
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
store.init_run(&run_id, &[]).unwrap();
let mut checkpoint = test_checkpoint(
"node_a",
vec!["start".to_string()],
Some("node_b".to_string()),
);
checkpoint
.context_values
.insert("goal".to_string(), serde_json::json!("test"));
let checkpoint_json = serde_json::to_vec_pretty(&checkpoint).unwrap();
store
.write_checkpoint(&run_id, &checkpoint_json, &[])
.unwrap();
let loaded = MetadataStore::read_checkpoint(dir.path(), &run_id)
.unwrap()
.unwrap();
assert_eq!(loaded.current_node, "node_a");
assert_eq!(loaded.completed_nodes, vec!["start"]);
assert_eq!(loaded.next_node_id.as_deref(), Some("node_b"));
assert_eq!(
loaded.context_values.get("goal"),
Some(&serde_json::json!("test"))
);
}
#[test]
fn metadata_store_write_checkpoint_overwrites() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let run_id = fixtures::RUN_3.to_string();
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
store.init_run(&run_id, &[]).unwrap();
let checkpoint_one =
serde_json::to_vec_pretty(&test_checkpoint("node_a", vec!["start".to_string()], None))
.unwrap();
store
.write_checkpoint(&run_id, &checkpoint_one, &[])
.unwrap();
let checkpoint_two = serde_json::to_vec_pretty(&test_checkpoint(
"node_b",
vec!["start".to_string(), "node_a".to_string()],
Some("node_c".to_string()),
))
.unwrap();
store
.write_checkpoint(&run_id, &checkpoint_two, &[])
.unwrap();
let loaded = MetadataStore::read_checkpoint(dir.path(), &run_id)
.unwrap()
.unwrap();
assert_eq!(loaded.current_node, "node_b");
assert_eq!(loaded.completed_nodes.len(), 2);
}
#[test]
fn metadata_store_read_checkpoint_missing_branch() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let result = MetadataStore::read_checkpoint(dir.path(), "NONEXISTENT").unwrap();
assert!(result.is_none());
}
#[test]
fn metadata_store_artifact_roundtrip() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let run_id = fixtures::RUN_4.to_string();
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
store.init_run(&run_id, &[]).unwrap();
let artifact_data = br#"{"large_output":"some data"}"#;
let checkpoint_json =
serde_json::to_vec_pretty(&test_checkpoint("node_a", Vec::new(), None)).unwrap();
store
.write_checkpoint(
&run_id,
&checkpoint_json,
&[("artifacts/response.plan.json", artifact_data.as_slice())],
)
.unwrap();
let read_back = MetadataStore::read_artifact(dir.path(), &run_id, "response.plan")
.unwrap()
.unwrap();
assert_eq!(read_back, artifact_data);
}
#[test]
fn metadata_store_write_files() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let run_id = fixtures::RUN_5.to_string();
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
let run_record = serde_json::to_vec_pretty(&test_run_record(fixtures::RUN_5)).unwrap();
store
.init_run(&run_id, &[("run.json", &run_record)])
.unwrap();
store
.write_files(
&run_id,
&[("retro.json", b"{\"status\":\"ok\"}")],
"finalize run",
)
.unwrap();
let data = branch_entry(dir.path(), &run_id, "retro.json");
assert_eq!(data, b"{\"status\":\"ok\"}");
let record = MetadataStore::read_run_record(dir.path(), &run_id)
.unwrap()
.unwrap();
assert_eq!(record.run_id, fixtures::RUN_5);
}
#[test]
fn metadata_store_init_run_with_extra_files() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let run_id = fixtures::RUN_6.to_string();
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
store
.init_run(&run_id, &[("sandbox.json", b"{\"type\":\"local\"}")])
.unwrap();
let data = branch_entry(dir.path(), &run_id, "sandbox.json");
assert_eq!(data, b"{\"type\":\"local\"}");
}
#[test]
fn metadata_store_read_start_record_roundtrip() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let run_id = fixtures::RUN_6.to_string();
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
let start_record = StartRecord {
run_id: fixtures::RUN_6,
start_time: Utc.with_ymd_and_hms(2025, 1, 1, 0, 0, 0).single().unwrap(),
run_branch: Some("fabro/run/test".to_string()),
base_sha: None,
};
let bytes = serde_json::to_vec_pretty(&start_record).unwrap();
store.init_run(&run_id, &[("start.json", &bytes)]).unwrap();
let loaded = MetadataStore::read_start_record(dir.path(), &run_id)
.unwrap()
.unwrap();
assert_eq!(loaded.run_id, fixtures::RUN_6);
assert_eq!(loaded.run_branch.as_deref(), Some("fabro/run/test"));
}
}

0
lib/crates/fabro-git-storage/src/trailerlink.rs → lib/crates/fabro-checkpoint/src/trailer.rs
View file

5
lib/crates/fabro-cli/Cargo.toml
View file

@ -32,7 +32,7 @@ fabro-mcp = { path = "../fabro-mcp" }
fabro-proctitle = { path = "../fabro-proctitle" }
fabro-retro = { path = "../fabro-retro" }
fabro-sandbox = { path = "../fabro-sandbox", features = ["daytona"] }
fabro-git-storage = { path = "../fabro-git-storage" }
fabro-checkpoint = { path = "../fabro-checkpoint" }
fabro-graphviz = { path = "../fabro-graphviz" }
fabro-validate = { path = "../fabro-validate" }
fabro-workflow = { path = "../fabro-workflow" }
@ -42,6 +42,7 @@ fabro-store = { path = "../fabro-store" }
fabro-types = { path = "../fabro-types" }
fabro-util = { path = "../fabro-util" }
clap.workspace = true
clap_complete.workspace = true
cli-table.workspace = true
console.workspace = true
indicatif.workspace = true
@ -92,7 +93,7 @@ chrono = { workspace = true }
[dev-dependencies]
assert_cmd = "2"
insta = { workspace = true }
insta = { workspace = true, features = ["filters"] }
paste = "1"
predicates = "3"
serde_json.workspace = true

35
lib/crates/fabro-cli/src/args.rs
View file

@ -764,9 +764,9 @@ pub(crate) enum Commands {
#[command(subcommand)]
command: Option<ModelsCommand>,
},
/// Start the HTTP API server
/// Server operations
#[cfg(feature = "server")]
Serve(fabro_server::serve::ServeArgs),
Server(ServerNamespace),
/// Check environment and integration health
Doctor {
/// Show detailed information for each check
@ -777,9 +777,6 @@ pub(crate) enum Commands {
#[arg(long)]
dry_run: bool,
},
/// Initialize a new project (deprecated: use `repo init`)
#[command(hide = true)]
Init,
/// Set up the Fabro environment (LLMs, certs, GitHub)
Install {
/// Base URL for the web UI (used for OAuth callback URLs)
@ -812,6 +809,8 @@ pub(crate) enum Commands {
#[command(subcommand)]
command: SandboxCommand,
},
/// Generate shell completions
Completion(CompletionArgs),
/// System maintenance commands
System(SystemNamespace),
/// Send a queued analytics event (internal)
@ -855,13 +854,14 @@ impl Commands {
None => "model",
},
#[cfg(feature = "server")]
Self::Serve(_) => "serve",
Self::Server(ns) => match &ns.command {
ServerCommand::Start(_) => "server start",
},
Self::Doctor { .. } => "doctor",
Self::Repo(ns) => match &ns.command {
RepoCommand::Init { .. } => "repo init",
RepoCommand::Deinit => "repo deinit",
},
Self::Init => "init",
Self::Install { .. } => "install",
Self::Pr(ns) => match &ns.command {
PrCommand::Create(_) => "pr create",
@ -891,6 +891,7 @@ impl Commands {
ProviderCommand::Login(_) => "provider login",
},
Self::Sandbox { command } => command.name(),
Self::Completion(_) => "completion",
Self::System(ns) => match &ns.command {
SystemCommand::Prune(_) => "system prune",
SystemCommand::Df(_) => "system df",
@ -966,6 +967,20 @@ pub(crate) enum SecretCommand {
Set(SecretSetArgs),
}
#[cfg(feature = "server")]
#[derive(Args)]
pub(crate) struct ServerNamespace {
#[command(subcommand)]
pub(crate) command: ServerCommand,
}
#[cfg(feature = "server")]
#[derive(Subcommand)]
pub(crate) enum ServerCommand {
/// Start the HTTP API server
Start(fabro_server::serve::ServeArgs),
}
#[derive(Args)]
pub(crate) struct SystemNamespace {
#[command(subcommand)]
@ -1024,6 +1039,12 @@ pub(crate) enum ProviderCommand {
Login(ProviderLoginArgs),
}
#[derive(Args)]
pub(crate) struct CompletionArgs {
/// Shell to generate completions for
pub shell: clap_complete::Shell,
}
#[derive(Args)]
pub(crate) struct LlmNamespace {
#[command(subcommand)]

2
lib/crates/fabro-cli/src/commands/install.rs
View file

@ -745,7 +745,7 @@ pub(crate) async fn run_install(web_url: &str) -> Result<()> {
eprintln!(" To start Arc, run these commands:");
eprintln!();
eprintln!(" fabro serve");
eprintln!(" fabro server start");
eprintln!(" cd apps/fabro-web && npx react-router dev");
eprintln!();
}

2
lib/crates/fabro-cli/src/commands/run/detached.rs
View file

@ -37,7 +37,7 @@ pub(crate) async fn execute(run_dir: PathBuf, launcher_path: PathBuf, resume: bo
let services = StartServices {
cancel_token: None,
emitter: Arc::new(EventEmitter::new()),
emitter: Arc::new(EventEmitter::new(run_record.run_id)),
interviewer: Arc::new(FileInterviewer::new(
runtime_state.interview_request_path(),
runtime_state.interview_response_path(),

2
lib/crates/fabro-cli/src/commands/run/fork.rs
View file

@ -1,7 +1,7 @@
use anyhow::Context;
use anyhow::Result;
use fabro_checkpoint::git::Store;
use fabro_config::FabroSettingsExt;
use fabro_git_storage::gitobj::Store;
use fabro_util::terminal::Styles;
use fabro_workflow::operations::{
ForkRunInput, RewindTarget, build_timeline_or_rebuild, find_run_id_by_prefix_or_store, fork,

241
lib/crates/fabro-cli/src/commands/run/logs.rs
View file

@ -341,8 +341,8 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
let ts = format_timestamp(envelope.get("ts")?.as_str()?);
match event {
"WorkflowRunStarted" => {
let name = str_field(&envelope, "workflow_name").unwrap_or("?");
"run.started" => {
let name = prop_str_field(&envelope, "name").unwrap_or("?");
let run_id = str_field(&envelope, "run_id").unwrap_or("?");
let header = format!(
"{} {} {} {}",
@ -351,7 +351,7 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.bold.apply_to(name),
styles.dim.apply_to(run_id),
);
match str_field(&envelope, "goal") {
match prop_str_field(&envelope, "goal") {
Some(goal) if !goal.is_empty() => {
let body = render_indented_markdown(styles, goal, " ");
Some(format!("{header}\n{body}\n"))
@ -359,9 +359,9 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
_ => Some(header),
}
}
"WorkflowRunCompleted" => {
let duration = format_duration_ms(envelope.get("duration_ms"));
let status_str = match str_field(&envelope, "status") {
"run.completed" => {
let duration = format_duration_ms(prop_field(&envelope, "duration_ms"));
let status_str = match prop_str_field(&envelope, "status") {
Some(status) if !status.is_empty() => status,
_ => "success",
};
@ -370,7 +370,7 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
"success" | "partial_success" => &styles.bold_green,
_ => &styles.bold_red,
};
let cost = format_cost(envelope.get("total_cost"));
let cost = format_cost(prop_field(&envelope, "total_cost"));
let mut lines = vec![format!(
"{} {} {} {}",
@ -380,7 +380,7 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.dim.apply_to(&cost),
)];
if let Some(usage) = envelope.get("usage") {
if let Some(usage) = prop_field(&envelope, "usage") {
let total = usage
.get("total_tokens")
.and_then(serde_json::Value::as_i64)
@ -433,8 +433,8 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
Some(lines.join("\n"))
}
"WorkflowRunFailed" => {
let error = str_field(&envelope, "error").unwrap_or("unknown error");
"run.failed" => {
let error = prop_str_field(&envelope, "error").unwrap_or("unknown error");
Some(format!(
"{} {} {}",
styles.dim.apply_to(&ts),
@ -442,10 +442,10 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.red.apply_to(error),
))
}
"RunNotice" => {
let level = str_field(&envelope, "level").unwrap_or("info");
let code = str_field(&envelope, "code").unwrap_or("");
let message = str_field(&envelope, "message").unwrap_or("");
"run.notice" => {
let level = prop_str_field(&envelope, "level").unwrap_or("info");
let code = prop_str_field(&envelope, "code").unwrap_or("");
let message = prop_str_field(&envelope, "message").unwrap_or("");
let label = match level {
"warn" => styles.yellow.apply_to("Warning:").to_string(),
"error" => styles.bold_red.apply_to("Error:").to_string(),
@ -464,7 +464,7 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
code_suffix,
))
}
"StageStarted" => {
"stage.started" => {
let label = str_field(&envelope, "node_label").unwrap_or("?");
Some(format!(
"{} {} {}",
@ -473,36 +473,39 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.bold.apply_to(label),
))
}
"StageCompleted" => {
"stage.completed" => {
let label = str_field(&envelope, "node_label").unwrap_or("?");
let duration = format_duration_ms(envelope.get("duration_ms"));
let cost = format_cost(envelope.get("cost"));
let turns = envelope
.get("turns")
let duration = format_duration_ms(prop_field(&envelope, "duration_ms"));
let usage = prop_field(&envelope, "usage");
let cost = format_cost(usage.and_then(|value| value.get("cost")));
let input_tokens = usage
.and_then(|value| value.get("input_tokens"))
.and_then(serde_json::Value::as_u64)
.unwrap_or(0);
let tools = envelope
.get("tool_calls")
let output_tokens = usage
.and_then(|value| value.get("output_tokens"))
.and_then(serde_json::Value::as_u64)
.unwrap_or(0);
let tokens = envelope
.get("total_tokens")
.and_then(serde_json::Value::as_u64)
.unwrap_or(0);
let stats = format!("({turns} turns, {tools} tools, {})", format_tokens(tokens));
Some(format!(
"{} {} {} {} {} {}",
let token_total = input_tokens.saturating_add(output_tokens);
let mut line = format!(
"{} {} {} {} {}",
styles.dim.apply_to(&ts),
styles.green.apply_to("\u{2713}"),
styles.bold.apply_to(label),
cost,
duration,
styles.dim.apply_to(&stats),
))
);
if token_total > 0 {
line.push_str(&format!(
" {}",
styles.dim.apply_to(format_tokens(token_total))
));
}
Some(line)
}
"StageFailed" => {
"stage.failed" => {
let label = str_field(&envelope, "node_label").unwrap_or("?");
let error = str_field(&envelope, "error").unwrap_or("unknown error");
let error = prop_str_field(&envelope, "error").unwrap_or("unknown error");
Some(format!(
"{} {} {} {}",
styles.dim.apply_to(&ts),
@ -511,10 +514,10 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.red.apply_to(error),
))
}
"Agent.AssistantMessage" => {
"agent.message" => {
let stage = str_field(&envelope, "node_id").unwrap_or("?");
let model = str_field(&envelope, "model").unwrap_or("?");
let text = str_field(&envelope, "text").unwrap_or("");
let model = prop_str_field(&envelope, "model").unwrap_or("?");
let text = prop_str_field(&envelope, "text").unwrap_or("");
let header = format!(
"{} {} {} {}{}{}",
styles.dim.apply_to(&ts),
@ -527,8 +530,8 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
let body = render_indented_markdown(styles, text, " ");
Some(format!("{header}\n{body}\n"))
}
"Agent.ToolCallStarted" => {
let tool = str_field(&envelope, "tool_name").unwrap_or("?");
"agent.tool.started" => {
let tool = prop_str_field(&envelope, "tool_name").unwrap_or("?");
let detail = tool_detail(&envelope);
let display = match detail {
Some(value) => format!("{tool}({value})"),
@ -541,10 +544,9 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.dim.apply_to(&display),
))
}
"Agent.ToolCallCompleted" => {
let tool = str_field(&envelope, "tool_name").unwrap_or("?");
let is_error = envelope
.get("is_error")
"agent.tool.completed" => {
let tool = prop_str_field(&envelope, "tool_name").unwrap_or("?");
let is_error = prop_field(&envelope, "is_error")
.and_then(serde_json::Value::as_bool)
.unwrap_or(false);
let detail = tool_detail(&envelope);
@ -561,10 +563,10 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
display,
))
}
"EdgeSelected" => {
let to = str_field(&envelope, "to_node_id").unwrap_or("?");
let reason = str_field(&envelope, "reason").unwrap_or("?");
let condition = str_field(&envelope, "condition");
"edge.selected" => {
let to = prop_str_field(&envelope, "to_node").unwrap_or("?");
let reason = prop_str_field(&envelope, "reason").unwrap_or("?");
let condition = prop_str_field(&envelope, "condition");
let detail = match condition {
Some(value) => format!(" [{value}]"),
None => String::new(),
@ -578,9 +580,9 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.dim.apply_to(&detail),
))
}
"Sandbox.Ready" => {
let provider = str_field(&envelope, "sandbox_provider").unwrap_or("?");
let duration = format_duration_ms(envelope.get("duration_ms"));
"sandbox.ready" => {
let provider = prop_str_field(&envelope, "provider").unwrap_or("?");
let duration = format_duration_ms(prop_field(&envelope, "duration_ms"));
Some(format!(
"{} Sandbox: {} {}",
styles.dim.apply_to(&ts),
@ -588,26 +590,28 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.dim.apply_to(&duration),
))
}
"SetupCompleted" => {
let count = envelope
.get("command_count")
.and_then(serde_json::Value::as_u64)
.unwrap_or(0);
let duration = format_duration_ms(envelope.get("duration_ms"));
Some(format!(
"{} Setup: {} commands {}",
styles.dim.apply_to(&ts),
count,
styles.dim.apply_to(&duration),
))
"setup.completed" => {
let count = prop_field(&envelope, "command_count").and_then(serde_json::Value::as_u64);
let duration = format_duration_ms(prop_field(&envelope, "duration_ms"));
Some(match count {
Some(count) => format!(
"{} Setup: {} commands {}",
styles.dim.apply_to(&ts),
count,
styles.dim.apply_to(&duration),
),
None => format!(
"{} Setup: {}",
styles.dim.apply_to(&ts),
styles.dim.apply_to(&duration),
),
})
}
"Agent.CompactionCompleted" => {
let original = envelope
.get("original_turn_count")
"agent.compaction.completed" => {
let original = prop_field(&envelope, "original_turn_count")
.and_then(serde_json::Value::as_u64)
.unwrap_or(0);
let preserved = envelope
.get("preserved_turn_count")
let preserved = prop_field(&envelope, "preserved_turn_count")
.and_then(serde_json::Value::as_u64)
.unwrap_or(0);
Some(format!(
@ -618,9 +622,8 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
.apply_to(format!("compaction: {original}\u{2192}{preserved} turns")),
))
}
"ParallelStarted" => {
let count = envelope
.get("branch_count")
"parallel.started" => {
let count = prop_field(&envelope, "branch_count")
.and_then(serde_json::Value::as_u64)
.unwrap_or(0);
Some(format!(
@ -630,7 +633,7 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
count,
))
}
"ParallelBranchStarted" => {
"parallel.branch.started" => {
let label = str_field(&envelope, "node_label").unwrap_or("?");
Some(format!(
"{} {} {}",
@ -639,7 +642,7 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
label,
))
}
"ParallelBranchCompleted" => {
"parallel.branch.completed" => {
let label = str_field(&envelope, "node_label").unwrap_or("?");
Some(format!(
"{} {} {}",
@ -648,8 +651,8 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
label,
))
}
"ParallelCompleted" => {
let duration = format_duration_ms(envelope.get("duration_ms"));
"parallel.completed" => {
let duration = format_duration_ms(prop_field(&envelope, "duration_ms"));
Some(format!(
"{} {} Parallel {}",
styles.dim.apply_to(&ts),
@ -657,10 +660,9 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
duration,
))
}
"PullRequestCreated" => {
let url = str_field(&envelope, "pr_url").unwrap_or("?");
let draft = envelope
.get("draft")
"pull_request.created" => {
let url = prop_str_field(&envelope, "pr_url").unwrap_or("?");
let draft = prop_field(&envelope, "draft")
.and_then(serde_json::Value::as_bool)
.unwrap_or(false);
let label = if draft { "Draft PR:" } else { "PR:" };
@ -671,8 +673,8 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
url,
))
}
"PullRequestFailed" => {
let error = str_field(&envelope, "error").unwrap_or("unknown error");
"pull_request.failed" => {
let error = prop_str_field(&envelope, "error").unwrap_or("unknown error");
Some(format!(
"{} {} {}",
styles.dim.apply_to(&ts),
@ -680,8 +682,8 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.red.apply_to(error),
))
}
"RetroCompleted" => {
let duration = format_duration_ms(envelope.get("duration_ms"));
"retro.completed" => {
let duration = format_duration_ms(prop_field(&envelope, "duration_ms"));
Some(format!(
"{} {} Retro {}",
styles.dim.apply_to(&ts),
@ -689,9 +691,9 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
duration,
))
}
"RetroFailed" => {
let error = str_field(&envelope, "error").unwrap_or("unknown error");
let duration = format_duration_ms(envelope.get("duration_ms"));
"retro.failed" => {
let error = prop_str_field(&envelope, "error").unwrap_or("unknown error");
let duration = format_duration_ms(prop_field(&envelope, "duration_ms"));
Some(format!(
"{} {} Retro {} {}",
styles.dim.apply_to(&ts),
@ -700,7 +702,7 @@ pub(crate) fn format_event_pretty(line: &str, styles: &Styles) -> Option<String>
styles.red.apply_to(error),
))
}
"RetroStarted" => Some(format!(
"retro.started" => Some(format!(
"{} {} Retro",
styles.dim.apply_to(&ts),
styles.bold_cyan.apply_to("\u{25b6}"),
@ -713,6 +715,14 @@ fn str_field<'a>(value: &'a serde_json::Value, key: &str) -> Option<&'a str> {
value.get(key)?.as_str()
}
fn prop_field<'a>(value: &'a serde_json::Value, key: &str) -> Option<&'a serde_json::Value> {
value.get("properties")?.get(key)
}
fn prop_str_field<'a>(value: &'a serde_json::Value, key: &str) -> Option<&'a str> {
prop_field(value, key)?.as_str()
}
fn format_timestamp(ts: &str) -> String {
ts.parse::<DateTime<Utc>>()
.map_or_else(|_| ts.to_string(), |dt| dt.format("%H:%M:%S").to_string())
@ -751,8 +761,8 @@ fn format_tokens(tokens: u64) -> String {
}
fn tool_detail(envelope: &serde_json::Value) -> Option<String> {
let tool_name = str_field(envelope, "tool_name")?;
let arguments = envelope.get("arguments")?;
let tool_name = prop_str_field(envelope, "tool_name")?;
let arguments = prop_field(envelope, "arguments")?;
let arg = |key: &str| arguments.get(key).and_then(|v| v.as_str());
match tool_name {
@ -861,9 +871,9 @@ mod tests {
fn since_filters_by_timestamp() {
let cutoff = "2026-01-01T12:00:00Z".parse::<DateTime<Utc>>().unwrap();
let lines = vec![
r#"{"ts":"2026-01-01T11:00:00Z","event":"StageStarted"}"#.to_string(),
r#"{"ts":"2026-01-01T12:30:00Z","event":"StageCompleted"}"#.to_string(),
r#"{"ts":"2026-01-01T13:00:00Z","event":"WorkflowRunCompleted"}"#.to_string(),
r#"{"ts":"2026-01-01T11:00:00Z","event":"stage.started"}"#.to_string(),
r#"{"ts":"2026-01-01T12:30:00Z","event":"stage.completed"}"#.to_string(),
r#"{"ts":"2026-01-01T13:00:00Z","event":"run.completed"}"#.to_string(),
];
let result = apply_filters(&lines, Some(&cutoff), None);
assert_eq!(result.len(), 2);
@ -872,7 +882,7 @@ mod tests {
#[test]
fn raw_lines_pass_through_verbatim() {
let lines = vec![
r#"{"ts":"2026-01-01T12:00:00Z","event":"StageStarted","node_label":"plan"}"#
r#"{"ts":"2026-01-01T12:00:00Z","event":"stage.started","node_label":"plan"}"#
.to_string(),
];
let result = apply_filters(&lines, None, None);
@ -882,7 +892,7 @@ mod tests {
#[test]
fn pretty_stage_started() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:09Z","event":"StageStarted","node_label":"plan","node_id":"plan","stage_index":0}"#;
let line = r#"{"ts":"2026-01-01T14:23:09Z","event":"stage.started","node_label":"plan","node_id":"plan","properties":{"index":0}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("plan"), "got: {result}");
assert!(result.contains("\u{25b6}"), "got: {result}");
@ -891,18 +901,18 @@ mod tests {
#[test]
fn pretty_stage_completed() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:15Z","event":"StageCompleted","node_label":"plan","cost":0.12,"duration_ms":8000,"turns":3,"tool_calls":2,"total_tokens":15200}"#;
let line = r#"{"ts":"2026-01-01T14:23:15Z","event":"stage.completed","node_label":"plan","properties":{"duration_ms":8000,"status":"success","usage":{"cost":0.12,"input_tokens":10000,"output_tokens":5200}}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("plan"), "got: {result}");
assert!(result.contains("$0.12"), "got: {result}");
assert!(result.contains("8s"), "got: {result}");
assert!(result.contains("3 turns"), "got: {result}");
assert!(result.contains("15.2k toks"), "got: {result}");
}
#[test]
fn pretty_assistant_message() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:12Z","event":"Agent.AssistantMessage","node_id":"plan","model":"claude-opus-4-6","text":"I'll start by reading the code.","usage":{"input_tokens":100,"output_tokens":50},"tool_call_count":0}"#;
let line = r#"{"ts":"2026-01-01T14:23:12Z","event":"agent.message","node_id":"plan","properties":{"model":"claude-opus-4-6","text":"I'll start by reading the code.","usage":{"input_tokens":100,"output_tokens":50},"tool_call_count":0}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("plan"), "got: {result}");
assert!(result.contains("claude-opus-4-6"), "got: {result}");
@ -912,7 +922,7 @@ mod tests {
#[test]
fn pretty_tool_call_started() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:12Z","event":"Agent.ToolCallStarted","tool_name":"read_file","tool_call_id":"tc_1","arguments":{"path":"src/main.rs"}}"#;
let line = r#"{"ts":"2026-01-01T14:23:12Z","event":"agent.tool.started","properties":{"tool_name":"read_file","tool_call_id":"tc_1","arguments":{"path":"src/main.rs"}}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("read_file"), "got: {result}");
assert!(result.contains("src/main.rs"), "got: {result}");
@ -921,15 +931,14 @@ mod tests {
#[test]
fn pretty_skips_noise_events() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:12Z","event":"Agent.TextDelta","delta":"hello"}"#;
let line = r#"{"ts":"2026-01-01T14:23:12Z","event":"agent.text.delta","properties":{"delta":"hello"}}"#;
assert!(format_event_pretty(line, &styles).is_none());
}
#[test]
fn pretty_skips_assistant_output_replace_noise_event() {
let styles = no_color_styles();
let line =
r#"{"ts":"2026-01-01T14:23:12Z","event":"Agent.AssistantOutputReplace","text":""}"#;
let line = r#"{"ts":"2026-01-01T14:23:12Z","event":"agent.output.replace","properties":{"text":""}}"#;
assert!(format_event_pretty(line, &styles).is_none());
}
@ -943,7 +952,7 @@ mod tests {
#[test]
fn pretty_workflow_run_started() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:01Z","run_id":"abc123","event":"WorkflowRunStarted","workflow_name":"smoke"}"#;
let line = r#"{"ts":"2026-01-01T14:23:01Z","run_id":"abc123","event":"run.started","properties":{"name":"smoke"}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("smoke"), "got: {result}");
assert!(result.contains("abc123"), "got: {result}");
@ -952,7 +961,7 @@ mod tests {
#[test]
fn pretty_workflow_run_started_with_goal() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:01Z","run_id":"abc123","event":"WorkflowRunStarted","workflow_name":"smoke","goal":"Fix the bug"}"#;
let line = r#"{"ts":"2026-01-01T14:23:01Z","run_id":"abc123","event":"run.started","properties":{"name":"smoke","goal":"Fix the bug"}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("smoke"), "got: {result}");
assert!(result.contains("abc123"), "got: {result}");
@ -963,7 +972,7 @@ mod tests {
#[test]
fn pretty_workflow_run_started_without_goal_no_extra_lines() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:01Z","run_id":"abc123","event":"WorkflowRunStarted","workflow_name":"smoke"}"#;
let line = r#"{"ts":"2026-01-01T14:23:01Z","run_id":"abc123","event":"run.started","properties":{"name":"smoke"}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(!result.contains('\n'), "got: {result}");
}
@ -971,7 +980,7 @@ mod tests {
#[test]
fn pretty_workflow_run_completed() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:32Z","run_id":"abc123","event":"WorkflowRunCompleted","duration_ms":25000,"status":"success","total_cost":0.57,"usage":{"input_tokens":5000,"output_tokens":2000,"total_tokens":7000,"cache_read_tokens":3000,"cache_write_tokens":500,"reasoning_tokens":800}}"#;
let line = r#"{"ts":"2026-01-01T14:23:32Z","run_id":"abc123","event":"run.completed","properties":{"duration_ms":25000,"status":"success","total_cost":0.57,"usage":{"input_tokens":5000,"output_tokens":2000,"total_tokens":7000,"cache_read_tokens":3000,"cache_write_tokens":500,"reasoning_tokens":800}}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("SUCCESS"), "got: {result}");
assert!(result.contains("25s"), "got: {result}");
@ -985,7 +994,7 @@ mod tests {
#[test]
fn pretty_workflow_run_completed_backward_compat() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:32Z","run_id":"abc123","event":"WorkflowRunCompleted","duration_ms":25000,"total_cost":0.57}"#;
let line = r#"{"ts":"2026-01-01T14:23:32Z","run_id":"abc123","event":"run.completed","properties":{"duration_ms":25000,"total_cost":0.57}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("SUCCESS"), "got: {result}");
assert!(result.contains("25s"), "got: {result}");
@ -996,7 +1005,7 @@ mod tests {
#[test]
fn pretty_workflow_run_completed_fail_status() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:32Z","event":"WorkflowRunCompleted","duration_ms":25000,"status":"fail"}"#;
let line = r#"{"ts":"2026-01-01T14:23:32Z","event":"run.completed","properties":{"duration_ms":25000,"status":"fail"}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("FAIL"), "got: {result}");
}
@ -1004,7 +1013,7 @@ mod tests {
#[test]
fn pretty_pull_request_created() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"PullRequestCreated","pr_url":"https://github.com/owner/repo/pull/42","pr_number":42,"draft":false}"#;
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"pull_request.created","properties":{"pr_url":"https://github.com/owner/repo/pull/42","pr_number":42,"draft":false}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("PR:"), "got: {result}");
assert!(
@ -1016,7 +1025,7 @@ mod tests {
#[test]
fn pretty_pull_request_created_draft() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"PullRequestCreated","pr_url":"https://github.com/owner/repo/pull/42","pr_number":42,"draft":true}"#;
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"pull_request.created","properties":{"pr_url":"https://github.com/owner/repo/pull/42","pr_number":42,"draft":true}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("Draft PR:"), "got: {result}");
}
@ -1024,7 +1033,7 @@ mod tests {
#[test]
fn pretty_pull_request_failed() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"PullRequestFailed","error":"auth token expired"}"#;
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"pull_request.failed","properties":{"error":"auth token expired"}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("PR failed:"), "got: {result}");
assert!(result.contains("auth token expired"), "got: {result}");
@ -1033,7 +1042,7 @@ mod tests {
#[test]
fn pretty_run_notice_warn() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"RunNotice","level":"warn","code":"sandbox_cleanup_failed","message":"sandbox cleanup failed: boom"}"#;
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"run.notice","properties":{"level":"warn","code":"sandbox_cleanup_failed","message":"sandbox cleanup failed: boom"}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("Warning:"), "got: {result}");
assert!(
@ -1046,7 +1055,7 @@ mod tests {
#[test]
fn pretty_run_notice_error() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"RunNotice","level":"error","code":"launch_failed","message":"failed to start engine"}"#;
let line = r#"{"ts":"2026-01-01T14:25:00Z","event":"run.notice","properties":{"level":"error","code":"launch_failed","message":"failed to start engine"}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("Error:"), "got: {result}");
assert!(result.contains("failed to start engine"), "got: {result}");
@ -1056,12 +1065,22 @@ mod tests {
#[test]
fn pretty_workflow_run_failed() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:32Z","run_id":"abc123","event":"WorkflowRunFailed","error":"sandbox timeout"}"#;
let line = r#"{"ts":"2026-01-01T14:23:32Z","run_id":"abc123","event":"run.failed","properties":{"error":"sandbox timeout"}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("Failed"), "got: {result}");
assert!(result.contains("sandbox timeout"), "got: {result}");
}
#[test]
fn pretty_setup_completed_without_command_count() {
let styles = no_color_styles();
let line = r#"{"ts":"2026-01-01T14:23:32Z","event":"setup.completed","properties":{"duration_ms":800}}"#;
let result = format_event_pretty(line, &styles).unwrap();
assert!(result.contains("Setup:"), "got: {result}");
assert!(result.contains("800ms"), "got: {result}");
assert!(!result.contains("0 commands"), "got: {result}");
}
#[test]
fn format_duration_ms_subsecond() {
assert_eq!(format_duration_ms(Some(&serde_json::json!(500))), "500ms");

2
lib/crates/fabro-cli/src/commands/run/rewind.rs
View file

@ -2,8 +2,8 @@ use anyhow::Context;
use anyhow::Result;
use cli_table::format::{Border, Separator};
use cli_table::{Cell, CellStruct, Color, Style, Table};
use fabro_checkpoint::git::Store;
use fabro_config::FabroSettingsExt;
use fabro_git_storage::gitobj::Store;
use fabro_util::terminal::Styles;
use fabro_workflow::git::MetadataStore;
use fabro_workflow::operations::{

2155
lib/crates/fabro-cli/src/commands/run/run_progress.rs
View file

File diff suppressed because it is too large Load diff

739
lib/crates/fabro-cli/src/commands/run/run_progress/event.rs Normal file
View file

@ -0,0 +1,739 @@
use std::convert::TryFrom;
use chrono::{DateTime, Utc};
use fabro_workflow::event::RunNoticeLevel;
use fabro_workflow::outcome::{StageUsage, compute_stage_cost};
use serde_json::{Map, Value};
#[derive(Debug, Clone)]
pub(super) struct ProgressUsage {
pub(super) model: Option<String>,
pub(super) input_tokens: u64,
pub(super) output_tokens: u64,
pub(super) speed: Option<String>,
pub(super) cost: Option<f64>,
}
impl ProgressUsage {
pub(super) fn from_value(value: &Value) -> Option<Self> {
let Value::Object(fields) = value else {
return None;
};
Some(Self {
model: string_field(fields, "model"),
input_tokens: u64_field(fields, "input_tokens"),
output_tokens: u64_field(fields, "output_tokens"),
speed: string_field(fields, "speed"),
cost: f64_field(fields, "cost"),
})
}
pub(super) fn total_tokens(&self) -> u64 {
self.input_tokens.saturating_add(self.output_tokens)
}
pub(super) fn display_cost(&self) -> Option<f64> {
self.cost.or_else(|| {
let model = self.model.clone()?;
let input_tokens = i64::try_from(self.input_tokens).ok()?;
let output_tokens = i64::try_from(self.output_tokens).ok()?;
let usage = StageUsage {
model,
input_tokens,
output_tokens,
cache_read_tokens: None,
cache_write_tokens: None,
reasoning_tokens: None,
speed: self.speed.clone(),
cost: None,
};
compute_stage_cost(&usage)
})
}
}
#[derive(Debug, Clone)]
pub(super) enum ProgressEvent {
WorkflowStarted {
worktree_dir: Option<String>,
base_branch: Option<String>,
base_sha: Option<String>,
},
WorkingDirectorySet {
working_directory: String,
},
SandboxInitializing {
provider: String,
},
SandboxReady {
provider: String,
duration_ms: u64,
name: Option<String>,
cpu: Option<f64>,
memory: Option<f64>,
url: Option<String>,
},
SshAccessReady {
ssh_command: String,
},
SetupStarted {
command_count: u64,
},
SetupCompleted {
duration_ms: u64,
},
SetupCommandCompleted {
command: String,
command_index: u64,
exit_code: i64,
duration_ms: u64,
},
CliEnsureStarted {
cli_name: String,
},
CliEnsureCompleted {
cli_name: String,
already_installed: bool,
duration_ms: u64,
},
CliEnsureFailed {
cli_name: String,
},
DevcontainerResolved {
dockerfile_lines: u64,
environment_count: u64,
lifecycle_command_count: u64,
workspace_folder: String,
},
DevcontainerLifecycleStarted {
phase: String,
command_count: u64,
},
DevcontainerLifecycleCompleted {
phase: String,
duration_ms: u64,
},
DevcontainerLifecycleFailed {
phase: String,
command: String,
exit_code: i64,
stderr: String,
},
DevcontainerLifecycleCommandCompleted {
command: String,
command_index: u64,
exit_code: i64,
duration_ms: u64,
},
StageStarted {
node_id: String,
name: String,
script: Option<String>,
},
StageCompleted {
node_id: String,
name: String,
duration_ms: u64,
status: String,
usage: Option<ProgressUsage>,
},
StageFailed {
node_id: String,
name: String,
error: String,
},
StageRetrying {
name: String,
attempt: u64,
max_attempts: u64,
delay_ms: u64,
},
ParallelStarted,
ParallelBranchStarted {
branch: String,
},
ParallelBranchCompleted {
branch: String,
duration_ms: u64,
status: String,
},
ParallelCompleted,
AssistantMessage {
stage_node_id: String,
model: String,
},
ToolCallStarted {
stage_node_id: String,
tool_name: String,
tool_call_id: String,
arguments: Value,
timestamp: Option<DateTime<Utc>>,
},
ToolCallCompleted {
stage_node_id: String,
tool_call_id: String,
is_error: bool,
duration_ms: Option<u64>,
timestamp: Option<DateTime<Utc>>,
},
ContextWindowWarning {
stage_node_id: String,
usage_percent: u64,
},
CompactionStarted {
stage_node_id: String,
},
CompactionCompleted {
stage_node_id: String,
original_turn_count: u64,
preserved_turn_count: u64,
tracked_file_count: u64,
},
LlmRetry {
stage_node_id: String,
model: String,
attempt: u64,
delay_ms: u64,
error: String,
},
SubagentSpawned {
stage_node_id: String,
agent_id: String,
task: String,
},
SubagentCompleted {
stage_node_id: String,
agent_id: String,
success: bool,
turns_used: u64,
},
EdgeSelected {
from_node: String,
to_node: String,
label: Option<String>,
condition: Option<String>,
},
LoopRestart {
from_node: String,
to_node: String,
},
RetroStarted,
RetroCompleted {
duration_ms: u64,
},
RetroFailed {
duration_ms: u64,
},
RunNotice {
level: RunNoticeLevel,
code: String,
message: String,
},
PullRequestCreated {
pr_url: String,
draft: bool,
},
PullRequestFailed {
error: String,
},
}
#[allow(clippy::needless_pass_by_value)]
pub(super) fn from_envelope_fields(
event_name: &str,
fields: &Map<String, Value>,
) -> Option<ProgressEvent> {
match event_name {
"run.started" => Some(ProgressEvent::WorkflowStarted {
worktree_dir: prop_string_field(fields, "worktree_dir"),
base_branch: prop_string_field(fields, "base_branch"),
base_sha: prop_string_field(fields, "base_sha"),
}),
"sandbox.initialized" => Some(ProgressEvent::WorkingDirectorySet {
working_directory: prop_string_field(fields, "working_directory")?,
}),
"sandbox.initializing" => Some(ProgressEvent::SandboxInitializing {
provider: prop_string_field(fields, "provider")
.unwrap_or_else(|| "unknown".to_string()),
}),
"sandbox.ready" => Some(ProgressEvent::SandboxReady {
provider: prop_string_field(fields, "provider")
.unwrap_or_else(|| "unknown".to_string()),
duration_ms: prop_u64_field(fields, "duration_ms"),
name: prop_string_field(fields, "name"),
cpu: prop_f64_field(fields, "cpu"),
memory: prop_f64_field(fields, "memory"),
url: prop_string_field(fields, "url"),
}),
"ssh.ready" => Some(ProgressEvent::SshAccessReady {
ssh_command: prop_string_field(fields, "ssh_command")?,
}),
"setup.started" => Some(ProgressEvent::SetupStarted {
command_count: prop_u64_field(fields, "command_count"),
}),
"setup.completed" => Some(ProgressEvent::SetupCompleted {
duration_ms: prop_u64_field(fields, "duration_ms"),
}),
"setup.command.completed" => Some(ProgressEvent::SetupCommandCompleted {
command: prop_string_field(fields, "command").unwrap_or_else(|| "?".to_string()),
command_index: prop_u64_field(fields, "index"),
exit_code: prop_i64_field(fields, "exit_code"),
duration_ms: prop_u64_field(fields, "duration_ms"),
}),
"cli.ensure.started" => Some(ProgressEvent::CliEnsureStarted {
cli_name: prop_string_field(fields, "cli_name").unwrap_or_else(|| "?".to_string()),
}),
"cli.ensure.completed" => Some(ProgressEvent::CliEnsureCompleted {
cli_name: prop_string_field(fields, "cli_name").unwrap_or_else(|| "?".to_string()),
already_installed: prop_bool_field(fields, "already_installed"),
duration_ms: prop_u64_field(fields, "duration_ms"),
}),
"cli.ensure.failed" => Some(ProgressEvent::CliEnsureFailed {
cli_name: prop_string_field(fields, "cli_name").unwrap_or_else(|| "?".to_string()),
}),
"devcontainer.resolved" => Some(ProgressEvent::DevcontainerResolved {
dockerfile_lines: prop_u64_field(fields, "dockerfile_lines"),
environment_count: prop_u64_field(fields, "environment_count"),
lifecycle_command_count: prop_u64_field(fields, "lifecycle_command_count"),
workspace_folder: prop_string_field(fields, "workspace_folder")
.unwrap_or_else(|| "?".to_string()),
}),
"devcontainer.lifecycle.started" => Some(ProgressEvent::DevcontainerLifecycleStarted {
phase: prop_string_field(fields, "phase").unwrap_or_else(|| "?".to_string()),
command_count: prop_u64_field(fields, "command_count"),
}),
"devcontainer.lifecycle.completed" => Some(ProgressEvent::DevcontainerLifecycleCompleted {
phase: prop_string_field(fields, "phase").unwrap_or_else(|| "?".to_string()),
duration_ms: prop_u64_field(fields, "duration_ms"),
}),
"devcontainer.lifecycle.failed" => Some(ProgressEvent::DevcontainerLifecycleFailed {
phase: prop_string_field(fields, "phase").unwrap_or_else(|| "?".to_string()),
command: prop_string_field(fields, "command").unwrap_or_else(|| "?".to_string()),
exit_code: prop_i64_field(fields, "exit_code"),
stderr: prop_display_field(fields, "stderr").unwrap_or_default(),
}),
"devcontainer.lifecycle.command.completed" => {
Some(ProgressEvent::DevcontainerLifecycleCommandCompleted {
command: prop_string_field(fields, "command").unwrap_or_else(|| "?".to_string()),
command_index: prop_u64_field(fields, "index"),
exit_code: prop_i64_field(fields, "exit_code"),
duration_ms: prop_u64_field(fields, "duration_ms"),
})
}
"stage.started" => Some(ProgressEvent::StageStarted {
node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
name: string_field(fields, "node_label").unwrap_or_else(|| "?".to_string()),
script: prop_string_field(fields, "script"),
}),
"stage.completed" => Some(ProgressEvent::StageCompleted {
node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
name: string_field(fields, "node_label").unwrap_or_else(|| "?".to_string()),
duration_ms: prop_u64_field(fields, "duration_ms"),
status: prop_string_field(fields, "status").unwrap_or_else(|| "success".to_string()),
usage: prop_value(fields, "usage").and_then(ProgressUsage::from_value),
}),
"stage.failed" => Some(ProgressEvent::StageFailed {
node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
name: string_field(fields, "node_label").unwrap_or_else(|| "?".to_string()),
error: prop_display_field(fields, "error")
.unwrap_or_else(|| "unknown error".to_string()),
}),
"stage.retrying" => Some(ProgressEvent::StageRetrying {
name: string_field(fields, "node_label").unwrap_or_else(|| "?".to_string()),
attempt: prop_u64_field(fields, "attempt"),
max_attempts: prop_u64_field(fields, "max_attempts"),
delay_ms: prop_u64_field(fields, "delay_ms"),
}),
"parallel.started" => Some(ProgressEvent::ParallelStarted),
"parallel.branch.started" => Some(ProgressEvent::ParallelBranchStarted {
branch: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
}),
"parallel.branch.completed" => Some(ProgressEvent::ParallelBranchCompleted {
branch: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
duration_ms: prop_u64_field(fields, "duration_ms"),
status: prop_string_field(fields, "status").unwrap_or_else(|| "success".to_string()),
}),
"parallel.completed" => Some(ProgressEvent::ParallelCompleted),
"agent.message" => Some(ProgressEvent::AssistantMessage {
stage_node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
model: prop_string_field(fields, "model").unwrap_or_else(|| "?".to_string()),
}),
"agent.tool.started" => Some(ProgressEvent::ToolCallStarted {
stage_node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
tool_name: prop_string_field(fields, "tool_name").unwrap_or_else(|| "?".to_string()),
tool_call_id: prop_string_field(fields, "tool_call_id")
.unwrap_or_else(|| "?".to_string()),
arguments: prop_value(fields, "arguments")
.cloned()
.unwrap_or_else(|| Value::Object(Map::new())),
timestamp: timestamp_field(fields, "ts"),
}),
"agent.tool.completed" => Some(ProgressEvent::ToolCallCompleted {
stage_node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
tool_call_id: prop_string_field(fields, "tool_call_id")
.unwrap_or_else(|| "?".to_string()),
is_error: prop_bool_field(fields, "is_error"),
duration_ms: prop_optional_u64_field(fields, "duration_ms"),
timestamp: timestamp_field(fields, "ts"),
}),
"agent.warning"
if prop_string_field(fields, "kind").as_deref() == Some("context_window") =>
{
let usage_percent = prop_value(fields, "details")
.and_then(Value::as_object)
.and_then(|details| details.get("usage_percent"))
.and_then(Value::as_u64)
.unwrap_or(0);
Some(ProgressEvent::ContextWindowWarning {
stage_node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
usage_percent,
})
}
"agent.compaction.started" => Some(ProgressEvent::CompactionStarted {
stage_node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
}),
"agent.compaction.completed" => Some(ProgressEvent::CompactionCompleted {
stage_node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
original_turn_count: prop_u64_field(fields, "original_turn_count"),
preserved_turn_count: prop_u64_field(fields, "preserved_turn_count"),
tracked_file_count: prop_u64_field(fields, "tracked_file_count"),
}),
"agent.llm.retry" => {
let delay_secs = prop_f64_field(fields, "delay_secs").unwrap_or(0.0);
#[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)]
let delay_ms = (delay_secs * 1000.0) as u64;
Some(ProgressEvent::LlmRetry {
stage_node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
model: prop_string_field(fields, "model").unwrap_or_else(|| "?".to_string()),
attempt: prop_u64_field(fields, "attempt"),
delay_ms,
error: prop_display_field(fields, "error")
.unwrap_or_else(|| "unknown error".to_string()),
})
}
"agent.sub.spawned" => Some(ProgressEvent::SubagentSpawned {
stage_node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
agent_id: prop_string_field(fields, "agent_id").unwrap_or_else(|| "?".to_string()),
task: prop_string_field(fields, "task").unwrap_or_default(),
}),
"agent.sub.completed" => Some(ProgressEvent::SubagentCompleted {
stage_node_id: string_field(fields, "node_id").unwrap_or_else(|| "?".to_string()),
agent_id: prop_string_field(fields, "agent_id").unwrap_or_else(|| "?".to_string()),
success: prop_bool_field(fields, "success"),
turns_used: prop_u64_field(fields, "turns_used"),
}),
"edge.selected" => Some(ProgressEvent::EdgeSelected {
from_node: prop_string_field(fields, "from_node").unwrap_or_else(|| "?".to_string()),
to_node: prop_string_field(fields, "to_node").unwrap_or_else(|| "?".to_string()),
label: prop_string_field(fields, "label"),
condition: prop_string_field(fields, "condition"),
}),
"loop.restart" => Some(ProgressEvent::LoopRestart {
from_node: prop_string_field(fields, "from_node").unwrap_or_else(|| "?".to_string()),
to_node: prop_string_field(fields, "to_node").unwrap_or_else(|| "?".to_string()),
}),
"retro.started" => Some(ProgressEvent::RetroStarted),
"retro.completed" => Some(ProgressEvent::RetroCompleted {
duration_ms: prop_u64_field(fields, "duration_ms"),
}),
"retro.failed" => Some(ProgressEvent::RetroFailed {
duration_ms: prop_u64_field(fields, "duration_ms"),
}),
"run.notice" => Some(ProgressEvent::RunNotice {
level: parse_run_notice_level(prop_string_field(fields, "level").as_deref()),
code: prop_string_field(fields, "code").unwrap_or_default(),
message: prop_string_field(fields, "message").unwrap_or_default(),
}),
"pull_request.created" => Some(ProgressEvent::PullRequestCreated {
pr_url: prop_string_field(fields, "pr_url").unwrap_or_else(|| "?".to_string()),
draft: prop_bool_field(fields, "draft"),
}),
"pull_request.failed" => Some(ProgressEvent::PullRequestFailed {
error: prop_display_field(fields, "error")
.unwrap_or_else(|| "unknown error".to_string()),
}),
_ => None,
}
}
fn parse_run_notice_level(level: Option<&str>) -> RunNoticeLevel {
match level.unwrap_or("info") {
"warn" => RunNoticeLevel::Warn,
"error" => RunNoticeLevel::Error,
_ => RunNoticeLevel::Info,
}
}
fn string_field(fields: &Map<String, Value>, key: &str) -> Option<String> {
fields.get(key).and_then(Value::as_str).map(str::to_owned)
}
fn prop_value<'a>(fields: &'a Map<String, Value>, key: &str) -> Option<&'a Value> {
fields
.get("properties")
.and_then(Value::as_object)
.and_then(|properties| properties.get(key))
}
fn prop_string_field(fields: &Map<String, Value>, key: &str) -> Option<String> {
prop_value(fields, key)
.and_then(Value::as_str)
.map(str::to_owned)
}
fn prop_display_field(fields: &Map<String, Value>, key: &str) -> Option<String> {
let value = prop_value(fields, key)?;
match value {
Value::Null => None,
Value::String(value) => Some(value.clone()),
Value::Object(map) => map
.get("message")
.and_then(Value::as_str)
.map(str::to_owned)
.or_else(|| {
map.get("detail")
.and_then(Value::as_object)
.and_then(|detail| detail.get("message"))
.and_then(Value::as_str)
.map(str::to_owned)
})
.or_else(|| {
map.get("data")
.and_then(Value::as_object)
.and_then(|detail| detail.get("message"))
.and_then(Value::as_str)
.map(str::to_owned)
})
.or_else(|| map.get("data").and_then(Value::as_str).map(str::to_owned))
.or_else(|| Some(value.to_string())),
_ => Some(value.to_string()),
}
}
fn u64_field(fields: &Map<String, Value>, key: &str) -> u64 {
fields.get(key).and_then(Value::as_u64).unwrap_or(0)
}
fn prop_u64_field(fields: &Map<String, Value>, key: &str) -> u64 {
prop_value(fields, key).and_then(Value::as_u64).unwrap_or(0)
}
fn prop_optional_u64_field(fields: &Map<String, Value>, key: &str) -> Option<u64> {
prop_value(fields, key).and_then(Value::as_u64)
}
fn prop_i64_field(fields: &Map<String, Value>, key: &str) -> i64 {
prop_value(fields, key).and_then(Value::as_i64).unwrap_or(0)
}
fn f64_field(fields: &Map<String, Value>, key: &str) -> Option<f64> {
fields.get(key).and_then(Value::as_f64)
}
fn prop_f64_field(fields: &Map<String, Value>, key: &str) -> Option<f64> {
prop_value(fields, key).and_then(Value::as_f64)
}
fn prop_bool_field(fields: &Map<String, Value>, key: &str) -> bool {
prop_value(fields, key)
.and_then(Value::as_bool)
.unwrap_or(false)
}
fn timestamp_field(fields: &Map<String, Value>, key: &str) -> Option<DateTime<Utc>> {
let value = fields.get(key)?.as_str()?;
DateTime::parse_from_rfc3339(value)
.ok()
.map(|timestamp| timestamp.with_timezone(&Utc))
}
#[cfg(test)]
mod tests {
use fabro_agent::AgentEvent;
use fabro_types::fixtures;
use fabro_workflow::event::{WorkflowRunEvent, canonicalize_event};
use super::*;
fn json_map(value: Value) -> Map<String, Value> {
value.as_object().cloned().expect("json object")
}
fn canonical_fields(event: &WorkflowRunEvent) -> (String, Map<String, Value>) {
let envelope = canonicalize_event(&fixtures::RUN_1, event);
let event_name = envelope.event.clone();
let fields = json_map(serde_json::to_value(envelope).expect("serializable envelope"));
(event_name, fields)
}
#[test]
fn parse_edge_selected() {
let fields = json_map(serde_json::json!({
"properties": {
"from_node": "a",
"to_node": "b",
"label": "yes"
}
}));
let event = from_envelope_fields("edge.selected", &fields).unwrap();
assert!(matches!(
event,
ProgressEvent::EdgeSelected {
from_node,
to_node,
label,
..
} if from_node == "a" && to_node == "b" && label.as_deref() == Some("yes")
));
}
#[test]
fn round_trip_stage_completed() {
let event = WorkflowRunEvent::StageCompleted {
node_id: "plan".into(),
name: "Plan".into(),
index: 0,
duration_ms: 5000,
status: "success".into(),
preferred_label: None,
suggested_next_ids: Vec::new(),
usage: None,
failure: None,
notes: None,
files_touched: Vec::new(),
attempt: 1,
max_attempts: 1,
};
let (name, fields) = canonical_fields(&event);
let parsed = from_envelope_fields(&name, &fields).unwrap();
assert!(matches!(
parsed,
ProgressEvent::StageCompleted {
node_id,
name,
duration_ms,
..
} if node_id == "plan" && name == "Plan" && duration_ms == 5000
));
}
#[test]
fn round_trip_agent_tool_call() {
let event = WorkflowRunEvent::Agent {
stage: "code".into(),
event: AgentEvent::ToolCallStarted {
tool_name: "read_file".into(),
tool_call_id: "tc1".into(),
arguments: serde_json::json!({"path": "src/main.rs"}),
},
session_id: None,
parent_session_id: None,
};
let (name, fields) = canonical_fields(&event);
let parsed = from_envelope_fields(&name, &fields).unwrap();
assert!(matches!(
parsed,
ProgressEvent::ToolCallStarted {
stage_node_id,
tool_name,
tool_call_id,
..
} if stage_node_id == "code" && tool_name == "read_file" && tool_call_id == "tc1"
));
}
#[test]
fn parse_tool_call_timestamps_from_jsonl_envelope() {
let started_fields = json_map(serde_json::json!({
"ts": "2026-03-30T12:00:00.000Z",
"node_id": "code",
"properties": {
"tool_name": "read_file",
"tool_call_id": "tc1",
"arguments": {"path": "src/main.rs"}
}
}));
let completed_fields = json_map(serde_json::json!({
"ts": "2026-03-30T12:00:00.500Z",
"node_id": "code",
"properties": {
"tool_call_id": "tc1",
"is_error": false,
"duration_ms": 500
}
}));
let started = from_envelope_fields("agent.tool.started", &started_fields).unwrap();
let completed = from_envelope_fields("agent.tool.completed", &completed_fields).unwrap();
assert!(matches!(
started,
ProgressEvent::ToolCallStarted {
timestamp: Some(timestamp),
..
} if timestamp == DateTime::parse_from_rfc3339("2026-03-30T12:00:00.000Z")
.unwrap()
.with_timezone(&Utc)
));
assert!(matches!(
completed,
ProgressEvent::ToolCallCompleted {
duration_ms: Some(500),
timestamp: Some(timestamp),
..
} if timestamp == DateTime::parse_from_rfc3339("2026-03-30T12:00:00.500Z")
.unwrap()
.with_timezone(&Utc)
));
}
#[test]
fn round_trip_sandbox_ready() {
let event = WorkflowRunEvent::Sandbox {
event: fabro_agent::SandboxEvent::Ready {
provider: "daytona".into(),
duration_ms: 2500,
name: Some("sandbox-1".into()),
cpu: Some(4.0),
memory: Some(8.0),
url: Some("https://example.test".into()),
},
};
let (name, fields) = canonical_fields(&event);
let parsed = from_envelope_fields(&name, &fields).unwrap();
assert!(matches!(
parsed,
ProgressEvent::SandboxReady {
provider,
duration_ms,
name,
..
} if provider == "daytona" && duration_ms == 2500 && name.as_deref() == Some("sandbox-1")
));
}
#[test]
fn round_trip_run_notice() {
let event = WorkflowRunEvent::RunNotice {
level: RunNoticeLevel::Warn,
code: "sandbox_cleanup_failed".into(),
message: "sandbox cleanup failed".into(),
};
let (name, fields) = canonical_fields(&event);
let parsed = from_envelope_fields(&name, &fields).unwrap();
assert!(matches!(
parsed,
ProgressEvent::RunNotice {
level: RunNoticeLevel::Warn,
code,
message,
} if code == "sandbox_cleanup_failed" && message == "sandbox cleanup failed"
));
}
}

137
lib/crates/fabro-cli/src/commands/run/run_progress/info_display.rs Normal file
View file

@ -0,0 +1,137 @@
use std::path::Path;
use fabro_workflow::event::RunNoticeLevel;
use super::renderer::ProgressRenderer;
use super::styles;
use crate::shared::{format_duration_ms, tilde_path};
pub(super) struct InfoDisplay {
verbose: bool,
}
impl InfoDisplay {
pub(super) fn new(verbose: bool) -> Self {
Self { verbose }
}
pub(super) fn show_worktree(renderer: &ProgressRenderer, path: &Path) {
Self::insert_info_line(renderer, &format!("Worktree: {}", tilde_path(path)));
}
pub(super) fn show_base_info(renderer: &ProgressRenderer, branch: Option<&str>, sha: &str) {
let short_sha = &sha[..sha.len().min(12)];
let text = match branch {
Some(branch) => format!("Base: {branch} ({short_sha})"),
None => format!("Base: {short_sha}"),
};
Self::insert_info_line(renderer, &text);
}
pub(super) fn on_run_notice(
renderer: &ProgressRenderer,
level: RunNoticeLevel,
code: &str,
message: &str,
) {
let styles = renderer.styles();
let label = match level {
RunNoticeLevel::Info => styles.bold.apply_to("Info:").to_string(),
RunNoticeLevel::Warn => styles.yellow.apply_to("Warning:").to_string(),
RunNoticeLevel::Error => styles.red.apply_to("Error:").to_string(),
};
let code_suffix = if code.is_empty() {
String::new()
} else {
format!(" {}", styles.dim.apply_to(format!("[{code}]")))
};
Self::insert_info_line(renderer, &format!("{label} {message}{code_suffix}"));
}
pub(super) fn on_pull_request_created(renderer: &ProgressRenderer, pr_url: &str, draft: bool) {
let label = if draft { "Draft PR:" } else { "PR:" };
Self::insert_info_line(
renderer,
&format!("{} {pr_url}", renderer.styles().bold.apply_to(label)),
);
}
pub(super) fn on_pull_request_failed(renderer: &ProgressRenderer, error: &str) {
Self::insert_info_line(
renderer,
&format!("{} {error}", renderer.styles().red.apply_to("PR failed:")),
);
}
pub(super) fn on_edge_selected(
&self,
renderer: &ProgressRenderer,
from_node: &str,
to_node: &str,
label: Option<&str>,
condition: Option<&str>,
) {
if !self.verbose {
return;
}
let detail = if let Some(condition) = condition {
format!(" [{condition}]")
} else if let Some(label) = label {
format!(" \"{label}\"")
} else {
String::new()
};
Self::insert_info_line(
renderer,
&format!("\u{2192} {from_node} \u{2192} {to_node}{detail}"),
);
}
pub(super) fn on_loop_restart(
&self,
renderer: &ProgressRenderer,
from_node: &str,
to_node: &str,
) {
if !self.verbose {
return;
}
Self::insert_info_line(
renderer,
&format!("\u{21ba} {from_node} \u{2192} {to_node} (loop restart)"),
);
}
pub(super) fn on_stage_retrying(
&self,
renderer: &ProgressRenderer,
name: &str,
attempt: u64,
max_attempts: u64,
delay_ms: u64,
) {
if !self.verbose {
return;
}
Self::insert_info_line(
renderer,
&format!(
"\u{21bb} {name}: retrying (attempt {attempt}/{max_attempts}, delay {})",
format_duration_ms(delay_ms)
),
);
}
fn insert_info_line(renderer: &ProgressRenderer, message: &str) {
if renderer.is_tty() {
let bar = renderer.add_spinner();
bar.set_style(styles::style_static_dim());
bar.finish_with_message(message.to_string());
} else {
renderer.print_line(4, message);
}
}
}

1134
lib/crates/fabro-cli/src/commands/run/run_progress/mod.rs Normal file
View file

File diff suppressed because it is too large Load diff

94
lib/crates/fabro-cli/src/commands/run/run_progress/renderer.rs Normal file
View file

@ -0,0 +1,94 @@
use std::io::Write;
use std::sync::Mutex;
use fabro_util::terminal::Styles;
use indicatif::{MultiProgress, ProgressBar, ProgressDrawTarget};
use super::styles;
enum RendererInner {
Tty { multi: MultiProgress },
Plain { out: Mutex<Box<dyn Write + Send>> },
}
pub(super) struct ProgressRenderer {
inner: RendererInner,
styles: Styles,
}
impl ProgressRenderer {
pub(super) fn new_tty() -> Self {
Self {
inner: RendererInner::Tty {
multi: MultiProgress::new(),
},
styles: Styles::new(console::colors_enabled_stderr()),
}
}
pub(super) fn new_plain(out: Box<dyn Write + Send>, colors: bool) -> Self {
Self {
inner: RendererInner::Plain {
out: Mutex::new(out),
},
styles: Styles::new(colors),
}
}
pub(super) fn add_spinner(&self) -> ProgressBar {
match &self.inner {
RendererInner::Tty { multi } => multi.add(ProgressBar::new_spinner()),
RendererInner::Plain { .. } => ProgressBar::hidden(),
}
}
pub(super) fn insert_after(&self, after: &ProgressBar) -> ProgressBar {
match &self.inner {
RendererInner::Tty { multi } => multi.insert_after(after, ProgressBar::new_spinner()),
RendererInner::Plain { .. } => ProgressBar::hidden(),
}
}
pub(super) fn insert_before(&self, before: &ProgressBar) -> ProgressBar {
match &self.inner {
RendererInner::Tty { multi } => multi.insert_before(before, ProgressBar::new_spinner()),
RendererInner::Plain { .. } => ProgressBar::hidden(),
}
}
pub(super) fn print_line(&self, indent: usize, message: &str) {
if let RendererInner::Plain { out } = &self.inner {
let mut out = out.lock().expect("plain renderer lock poisoned");
let _ = writeln!(out, "{}{message}", " ".repeat(indent));
}
}
pub(super) fn is_tty(&self) -> bool {
matches!(self.inner, RendererInner::Tty { .. })
}
pub(super) fn styles(&self) -> &Styles {
&self.styles
}
pub(super) fn hide(&self) {
if let RendererInner::Tty { multi } = &self.inner {
multi.set_draw_target(ProgressDrawTarget::hidden());
}
}
pub(super) fn show(&self) {
if let RendererInner::Tty { multi } = &self.inner {
multi.set_draw_target(ProgressDrawTarget::stderr());
}
}
pub(super) fn finish(&self) {
if let RendererInner::Tty { multi } = &self.inner {
let sep = multi.add(ProgressBar::new_spinner());
sep.set_style(styles::style_empty());
sep.finish();
multi.set_draw_target(ProgressDrawTarget::hidden());
}
}
}

378
lib/crates/fabro-cli/src/commands/run/run_progress/setup_display.rs Normal file
View file

@ -0,0 +1,378 @@
use std::time::Duration;
use indicatif::ProgressBar;
use super::renderer::ProgressRenderer;
use super::styles;
use crate::shared::format_duration_ms;
pub(super) struct SetupDisplay {
verbose: bool,
pub(super) sandbox_bar: Option<ProgressBar>,
pub(super) setup_bar: Option<ProgressBar>,
pub(super) setup_command_count: u64,
pub(super) devcontainer_bar: Option<ProgressBar>,
pub(super) devcontainer_command_count: u64,
pub(super) cli_ensure_bar: Option<ProgressBar>,
}
impl SetupDisplay {
pub(super) fn new(verbose: bool) -> Self {
Self {
verbose,
sandbox_bar: None,
setup_bar: None,
setup_command_count: 0,
devcontainer_bar: None,
devcontainer_command_count: 0,
cli_ensure_bar: None,
}
}
pub(super) fn finish(&mut self) {
if let Some(bar) = self.sandbox_bar.take() {
bar.finish_and_clear();
}
if let Some(bar) = self.setup_bar.take() {
bar.finish_and_clear();
}
if let Some(bar) = self.devcontainer_bar.take() {
bar.finish_and_clear();
}
if let Some(bar) = self.cli_ensure_bar.take() {
bar.finish_and_clear();
}
}
pub(super) fn on_sandbox_initializing(&mut self, renderer: &ProgressRenderer, provider: &str) {
if renderer.is_tty() {
let bar = renderer.add_spinner();
bar.set_style(styles::style_header_running());
bar.set_message(format!("Initializing {provider} sandbox..."));
bar.enable_steady_tick(Duration::from_millis(100));
self.sandbox_bar = Some(bar);
}
}
pub(super) fn on_sandbox_ready(
&mut self,
renderer: &ProgressRenderer,
provider: &str,
duration_ms: u64,
name: Option<&str>,
cpu: Option<f64>,
memory: Option<f64>,
url: Option<&str>,
) {
let dur = format_duration_ms(duration_ms);
let detail = match (name, cpu, memory) {
(Some(name), Some(cpu), Some(memory)) => Some(format!(
"{name} ({} cpu, {} GB)",
styles::format_number(cpu),
styles::format_number(memory)
)),
(Some(name), _, _) => Some(name.to_string()),
_ => None,
};
if renderer.is_tty() {
let display_provider = match url {
Some(url) => styles::terminal_hyperlink(url, provider),
None => provider.to_string(),
};
if let Some(bar) = self.sandbox_bar.take() {
bar.set_style(styles::style_header_done());
bar.set_prefix(dur);
bar.finish_with_message(format!("Sandbox: {display_provider}"));
if let Some(detail) = detail {
let detail_bar = renderer.insert_after(&bar);
detail_bar.set_style(styles::style_sandbox_detail());
detail_bar.finish_with_message(detail);
}
}
} else {
renderer.print_line(4, &format!("Sandbox: {provider} (ready in {dur})"));
if let Some(detail) = detail {
renderer.print_line(13, &detail);
}
}
}
pub(super) fn on_ssh_access_ready(renderer: &ProgressRenderer, ssh_command: &str) {
if renderer.is_tty() {
let bar = renderer.add_spinner();
bar.set_style(styles::style_sandbox_detail());
bar.finish_with_message(ssh_command.to_string());
} else {
renderer.print_line(13, ssh_command);
}
}
pub(super) fn on_setup_started(&mut self, renderer: &ProgressRenderer, command_count: u64) {
self.setup_command_count = command_count;
if renderer.is_tty() {
let bar = renderer.add_spinner();
bar.set_style(styles::style_header_running());
bar.set_message(format!(
"Setup: {command_count} command{}...",
if command_count == 1 { "" } else { "s" }
));
bar.enable_steady_tick(Duration::from_millis(100));
self.setup_bar = Some(bar);
}
}
pub(super) fn on_setup_completed(&mut self, renderer: &ProgressRenderer, duration_ms: u64) {
let dur = format_duration_ms(duration_ms);
let suffix = if self.setup_command_count == 1 {
""
} else {
"s"
};
if renderer.is_tty() {
if let Some(bar) = self.setup_bar.take() {
bar.set_style(styles::style_header_done());
bar.set_prefix(dur);
bar.finish_with_message(format!(
"Setup: {} command{suffix}",
self.setup_command_count
));
}
} else {
renderer.print_line(
4,
&format!(
"Setup: {} command{suffix} ({dur})",
self.setup_command_count
),
);
}
}
pub(super) fn on_setup_command_completed(
&self,
renderer: &ProgressRenderer,
command: &str,
command_index: u64,
exit_code: i64,
duration_ms: u64,
) {
if !self.verbose {
return;
}
let glyph = if exit_code == 0 {
styles::green_check(renderer.styles())
} else {
styles::red_cross(renderer.styles())
};
let msg = format!(
"{glyph} [{}/{}] {}",
command_index + 1,
self.setup_command_count,
styles::truncate(command, 60)
);
let dur = format_duration_ms(duration_ms);
if renderer.is_tty() {
let bar = match &self.setup_bar {
Some(setup_bar) => renderer.insert_before(setup_bar),
None => renderer.add_spinner(),
};
bar.set_style(styles::style_tool_done());
bar.set_prefix(dur);
bar.finish_with_message(msg);
} else {
renderer.print_line(6, &format!("{msg} {dur}"));
}
}
pub(super) fn on_cli_ensure_started(&mut self, renderer: &ProgressRenderer, cli_name: &str) {
if renderer.is_tty() {
let bar = renderer.add_spinner();
bar.set_style(styles::style_header_running());
bar.set_message(format!("CLI: ensuring {cli_name}..."));
bar.enable_steady_tick(Duration::from_millis(100));
self.cli_ensure_bar = Some(bar);
}
}
pub(super) fn on_cli_ensure_completed(
&mut self,
renderer: &ProgressRenderer,
cli_name: &str,
already_installed: bool,
duration_ms: u64,
) {
let status = if already_installed {
"found"
} else {
"installed"
};
let dur = format_duration_ms(duration_ms);
if renderer.is_tty() {
if let Some(bar) = self.cli_ensure_bar.take() {
bar.set_style(styles::style_header_done());
bar.set_prefix(dur);
bar.finish_with_message(format!("CLI: {cli_name} ({status})"));
}
} else {
renderer.print_line(4, &format!("CLI: {cli_name} ({status}, {dur})"));
}
}
pub(super) fn on_cli_ensure_failed(&mut self, renderer: &ProgressRenderer, cli_name: &str) {
let message = format!(
"{} CLI: {cli_name} install failed",
styles::red_cross(renderer.styles())
);
if renderer.is_tty() {
if let Some(bar) = self.cli_ensure_bar.take() {
bar.set_style(styles::style_header_done());
bar.finish_with_message(message);
}
} else {
renderer.print_line(4, &message);
}
}
pub(super) fn on_devcontainer_resolved(
renderer: &ProgressRenderer,
dockerfile_lines: u64,
environment_count: u64,
lifecycle_command_count: u64,
workspace_folder: &str,
) {
let detail = format!(
"{dockerfile_lines} Dockerfile lines, {environment_count} env vars, \
{lifecycle_command_count} lifecycle cmds, {workspace_folder}"
);
if renderer.is_tty() {
let bar = renderer.add_spinner();
bar.set_style(styles::style_header_done());
bar.finish_with_message("Devcontainer: resolved".to_string());
let detail_bar = renderer.insert_after(&bar);
detail_bar.set_style(styles::style_sandbox_detail());
detail_bar.finish_with_message(detail);
} else {
renderer.print_line(4, "Devcontainer: resolved");
renderer.print_line(13, &detail);
}
}
pub(super) fn on_devcontainer_lifecycle_started(
&mut self,
renderer: &ProgressRenderer,
phase: &str,
command_count: u64,
) {
self.devcontainer_command_count = command_count;
if renderer.is_tty() {
let bar = renderer.add_spinner();
bar.set_style(styles::style_header_running());
bar.set_message(format!(
"Running devcontainer {phase} ({command_count} commands)..."
));
bar.enable_steady_tick(Duration::from_millis(100));
self.devcontainer_bar = Some(bar);
} else {
renderer.print_line(
4,
&format!("Running devcontainer {phase} ({command_count} commands)..."),
);
}
}
pub(super) fn on_devcontainer_lifecycle_completed(
&mut self,
renderer: &ProgressRenderer,
phase: &str,
duration_ms: u64,
) {
let dur = format_duration_ms(duration_ms);
if renderer.is_tty() {
if let Some(bar) = self.devcontainer_bar.take() {
bar.set_style(styles::style_header_done());
bar.set_prefix(dur);
bar.finish_with_message(format!("Devcontainer: {phase}"));
}
} else {
renderer.print_line(4, &format!("Devcontainer: {phase} ({dur})"));
}
}
pub(super) fn on_devcontainer_lifecycle_failed(
&mut self,
renderer: &ProgressRenderer,
phase: &str,
command: &str,
exit_code: i64,
stderr: &str,
) {
if let Some(bar) = self.devcontainer_bar.take() {
bar.abandon();
}
let summary = if stderr.len() > 120 {
&stderr[..120]
} else {
stderr
};
let message = format!(
"{} Devcontainer {phase} command failed (exit {exit_code}): {command}\n {summary}",
renderer.styles().red.apply_to("Error:")
);
if renderer.is_tty() {
let bar = renderer.add_spinner();
bar.set_style(styles::style_static_dim());
bar.finish_with_message(message);
} else {
renderer.print_line(4, &message);
}
}
pub(super) fn on_devcontainer_lifecycle_command_completed(
&self,
renderer: &ProgressRenderer,
command: &str,
command_index: u64,
exit_code: i64,
duration_ms: u64,
) {
if !self.verbose {
return;
}
let glyph = if exit_code == 0 {
styles::green_check(renderer.styles())
} else {
styles::red_cross(renderer.styles())
};
let msg = format!(
"{glyph} [{}/{}] {}",
command_index + 1,
self.devcontainer_command_count,
styles::truncate(command, 60)
);
let dur = format_duration_ms(duration_ms);
if renderer.is_tty() {
let bar = match &self.devcontainer_bar {
Some(devcontainer_bar) => renderer.insert_before(devcontainer_bar),
None => renderer.add_spinner(),
};
bar.set_style(styles::style_tool_done());
bar.set_prefix(dur);
bar.finish_with_message(msg);
} else {
renderer.print_line(6, &format!("{msg} {dur}"));
}
}
}

743
lib/crates/fabro-cli/src/commands/run/run_progress/stage_display.rs Normal file
View file

@ -0,0 +1,743 @@
use std::collections::{HashMap, VecDeque};
use std::convert::TryFrom;
use std::time::Duration;
use chrono::{DateTime, Utc};
use indicatif::ProgressBar;
use fabro_workflow::outcome::{StageStatus, format_cost};
use super::event::ProgressUsage;
use super::renderer::ProgressRenderer;
use super::styles;
use crate::shared::{format_duration_ms, format_tokens_human};
const MAX_TOOL_CALLS: usize = 5;
#[derive(Debug)]
pub(super) enum ToolCallStatus {
Running,
Succeeded,
Failed,
}
#[derive(Debug)]
pub(super) struct ToolCallEntry {
pub(super) display_name: String,
pub(super) tool_call_id: String,
pub(super) status: ToolCallStatus,
pub(super) bar: ProgressBar,
pub(super) is_branch: bool,
pub(super) started_at: Option<DateTime<Utc>>,
}
#[derive(Debug)]
pub(super) struct ActiveStage {
pub(super) display_name: String,
pub(super) has_model: bool,
pub(super) spinner: ProgressBar,
pub(super) tool_calls: VecDeque<ToolCallEntry>,
pub(super) compaction_bar: Option<ProgressBar>,
}
impl ActiveStage {
fn last_bar(&self) -> &ProgressBar {
self.tool_calls
.back()
.map_or(&self.spinner, |entry| &entry.bar)
}
}
pub(super) struct StageDisplay {
verbose: bool,
pub(super) active_stages: HashMap<String, ActiveStage>,
pub(super) stage_counts: HashMap<String, (u64, u64)>,
pub(super) parallel_parent: Option<String>,
any_stage_started: bool,
working_directory: Option<String>,
}
impl StageDisplay {
pub(super) fn new(verbose: bool) -> Self {
Self {
verbose,
active_stages: HashMap::new(),
stage_counts: HashMap::new(),
parallel_parent: None,
any_stage_started: false,
working_directory: None,
}
}
pub(super) fn set_working_directory(&mut self, dir: String) {
self.working_directory = Some(dir);
}
pub(super) fn finish(&mut self) {
for (_node_id, stage) in self.active_stages.drain() {
if let Some(bar) = stage.compaction_bar {
bar.finish_and_clear();
}
for entry in &stage.tool_calls {
if entry.is_branch || self.verbose {
entry.bar.abandon();
} else {
entry.bar.finish_and_clear();
}
}
stage.spinner.finish_and_clear();
}
}
pub(super) fn on_stage_started(
&mut self,
renderer: &ProgressRenderer,
node_id: &str,
name: &str,
script: Option<&str>,
) {
self.stage_counts.insert(node_id.to_string(), (0, 0));
let display_name = match script {
Some(script) => format!(
"{name} {}",
renderer.styles().dim.apply_to(styles::truncate(script, 60))
),
None => name.to_string(),
};
if renderer.is_tty() && !self.any_stage_started {
self.any_stage_started = true;
let sep = renderer.add_spinner();
sep.set_style(styles::style_empty());
sep.finish();
}
let bar = renderer.add_spinner();
bar.set_style(styles::style_stage_running());
bar.set_message(display_name.clone());
if renderer.is_tty() {
bar.enable_steady_tick(Duration::from_millis(100));
}
self.active_stages.insert(
node_id.to_string(),
ActiveStage {
display_name,
has_model: false,
spinner: bar,
tool_calls: VecDeque::new(),
compaction_bar: None,
},
);
}
pub(super) fn on_stage_completed(
&mut self,
renderer: &ProgressRenderer,
node_id: &str,
name: &str,
duration_ms: u64,
status: &str,
usage: Option<&ProgressUsage>,
) {
let succeeded = status.parse::<StageStatus>().map_or_else(
|_| matches!(status, "success" | "partial_success"),
|status| matches!(status, StageStatus::Success | StageStatus::PartialSuccess),
);
let cost_str = usage
.and_then(ProgressUsage::display_cost)
.map(|cost| format!("{} ", format_cost(cost)))
.unwrap_or_default();
let stats_str = if self.verbose {
let (turn_count, tool_call_count) =
self.stage_counts.get(node_id).copied().unwrap_or((0, 0));
let total_tokens = usage.map_or(0, ProgressUsage::total_tokens);
if turn_count > 0 || tool_call_count > 0 || total_tokens > 0 {
let total_tokens = i64::try_from(total_tokens).unwrap_or(i64::MAX);
format!(
" {}",
renderer.styles().dim.apply_to(format!(
"({} turns, {} tools, {} toks)",
turn_count,
tool_call_count,
format_tokens_human(total_tokens),
))
)
} else {
String::new()
}
} else {
String::new()
};
let prefix = format!("{cost_str}{}{stats_str}", format_duration_ms(duration_ms));
let glyph = if succeeded {
styles::green_check(renderer.styles())
} else {
styles::red_cross(renderer.styles())
};
self.finish_stage(renderer, node_id, name, &glyph, &prefix);
}
pub(super) fn on_stage_failed(
&mut self,
renderer: &ProgressRenderer,
node_id: &str,
name: &str,
error: &str,
) {
self.finish_stage(
renderer,
node_id,
name,
&styles::red_cross(renderer.styles()),
"",
);
let summary = styles::last_line_truncated(error, 120);
Self::insert_global_info_line(
renderer,
&format!("{} {summary}", renderer.styles().red.apply_to("Error:")),
);
}
pub(super) fn on_parallel_started(&mut self) {
self.parallel_parent = self
.active_stages
.keys()
.next()
.cloned()
.or_else(|| Some(String::new()));
}
pub(super) fn on_parallel_completed(&mut self) {
self.parallel_parent = None;
}
pub(super) fn on_parallel_branch_started(&mut self, renderer: &ProgressRenderer, branch: &str) {
let Some(parent_id) = self.parallel_parent.clone() else {
return;
};
let Some(stage) = self.active_stages.get_mut(&parent_id) else {
return;
};
let bar = renderer.insert_after(stage.last_bar());
bar.set_style(styles::style_subagent_info());
bar.set_message(
renderer
.styles()
.dim
.apply_to(format!("\u{25b8} {branch}"))
.to_string(),
);
stage.tool_calls.push_back(ToolCallEntry {
display_name: branch.to_string(),
tool_call_id: branch.to_string(),
status: ToolCallStatus::Running,
bar,
is_branch: true,
started_at: None,
});
}
pub(super) fn on_parallel_branch_completed(
&mut self,
renderer: &ProgressRenderer,
branch: &str,
duration_ms: u64,
status: &str,
) {
let Some(parent_id) = self.parallel_parent.clone() else {
return;
};
let Some(stage) = self.active_stages.get_mut(&parent_id) else {
return;
};
let Some(entry) = stage
.tool_calls
.iter_mut()
.find(|entry| entry.tool_call_id == branch)
else {
return;
};
let succeeded = matches!(status, "success" | "partial_success");
entry.status = if succeeded {
ToolCallStatus::Succeeded
} else {
ToolCallStatus::Failed
};
let glyph = if succeeded {
styles::green_check(renderer.styles())
} else {
styles::red_cross(renderer.styles())
};
if renderer.is_tty() {
entry.bar.set_style(styles::style_branch_done());
set_duration_prefix(&entry.bar, Some(duration_ms));
entry
.bar
.finish_with_message(format!("{glyph} {}", entry.display_name));
} else {
renderer.print_line(
8,
&format!("{glyph} {branch} {}", format_duration_ms(duration_ms)),
);
}
}
pub(super) fn on_assistant_message(
&mut self,
renderer: &ProgressRenderer,
stage_node_id: &str,
model: &str,
) {
if let Some(counts) = self.stage_counts.get_mut(stage_node_id) {
counts.0 += 1;
}
if let Some(stage) = self.active_stages.get_mut(stage_node_id) {
if !stage.has_model {
stage.has_model = true;
let suffix = format!(" {}", renderer.styles().dim.apply_to(format!("[{model}]")));
stage.display_name.push_str(&suffix);
stage.spinner.set_message(stage.display_name.clone());
}
}
}
pub(super) fn on_tool_call_started(
&mut self,
renderer: &ProgressRenderer,
stage_node_id: &str,
tool_name: &str,
tool_call_id: &str,
arguments: &serde_json::Value,
timestamp: Option<DateTime<Utc>>,
) {
let display_name = self.tool_display_name(renderer, tool_name, arguments);
let Some(stage) = self.active_stages.get_mut(stage_node_id) else {
return;
};
if !self.verbose && stage.tool_calls.len() >= MAX_TOOL_CALLS {
let evict_idx = stage
.tool_calls
.iter()
.position(|entry| !matches!(entry.status, ToolCallStatus::Running))
.unwrap_or(0);
if let Some(evicted) = stage.tool_calls.remove(evict_idx) {
evicted.bar.finish_and_clear();
}
}
let bar = renderer.insert_after(stage.last_bar());
bar.set_style(styles::style_tool_running());
bar.set_message(display_name.clone());
if renderer.is_tty() {
bar.enable_steady_tick(Duration::from_millis(100));
}
stage.tool_calls.push_back(ToolCallEntry {
display_name,
tool_call_id: tool_call_id.to_string(),
status: ToolCallStatus::Running,
bar,
is_branch: false,
started_at: timestamp,
});
}
pub(super) fn on_tool_call_completed(
&mut self,
renderer: &ProgressRenderer,
stage_node_id: &str,
tool_call_id: &str,
is_error: bool,
duration_ms: Option<u64>,
timestamp: Option<DateTime<Utc>>,
) {
if let Some(counts) = self.stage_counts.get_mut(stage_node_id) {
counts.1 += 1;
}
let Some(stage) = self.active_stages.get_mut(stage_node_id) else {
return;
};
let Some(entry) = stage
.tool_calls
.iter_mut()
.find(|entry| entry.tool_call_id == tool_call_id)
else {
return;
};
let glyph = if is_error {
styles::red_cross(renderer.styles())
} else {
styles::green_check(renderer.styles())
};
entry.status = if is_error {
ToolCallStatus::Failed
} else {
ToolCallStatus::Succeeded
};
if renderer.is_tty() {
entry.bar.set_style(styles::style_tool_done());
let computed_duration_ms = duration_ms.or_else(|| {
entry
.started_at
.zip(timestamp)
.and_then(|(started_at, completed_at)| {
u64::try_from(
completed_at
.signed_duration_since(started_at)
.num_milliseconds(),
)
.ok()
})
});
set_duration_prefix(&entry.bar, computed_duration_ms);
entry
.bar
.finish_with_message(format!("{glyph} {}", entry.display_name));
}
}
pub(super) fn on_context_window_warning(
&mut self,
renderer: &ProgressRenderer,
stage_node_id: &str,
usage_percent: u64,
) {
if !self.verbose {
return;
}
self.insert_info_line_for_stage(
renderer,
stage_node_id,
&format!(
"{} context window: {usage_percent}% used",
styles::warning_glyph(renderer.styles())
),
);
}
pub(super) fn on_compaction_started(
&mut self,
renderer: &ProgressRenderer,
stage_node_id: &str,
) {
if !renderer.is_tty() {
return;
}
let Some(stage) = self.active_stages.get_mut(stage_node_id) else {
return;
};
if let Some(old) = stage.compaction_bar.take() {
old.finish_and_clear();
}
let bar = renderer.insert_after(stage.last_bar());
bar.set_style(styles::style_tool_running());
bar.set_message("\u{27f3} compacting context\u{2026}");
bar.enable_steady_tick(Duration::from_millis(100));
stage.compaction_bar = Some(bar);
}
pub(super) fn on_compaction_completed(
&mut self,
renderer: &ProgressRenderer,
stage_node_id: &str,
original_turn_count: u64,
preserved_turn_count: u64,
tracked_file_count: u64,
) {
let message = format!(
"\u{27f3} compaction: {original_turn_count} \u{2192} {preserved_turn_count} turns, {tracked_file_count} files"
);
if renderer.is_tty() {
if let Some(bar) = self
.active_stages
.get_mut(stage_node_id)
.and_then(|stage| stage.compaction_bar.take())
{
bar.set_style(styles::style_tool_done());
bar.finish_with_message(message);
} else {
self.insert_info_line_for_stage(renderer, stage_node_id, &message);
}
} else {
renderer.print_line(6, &message);
}
}
pub(super) fn on_llm_retry(
&mut self,
renderer: &ProgressRenderer,
stage_node_id: &str,
model: &str,
attempt: u64,
delay_ms: u64,
error: &str,
) {
if !self.verbose {
return;
}
self.insert_info_line_for_stage(
renderer,
stage_node_id,
&format!(
"{} retry: {model} attempt {attempt} ({error}, delay {})",
styles::warning_glyph(renderer.styles()),
format_duration_ms(delay_ms)
),
);
}
pub(super) fn on_subagent_spawned(
&mut self,
renderer: &ProgressRenderer,
stage_node_id: &str,
agent_id: &str,
task: &str,
) {
if !self.verbose {
return;
}
self.insert_subagent_line_for_stage(
renderer,
stage_node_id,
&renderer
.styles()
.dim
.apply_to(format!(
"\u{25b8} subagent[{agent_id}] \"{}\"",
styles::truncate(task, 50)
))
.to_string(),
);
}
pub(super) fn on_subagent_completed(
&mut self,
renderer: &ProgressRenderer,
stage_node_id: &str,
agent_id: &str,
success: bool,
turns_used: u64,
) {
if !self.verbose {
return;
}
let glyph = if success {
styles::green_check(renderer.styles())
} else {
styles::red_cross(renderer.styles())
};
self.insert_subagent_line_for_stage(
renderer,
stage_node_id,
&format!("{glyph} subagent[{agent_id}] ({turns_used} turns)"),
);
}
pub(super) fn on_retro_started(&mut self, renderer: &ProgressRenderer) {
self.on_stage_started(renderer, "retro", "Retro", None);
}
pub(super) fn on_retro_completed(&mut self, renderer: &ProgressRenderer, duration_ms: u64) {
self.finish_stage(
renderer,
"retro",
"Retro",
&styles::green_check(renderer.styles()),
&format_duration_ms(duration_ms),
);
}
pub(super) fn on_retro_failed(&mut self, renderer: &ProgressRenderer, duration_ms: u64) {
self.finish_stage(
renderer,
"retro",
"Retro",
&styles::red_cross(renderer.styles()),
&format_duration_ms(duration_ms),
);
}
fn finish_stage(
&mut self,
renderer: &ProgressRenderer,
node_id: &str,
name: &str,
glyph: &str,
prefix: &str,
) {
let Some(stage) = self.active_stages.remove(node_id) else {
if !renderer.is_tty() {
Self::print_plain_stage_completion(renderer, name, glyph, prefix);
}
return;
};
if let Some(bar) = stage.compaction_bar {
bar.finish_and_clear();
}
for entry in &stage.tool_calls {
if entry.is_branch || self.verbose {
entry.bar.abandon();
} else {
entry.bar.finish_and_clear();
}
}
if renderer.is_tty() {
stage.spinner.set_style(styles::style_stage_done());
stage.spinner.set_prefix(prefix.to_string());
stage
.spinner
.finish_with_message(format!("{glyph} {}", stage.display_name));
} else {
Self::print_plain_stage_completion(renderer, name, glyph, prefix);
}
}
fn print_plain_stage_completion(
renderer: &ProgressRenderer,
name: &str,
glyph: &str,
prefix: &str,
) {
if prefix.is_empty() {
renderer.print_line(4, &format!("{glyph} {name}"));
} else {
renderer.print_line(4, &format!("{glyph} {name} {prefix}"));
}
}
fn insert_global_info_line(renderer: &ProgressRenderer, message: &str) {
if renderer.is_tty() {
let bar = renderer.add_spinner();
bar.set_style(styles::style_static_dim());
bar.finish_with_message(message.to_string());
} else {
renderer.print_line(4, message);
}
}
fn insert_info_line_for_stage(
&self,
renderer: &ProgressRenderer,
stage_node_id: &str,
message: &str,
) {
if renderer.is_tty() {
let bar = if let Some(stage) = self.active_stages.get(stage_node_id) {
renderer.insert_after(stage.last_bar())
} else {
renderer.add_spinner()
};
bar.set_style(styles::style_tool_done());
bar.finish_with_message(message.to_string());
} else {
renderer.print_line(6, message);
}
}
fn insert_subagent_line_for_stage(
&self,
renderer: &ProgressRenderer,
stage_node_id: &str,
message: &str,
) {
if renderer.is_tty() {
let bar = if let Some(stage) = self.active_stages.get(stage_node_id) {
renderer.insert_after(stage.last_bar())
} else {
renderer.add_spinner()
};
bar.set_style(styles::style_subagent_info());
bar.finish_with_message(message.to_string());
} else {
renderer.print_line(8, message);
}
}
fn tool_display_name(
&self,
renderer: &ProgressRenderer,
tool_name: &str,
arguments: &serde_json::Value,
) -> String {
let arg = |key: &str| arguments.get(key).and_then(serde_json::Value::as_str);
let working_directory = self.working_directory.as_deref();
let path_arg = || {
arg("path")
.or_else(|| arg("file_path"))
.map(|path| styles::truncate(&styles::shorten_path(path, working_directory), 60))
};
let detail = match tool_name {
"bash" | "shell" | "execute_command" => {
arg("command").map(|command| styles::truncate(command, 60))
}
"glob" => arg("pattern").map(String::from),
"grep" | "ripgrep" => arg("pattern").map(|pattern| styles::truncate(pattern, 40)),
"read_file" | "read" | "write_file" | "write" | "create_file" | "edit_file"
| "edit" | "list_dir" => path_arg(),
"web_search" => arg("query").map(|query| styles::truncate(query, 60)),
"web_fetch" => arg("url").map(|url| styles::truncate(url, 60)),
"spawn_agent" => arg("task").map(|task| styles::truncate(task, 60)),
"wait" | "send_input" | "close_agent" => arg("agent_id").map(String::from),
"use_skill" => arg("skill_name").map(String::from),
"apply_patch" => Some("...".to_string()),
"read_many_files" => arguments
.get("paths")
.and_then(serde_json::Value::as_array)
.map(|paths| format!("{} files", paths.len())),
_ => None,
};
match detail {
Some(detail) => format!(
"{tool_name}{}",
renderer.styles().dim.apply_to(format!("({detail})"))
),
None => tool_name.to_string(),
}
}
}
fn set_duration_prefix(bar: &ProgressBar, duration_ms: Option<u64>) {
let prefix = duration_ms.map_or_else(
|| styles::format_duration_short(bar.elapsed()),
|duration_ms| styles::format_duration_short(Duration::from_millis(duration_ms)),
);
bar.set_prefix(prefix);
}
#[cfg(test)]
mod tests {
use super::*;
use crate::commands::run::run_progress::renderer::ProgressRenderer;
#[test]
fn tool_display_name_shortens_paths_relative_to_working_directory() {
let renderer = ProgressRenderer::new_plain(Box::new(std::io::sink()), false);
let mut stage = StageDisplay::new(false);
stage.set_working_directory("/workspace".into());
let display_name = stage.tool_display_name(
&renderer,
"read_file",
&serde_json::json!({"file_path": "/workspace/src/main.rs"}),
);
assert_eq!(display_name, "read_file(src/main.rs)");
}
}

116
lib/crates/fabro-cli/src/commands/run/run_progress/styles.rs Normal file
View file

@ -0,0 +1,116 @@
use std::path::Path;
use std::sync::OnceLock;
use std::time::Duration;
use fabro_util::terminal::Styles;
use indicatif::ProgressStyle;
macro_rules! cached_style {
($name:ident, $template:expr) => {
pub(super) fn $name() -> ProgressStyle {
static STYLE: OnceLock<ProgressStyle> = OnceLock::new();
STYLE
.get_or_init(|| ProgressStyle::with_template($template).expect("valid template"))
.clone()
}
};
}
cached_style!(
style_header_running,
" {spinner:.dim} {wide_msg} {elapsed:.dim}"
);
cached_style!(style_header_done, " {wide_msg:.dim} {prefix:.dim}");
cached_style!(
style_stage_running,
" {spinner:.cyan} {wide_msg} {elapsed:.dim}"
);
cached_style!(style_stage_done, " {wide_msg} {prefix:.dim}");
cached_style!(
style_tool_running,
" {spinner:.dim} {wide_msg} {elapsed:.dim}"
);
cached_style!(style_tool_done, " {wide_msg} {prefix:.dim}");
cached_style!(style_subagent_info, " {wide_msg}");
cached_style!(style_branch_done, " {wide_msg} {prefix:.dim}");
cached_style!(style_static_dim, " {wide_msg:.dim}");
cached_style!(style_sandbox_detail, " {wide_msg:.dim}");
cached_style!(style_empty, " ");
pub(super) fn green_check(styles: &Styles) -> String {
styles.green.apply_to("\u{2713}").to_string()
}
pub(super) fn red_cross(styles: &Styles) -> String {
styles.red.apply_to("\u{2717}").to_string()
}
pub(super) fn warning_glyph(styles: &Styles) -> String {
styles.yellow.apply_to("\u{26a0}").to_string()
}
pub(crate) fn format_duration_short(d: Duration) -> String {
let secs = d.as_secs();
if secs >= 60 {
format!("{}m{:02}s", secs / 60, secs % 60)
} else if d.as_millis() >= 1000 {
format!("{secs}s")
} else {
format!("{}ms", d.as_millis())
}
}
pub(super) fn terminal_hyperlink(url: &str, text: &str) -> String {
format!("\x1b]8;;{url}\x1b\\{text}\x1b]8;;\x1b\\")
}
pub(super) fn format_number(n: f64) -> String {
if (n - n.round()).abs() < f64::EPSILON {
#[allow(clippy::cast_possible_truncation)]
let i = n as i64;
format!("{i}")
} else {
format!("{n:.1}")
}
}
pub(super) fn truncate(s: &str, max: usize) -> String {
let single_line = s.split_whitespace().collect::<Vec<_>>().join(" ");
if single_line.len() > max {
let mut truncated: String = single_line.chars().take(max - 3).collect();
truncated.push_str("...");
truncated
} else {
single_line
}
}
pub(super) fn last_line_truncated(s: &str, max: usize) -> String {
let line = s
.trim()
.lines()
.rfind(|line| !line.trim().is_empty())
.unwrap_or("")
.trim();
if line.len() > max {
let mut truncated: String = line.chars().take(max - 3).collect();
truncated.push_str("...");
truncated
} else {
line.to_string()
}
}
pub(super) fn shorten_path(path: &str, working_directory: Option<&str>) -> String {
if let Some(wd) = working_directory {
if let Ok(rel) = Path::new(path).strip_prefix(wd) {
return rel.display().to_string();
}
}
if let Ok(cwd) = std::env::current_dir() {
if let Ok(rel) = Path::new(path).strip_prefix(&cwd) {
return rel.display().to_string();
}
}
path.to_string()
}

5
lib/crates/fabro-cli/src/commands/store/dump.rs
View file

@ -486,6 +486,7 @@ mod tests {
fn event_payload(run_id: RunId, ts: &str, event: &str) -> EventPayload {
EventPayload::new(
serde_json::json!({
"id": format!("evt-{run_id}-{event}"),
"ts": ts,
"run_id": run_id.to_string(),
"event": event
@ -540,14 +541,14 @@ mod tests {
run.append_event(&event_payload(
run_id,
"2026-03-27T12:00:00.000Z",
"WorkflowRunStarted",
"run.started",
))
.await
.unwrap();
run.append_event(&event_payload(
run_id,
"2026-03-27T12:00:01.000Z",
"StageCompleted",
"stage.completed",
))
.await
.unwrap();

42
lib/crates/fabro-cli/src/main.rs
View file

@ -11,7 +11,9 @@ mod user_config;
use anyhow::Result;
use args::{Commands, GlobalArgs, LONG_VERSION, RunCommands};
use clap::Parser;
#[cfg(feature = "server")]
use args::{ServerCommand, ServerNamespace};
use clap::{CommandFactory, Parser};
use fabro_telemetry::{git, panic as tel_panic, sanitize, sender};
use fabro_util::printer::Printer;
use fabro_util::terminal::Styles;
@ -106,7 +108,10 @@ async fn main_inner() -> (String, Result<()>) {
let (config_log_level, upgrade_check_enabled) = {
#[cfg(feature = "server")]
{
if let Commands::Serve(args) = command.as_ref() {
if let Commands::Server(ServerNamespace {
command: ServerCommand::Start(args),
}) = command.as_ref()
{
match fabro_config::server::load_server_settings(args.config.as_deref()) {
Ok(server_settings) => (
server_settings.log.as_ref().and_then(|l| l.level.clone()),
@ -136,8 +141,8 @@ async fn main_inner() -> (String, Result<()>) {
}
};
let log_prefix = if command_name == "serve" {
"serve"
let log_prefix = if command_name == "server start" {
"server"
} else {
"cli"
};
@ -153,7 +158,6 @@ async fn main_inner() -> (String, Result<()>) {
Commands::RunCmd(RunCommands::Run(_) | RunCommands::Create(_))
| Commands::Exec(_)
| Commands::Repo(_)
| Commands::Init
| Commands::Install { .. }
) {
commands::upgrade::spawn_upgrade_check(globals.no_upgrade_check, upgrade_check_enabled)
@ -183,7 +187,8 @@ async fn main_inner() -> (String, Result<()>) {
Commands::RunsCmd(cmd) => commands::runs::dispatch(cmd, &globals).await?,
Commands::Model { command } => commands::model::execute(command, &globals).await?,
#[cfg(feature = "server")]
Commands::Serve(args) => {
Commands::Server(ns) => {
let ServerCommand::Start(args) = ns.command;
let styles: &'static Styles = Box::leak(Box::new(Styles::detect_stderr()));
fabro_server::serve::serve_command(args, styles, globals.storage_dir.clone())
.await?;
@ -201,10 +206,6 @@ async fn main_inner() -> (String, Result<()>) {
open::that("https://docs.fabro.sh/")?;
}
Commands::Repo(ns) => commands::repo::dispatch(ns).await?,
Commands::Init => {
fabro_util::warn_user!("`fabro init` is deprecated, use `fabro repo init` instead");
commands::repo::init::run_init().await?;
}
Commands::Install { web_url } => {
commands::install::run_install(&web_url).await?;
}
@ -219,6 +220,27 @@ async fn main_inner() -> (String, Result<()>) {
Commands::Provider(ns) => commands::provider::dispatch(ns).await?,
Commands::Sandbox { command } => commands::sandbox::dispatch(command, &globals).await?,
Commands::System(ns) => commands::system::dispatch(ns, &globals).await?,
Commands::Completion(args) => {
let mut cmd = Cli::command();
let shell = args.shell;
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let mut buf = Vec::new();
clap_complete::generate(shell, &mut cmd, "fabro", &mut buf);
buf
}));
match result {
Ok(buf) => {
use std::io::Write;
std::io::stdout().write_all(&buf)?;
}
Err(_) => {
anyhow::bail!(
"Failed to generate completions for {shell}. \
Try zsh, fish, elvish, or powershell instead."
);
}
}
}
Commands::SendAnalytics { path } => {
let result = sender::upload(&path).await;
let _ = std::fs::remove_file(&path);

64
lib/crates/fabro-cli/tests/it/cmd/attach.rs
View file

@ -1,5 +1,7 @@
use fabro_test::{fabro_snapshot, test_context};
use crate::support::{example_fixture, run_output_filters};
use super::support::{output_stdout, write_sleep_workflow};
#[test]
@ -29,6 +31,68 @@ fn help() {
");
}
#[test]
fn attach_requires_run_arg() {
let context = test_context!();
let mut cmd = context.command();
cmd.arg("attach");
fabro_snapshot!(context.filters(), cmd, @"
success: false
exit_code: 2
----- stdout -----
----- stderr -----
error: the following required arguments were not provided:
<RUN>
Usage: fabro attach --no-upgrade-check --storage-dir <STORAGE_DIR> <RUN>
For more information, try '--help'.
");
}
#[test]
fn attach_replays_completed_detached_run() {
let context = test_context!();
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAQ";
context
.command()
.args([
"run",
"--dry-run",
"--auto-approve",
"--no-retro",
"--detach",
"--run-id",
run_id,
example_fixture("simple.fabro").to_str().unwrap(),
])
.assert()
.success();
context
.command()
.args(["wait", run_id])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
let mut cmd = context.command();
cmd.args(["attach", run_id]);
cmd.timeout(std::time::Duration::from_secs(10));
fabro_snapshot!(run_output_filters(&context), cmd, @"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
Sandbox: local (ready in [TIME])
✓ Start [TIME]
✓ Run Tests [TIME]
✓ Report [TIME]
✓ Exit [TIME]
");
}
#[test]
fn attach_before_completion_streams_to_finished_state() {
let context = test_context!();

26
lib/crates/fabro-cli/tests/it/cmd/config_show.rs → lib/crates/fabro-cli/tests/it/cmd/completion.rs
View file

@ -3,18 +3,18 @@ use fabro_test::{fabro_snapshot, test_context};
#[test]
fn help() {
let context = test_context!();
let mut cmd = context.settings();
cmd.arg("--help");
let mut cmd = context.command();
cmd.args(["completion", "--help"]);
fabro_snapshot!(context.filters(), cmd, @"
success: true
exit_code: 0
----- stdout -----
Inspect merged configuration
Generate shell completions
Usage: fabro settings [OPTIONS] [WORKFLOW]
Usage: fabro completion [OPTIONS] <SHELL>
Arguments:
[WORKFLOW] Optional workflow name, .fabro path, or .toml run config to overlay
<SHELL> Shell to generate completions for [possible values: bash, elvish, fish, powershell, zsh]
Options:
--debug Enable DEBUG-level logging (default is INFO) [env: FABRO_DEBUG=]
@ -26,3 +26,19 @@ fn help() {
----- stderr -----
");
}
#[test]
fn generates_zsh_completions() {
let context = test_context!();
let mut cmd = context.command();
cmd.args(["completion", "zsh"]);
cmd.assert().success();
}
#[test]
fn generates_fish_completions() {
let context = test_context!();
let mut cmd = context.command();
cmd.args(["completion", "fish"]);
cmd.assert().success();
}

118
lib/crates/fabro-cli/tests/it/cmd/create.rs
View file

@ -3,7 +3,9 @@ use serde_json::json;
use fabro_test::{fabro_snapshot, test_context};
use super::support::{fixture, output_stdout, read_json, resolve_run};
use crate::support::{fabro_json_snapshot, read_json};
use super::support::{fixture, output_stdout, resolve_run};
#[test]
fn help() {
@ -43,6 +45,118 @@ fn help() {
");
}
#[test]
fn create_persists_directory_workflow_slug_and_cached_graph() {
let context = test_context!();
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAA";
let workflow_path = context.temp_dir.join("sluggy/workflow.fabro");
context.write_temp(
"sluggy/workflow.fabro",
"\
digraph BarBaz {
start [shape=Mdiamond, label=\"Start\"]
exit [shape=Msquare, label=\"Exit\"]
start -> exit
}
",
);
context
.command()
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
run_id,
workflow_path.to_str().unwrap(),
])
.assert()
.success();
let run_dir = context.find_run_dir(run_id);
let run_record = read_json(run_dir.join("run.json"));
let cached_graph = std::fs::read_to_string(run_dir.join("workflow.fabro")).unwrap();
fabro_json_snapshot!(
context,
serde_json::json!({
"workflow_slug": run_record["workflow_slug"],
"graph_name": run_record["graph"]["name"],
"cached_graph_lines": cached_graph.lines().collect::<Vec<_>>(),
}),
@r#"
{
"workflow_slug": "sluggy",
"graph_name": "BarBaz",
"cached_graph_lines": [
"digraph BarBaz {",
" start [shape=Mdiamond, label=\"Start\"]",
" exit [shape=Msquare, label=\"Exit\"]",
" start -> exit",
"}"
]
}
"#
);
}
#[test]
fn create_persists_file_stem_slug_for_standalone_file() {
let context = test_context!();
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAB";
let workflow_path = context.temp_dir.join("alpha.fabro");
context.write_temp(
"alpha.fabro",
"\
digraph FooWorkflow {
start [shape=Mdiamond, label=\"Start\"]
exit [shape=Msquare, label=\"Exit\"]
start -> exit
}
",
);
context
.command()
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
run_id,
workflow_path.to_str().unwrap(),
])
.assert()
.success();
let run_dir = context.find_run_dir(run_id);
let run_record = read_json(run_dir.join("run.json"));
let cached_graph = std::fs::read_to_string(run_dir.join("workflow.fabro")).unwrap();
fabro_json_snapshot!(
context,
serde_json::json!({
"workflow_slug": run_record["workflow_slug"],
"graph_name": run_record["graph"]["name"],
"cached_graph_lines": cached_graph.lines().collect::<Vec<_>>(),
}),
@r#"
{
"workflow_slug": "alpha",
"graph_name": "FooWorkflow",
"cached_graph_lines": [
"digraph FooWorkflow {",
" start [shape=Mdiamond, label=\"Start\"]",
" exit [shape=Msquare, label=\"Exit\"]",
" start -> exit",
"}"
]
}
"#
);
}
#[test]
fn create_persists_requested_overrides_into_run_json() {
let context = test_context!();
@ -85,7 +199,7 @@ fn create_persists_requested_overrides_into_run_json() {
.expect("create should print a run ID")
.to_string();
let run = resolve_run(&context, &run_id);
let run_json = read_json(&run.run_dir.join("run.json"));
let run_json = read_json(run.run_dir.join("run.json"));
let labels = json!({
"env": run_json.pointer("/labels/env"),
"team": run_json.pointer("/labels/team"),

240
lib/crates/fabro-cli/tests/it/cmd/detached.rs
View file

@ -1,5 +1,7 @@
use fabro_test::{fabro_snapshot, test_context};
use crate::support::{fabro_json_snapshot, read_json};
#[test]
fn help() {
let context = test_context!();
@ -26,3 +28,241 @@ fn help() {
----- stderr -----
");
}
fn launcher_path(context: &fabro_test::TestContext, run_id: &str) -> std::path::PathBuf {
context
.storage_dir
.join("launchers")
.join(format!("{run_id}.json"))
}
#[test]
fn detached_uses_cached_graph_after_source_deleted() {
let context = test_context!();
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAF";
let workflow_path = context.temp_dir.join("workflow.fabro");
context.write_temp(
"workflow.fabro",
"\
digraph CachedGraph {
start [shape=Mdiamond, label=\"Start\"]
exit [shape=Msquare, label=\"Exit\"]
start -> exit
}
",
);
context
.command()
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
run_id,
workflow_path.to_str().unwrap(),
])
.assert()
.success();
let run_dir = context.find_run_dir(run_id);
std::fs::remove_file(&workflow_path).unwrap();
context
.command()
.args([
"__detached",
"--run-dir",
run_dir.to_str().unwrap(),
"--launcher-path",
launcher_path(&context, run_id).to_str().unwrap(),
])
.timeout(std::time::Duration::from_secs(15))
.assert()
.success();
let conclusion = read_json(run_dir.join("conclusion.json"));
fabro_json_snapshot!(
context,
serde_json::json!({
"status": conclusion["status"],
}),
@r#"
{
"status": "success"
}
"#
);
}
#[test]
fn detached_uses_snapshotted_app_id_for_github_credentials() {
let context = test_context!();
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAG";
let workflow_path = context.temp_dir.join("workflow.fabro");
context.write_home(
".fabro/user.toml",
"\
version = 1
[git]
app_id = \"snapshotted-app-id\"
",
);
context.write_temp(
"workflow.fabro",
"\
digraph GitHubApp {
start [shape=Mdiamond, label=\"Start\"]
exit [shape=Msquare, label=\"Exit\"]
start -> exit
}
",
);
context
.command()
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
run_id,
workflow_path.to_str().unwrap(),
])
.assert()
.success();
let run_dir = context.find_run_dir(run_id);
let run_record = read_json(run_dir.join("run.json"));
fabro_json_snapshot!(
context,
serde_json::json!({
"app_id": run_record["settings"]["git"]["app_id"],
}),
@r#"
{
"app_id": "snapshotted-app-id"
}
"#
);
context.write_home(".fabro/user.toml", "version = 1\n");
let mut cmd = context.command();
cmd.env("GITHUB_APP_PRIVATE_KEY", "%%%not-base64%%%");
cmd.args([
"__detached",
"--run-dir",
run_dir.to_str().unwrap(),
"--launcher-path",
launcher_path(&context, run_id).to_str().unwrap(),
]);
cmd.timeout(std::time::Duration::from_secs(10));
fabro_snapshot!(context.filters(), cmd, @"
success: false
exit_code: 1
----- stdout -----
----- stderr -----
error: GITHUB_APP_PRIVATE_KEY is not valid PEM or base64: Invalid symbol 37, offset 0.
");
}
#[test]
fn detached_resume_rejects_completed_run_without_mutating_it() {
let context = test_context!();
context.write_temp(
"workflow.fabro",
"\
digraph Test {
start [shape=Mdiamond, label=\"Start\"]
exit [shape=Msquare, label=\"Exit\"]
start -> exit
}
",
);
let run = context
.command()
.current_dir(&context.temp_dir)
.args([
"run",
"--dry-run",
"--auto-approve",
"--no-retro",
"--detach",
context.temp_dir.join("workflow.fabro").to_str().unwrap(),
])
.assert()
.success();
let run_id = String::from_utf8(run.get_output().stdout.clone())
.unwrap()
.trim()
.to_string();
context
.command()
.args(["wait", &run_id])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
let inspect_before = context
.command()
.args(["inspect", &run_id])
.assert()
.success();
let before: serde_json::Value =
serde_json::from_slice(&inspect_before.get_output().stdout).unwrap();
let before_summary = serde_json::json!({
"run_dir": before[0]["run_dir"],
"start_time": before[0]["start_record"]["start_time"],
"conclusion_timestamp": before[0]["conclusion"]["timestamp"],
"conclusion_status": before[0]["conclusion"]["status"],
});
let run_dir = before_summary["run_dir"].as_str().unwrap().to_string();
fabro_json_snapshot!(context, &before_summary, @r#"
{
"run_dir": "[DRY_RUN_DIR]",
"start_time": "[TIMESTAMP]",
"conclusion_timestamp": "[TIMESTAMP]",
"conclusion_status": "success"
}
"#);
let mut cmd = context.command();
cmd.args([
"__detached",
"--run-dir",
&run_dir,
"--launcher-path",
launcher_path(&context, &run_id).to_str().unwrap(),
"--resume",
]);
cmd.timeout(std::time::Duration::from_secs(10));
fabro_snapshot!(context.filters(), cmd, @"
success: false
exit_code: 1
----- stdout -----
----- stderr -----
error: Precondition failed: run already finished successfully — nothing to resume
");
let inspect_after = context
.command()
.args(["inspect", &run_id])
.assert()
.success();
let after: serde_json::Value =
serde_json::from_slice(&inspect_after.get_output().stdout).unwrap();
let after_summary = serde_json::json!({
"run_dir": after[0]["run_dir"],
"start_time": after[0]["start_record"]["start_time"],
"conclusion_timestamp": after[0]["conclusion"]["timestamp"],
"conclusion_status": after[0]["conclusion"]["status"],
});
assert_eq!(after_summary, before_summary);
}

63
lib/crates/fabro-cli/tests/it/cmd/fabro.rs
View file

@ -12,37 +12,38 @@ fn help() {
Usage: fabro [OPTIONS] <COMMAND>
Commands:
run Launch a workflow run
create Create a workflow run (allocate run dir, persist spec)
start Start a created workflow run (spawn engine process)
attach Attach to a running or finished workflow run
logs View the event log of a workflow run
resume Resume an interrupted workflow run
rewind Rewind a workflow run to an earlier checkpoint
fork Fork a workflow run from an earlier checkpoint into a new run
wait Block until a workflow run completes
preflight Validate run configuration without executing
validate Validate a workflow
graph Render a workflow graph as SVG or PNG
asset Inspect and copy run assets (screenshots, reports, traces)
store Export store-backed run state for debugging
rm Remove one or more workflow runs
inspect Show detailed information about a workflow run
model List and test LLM models
doctor Check environment and integration health
install Set up the Fabro environment (LLMs, certs, GitHub)
pr Pull request operations
secret Manage secrets in ~/.fabro/.env
settings Inspect merged configuration
workflow Workflow operations
discord Open the Discord community in the browser
docs Open the docs website in the browser
upgrade Upgrade fabro to the latest version
repo Repository commands
provider Provider operations
sandbox Sandbox operations (cp, ssh, preview)
system System maintenance commands
help Print this message or the help of the given subcommand(s)
run Launch a workflow run
create Create a workflow run (allocate run dir, persist spec)
start Start a created workflow run (spawn engine process)
attach Attach to a running or finished workflow run
logs View the event log of a workflow run
resume Resume an interrupted workflow run
rewind Rewind a workflow run to an earlier checkpoint
fork Fork a workflow run from an earlier checkpoint into a new run
wait Block until a workflow run completes
preflight Validate run configuration without executing
validate Validate a workflow
graph Render a workflow graph as SVG or PNG
asset Inspect and copy run assets (screenshots, reports, traces)
store Export store-backed run state for debugging
rm Remove one or more workflow runs
inspect Show detailed information about a workflow run
model List and test LLM models
doctor Check environment and integration health
install Set up the Fabro environment (LLMs, certs, GitHub)
pr Pull request operations
secret Manage secrets in ~/.fabro/.env
settings Inspect merged configuration
workflow Workflow operations
discord Open the Discord community in the browser
docs Open the docs website in the browser
upgrade Upgrade fabro to the latest version
repo Repository commands
provider Provider operations
sandbox Sandbox operations (cp, ssh, preview)
completion Generate shell completions
system System maintenance commands
help Print this message or the help of the given subcommand(s)
Options:
--debug Enable DEBUG-level logging (default is INFO) [env: FABRO_DEBUG=]

25
lib/crates/fabro-cli/tests/it/cmd/init.rs
View file

@ -1,25 +0,0 @@
use fabro_test::{fabro_snapshot, test_context};
#[test]
fn help() {
let context = test_context!();
let mut cmd = context.init_cmd();
cmd.arg("--help");
fabro_snapshot!(context.filters(), cmd, @"
success: true
exit_code: 0
----- stdout -----
Initialize a new project (deprecated: use `repo init`)
Usage: fabro init [OPTIONS]
Options:
--debug Enable DEBUG-level logging (default is INFO) [env: FABRO_DEBUG=]
--no-upgrade-check Disable automatic upgrade check [env: FABRO_NO_UPGRADE_CHECK=true]
--quiet Suppress non-essential output [env: FABRO_QUIET=]
--verbose Enable verbose output [env: FABRO_VERBOSE=]
--storage-dir <STORAGE_DIR> Storage directory (default: ~/.fabro) [env: FABRO_STORAGE_DIR=[STORAGE_DIR]]
-h, --help Print help
----- stderr -----
");
}

11
lib/crates/fabro-cli/tests/it/cmd/mod.rs
View file

@ -2,9 +2,8 @@ mod asset;
mod asset_cp;
mod asset_list;
mod attach;
mod completion;
mod config;
mod config_show;
mod cp;
mod create;
mod detached;
mod diff;
@ -15,7 +14,6 @@ mod exec;
mod fabro;
mod fork;
mod graph;
mod init;
mod inspect;
mod install;
mod llm;
@ -32,7 +30,6 @@ mod pr_list;
mod pr_merge;
mod pr_view;
mod preflight;
mod preview;
mod provider;
mod provider_login;
mod ps;
@ -43,6 +40,9 @@ mod resume;
mod rewind;
mod rm;
mod run;
mod sandbox_cp;
mod sandbox_preview;
mod sandbox_ssh;
mod secret;
mod secret_get;
mod secret_list;
@ -50,8 +50,7 @@ mod secret_rm;
mod secret_set;
mod send_analytics;
mod send_panic;
mod serve;
mod ssh;
mod server;
mod start;
mod store;
mod store_dump;

19
lib/crates/fabro-cli/tests/it/cmd/resume.rs
View file

@ -30,6 +30,25 @@ fn help() {
");
}
#[test]
fn resume_requires_run_arg() {
let context = test_context!();
let mut cmd = context.command();
cmd.args(["resume"]);
fabro_snapshot!(context.filters(), cmd, @"
success: false
exit_code: 2
----- stdout -----
----- stderr -----
error: the following required arguments were not provided:
<RUN>
Usage: fabro resume --no-upgrade-check --storage-dir <STORAGE_DIR> <RUN>
For more information, try '--help'.
");
}
#[test]
fn resume_rewound_run_succeeds() {
let context = test_context!();

1600
lib/crates/fabro-cli/tests/it/cmd/run.rs
View file

File diff suppressed because it is too large Load diff

4
lib/crates/fabro-cli/tests/it/cmd/cp.rs → lib/crates/fabro-cli/tests/it/cmd/sandbox_cp.rs
View file

@ -5,8 +5,8 @@ use super::support::{read_text, setup_asset_sandbox_run, setup_created_dry_run,
#[test]
fn help() {
let context = test_context!();
let mut cmd = context.cp();
cmd.arg("--help");
let mut cmd = context.command();
cmd.args(["sandbox", "cp", "--help"]);
fabro_snapshot!(context.filters(), cmd, @"
success: true
exit_code: 0

4
lib/crates/fabro-cli/tests/it/cmd/preview.rs → lib/crates/fabro-cli/tests/it/cmd/sandbox_preview.rs
View file

@ -5,8 +5,8 @@ use super::support::setup_asset_sandbox_run;
#[test]
fn help() {
let context = test_context!();
let mut cmd = context.preview();
cmd.arg("--help");
let mut cmd = context.command();
cmd.args(["sandbox", "preview", "--help"]);
fabro_snapshot!(context.filters(), cmd, @"
success: true
exit_code: 0

4
lib/crates/fabro-cli/tests/it/cmd/ssh.rs → lib/crates/fabro-cli/tests/it/cmd/sandbox_ssh.rs
View file

@ -5,8 +5,8 @@ use super::support::setup_asset_sandbox_run;
#[test]
fn help() {
let context = test_context!();
let mut cmd = context.ssh();
cmd.arg("--help");
let mut cmd = context.command();
cmd.args(["sandbox", "ssh", "--help"]);
fabro_snapshot!(context.filters(), cmd, @"
success: true
exit_code: 0

4
lib/crates/fabro-cli/tests/it/cmd/serve.rs → lib/crates/fabro-cli/tests/it/cmd/server.rs
View file

@ -5,14 +5,14 @@ fn help() {
let context = test_context!();
let mut cmd = context.command();
cmd.args(["serve", "--help"]);
cmd.args(["server", "start", "--help"]);
fabro_snapshot!(context.filters(), cmd, @"
success: true
exit_code: 0
----- stdout -----
Start the HTTP API server
Usage: fabro serve [OPTIONS]
Usage: fabro server start [OPTIONS]
Options:
--debug

127
lib/crates/fabro-cli/tests/it/cmd/start.rs
View file

@ -1,5 +1,7 @@
use fabro_test::{fabro_snapshot, test_context};
use crate::support::{example_fixture, fabro_json_snapshot, read_json};
use super::support::{output_stdout, resolve_run, wait_for_status, write_sleep_workflow};
#[test]
@ -29,6 +31,131 @@ fn help() {
");
}
#[test]
fn start_by_run_id_starts_created_run() {
let context = test_context!();
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAC";
context
.command()
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
run_id,
example_fixture("simple.fabro").to_str().unwrap(),
])
.assert()
.success();
context.command().args(["start", run_id]).assert().success();
context
.command()
.args(["wait", run_id])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
let run_dir = context.find_run_dir(run_id);
let status = read_json(run_dir.join("status.json"));
let conclusion = read_json(run_dir.join("conclusion.json"));
fabro_json_snapshot!(
context,
serde_json::json!({
"status": status["status"],
"reason": status["reason"],
"conclusion_status": conclusion["status"],
}),
@r#"
{
"status": "succeeded",
"reason": "completed",
"conclusion_status": "success"
}
"#
);
}
#[test]
fn start_by_workflow_name_prefers_newly_created_submitted_run() {
let context = test_context!();
let workflow_path = context.temp_dir.join("smoke/workflow.fabro");
let old_run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAD";
let new_run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAE";
context.write_temp(
"smoke/workflow.fabro",
"\
digraph Smoke {
start [shape=Mdiamond, label=\"Start\"]
work [label=\"Work\", prompt=\"Do the work.\"]
exit [shape=Msquare, label=\"Exit\"]
start -> work -> exit
}
",
);
context
.command()
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
old_run_id,
workflow_path.to_str().unwrap(),
])
.assert()
.success();
context
.command()
.args(["start", old_run_id])
.assert()
.success();
context
.command()
.args(["wait", old_run_id])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
context
.command()
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
new_run_id,
workflow_path.to_str().unwrap(),
])
.assert()
.success();
context
.command()
.args(["start", "smoke"])
.assert()
.success();
context
.command()
.args(["attach", new_run_id])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
let new_run_dir = context.find_run_dir(new_run_id);
let status = read_json(new_run_dir.join("status.json"));
fabro_json_snapshot!(context, &status, @r#"
{
"status": "succeeded",
"reason": "completed",
"updated_at": "[TIMESTAMP]"
}
"#);
}
#[test]
fn start_rejects_already_active_or_completed_run() {
let context = test_context!();

1
lib/crates/fabro-cli/tests/it/main.rs
View file

@ -1,3 +1,4 @@
mod cmd;
mod scenario;
mod support;
mod workflow;

318
lib/crates/fabro-cli/tests/it/scenario/lifecycle.rs
View file

@ -1,10 +1,8 @@
use std::path::PathBuf;
use fabro_store::RuntimeState;
use fabro_test::test_context;
use serde_json::Value;
use super::{fixture, read_json, timeout_for};
use crate::support::fabro_json_snapshot;
#[test]
#[ignore = "scenario: requires local sandbox"]
@ -61,12 +59,6 @@ fn local_run_lifecycle() {
items[0]["conclusion"].is_object(),
"inspect should include conclusion"
);
let run_dir = PathBuf::from(
items[0]["run_dir"]
.as_str()
.expect("inspect should include run_dir"),
);
// 4. logs <run_id> — non-empty, first line is valid JSONL with event field
let logs_out = cmd(&["logs", &run_id]).success();
let logs_stdout = String::from_utf8(logs_out.get_output().stdout.clone()).unwrap();
@ -87,95 +79,7 @@ fn local_run_lifecycle() {
"asset list should report no assets: {asset_list_stdout}"
);
// 6. Seed a synthetic asset so asset list/cp have something to work with.
let asset_dir = RuntimeState::new(&run_dir).asset_stage_dir("step1", 1);
std::fs::create_dir_all(&asset_dir).unwrap();
std::fs::write(asset_dir.join("output.txt"), "asset-content-42").unwrap();
std::fs::write(
asset_dir.join("manifest.json"),
r#"{"files_copied":1,"total_bytes":16,"files_skipped":0,"download_errors":0,"hash_errors":0,"captured_assets":[{"path":"output.txt","mime":"text/plain","content_md5":"f02439728c0a94b7bfc465acb1201a1f","content_sha256":"0af9dea3e1c2dec968531c18c9331659b8268e8c9cf24b01cda7b8ce51d2ff00","bytes":16}]}"#,
)
.unwrap();
let retry_two_dir = RuntimeState::new(&run_dir).asset_stage_dir("step1", 2);
std::fs::create_dir_all(&retry_two_dir).unwrap();
std::fs::write(retry_two_dir.join("output.txt"), "asset-content-84").unwrap();
std::fs::write(
retry_two_dir.join("manifest.json"),
r#"{"files_copied":1,"total_bytes":16,"files_skipped":0,"download_errors":0,"hash_errors":0,"captured_assets":[{"path":"output.txt","mime":"text/plain","content_md5":"5b4e23e40a1630f9caa15a4cb6cfb79b","content_sha256":"1f71e0df61fc3b4e1ee3aba7ceac9ae391af22595b5b5630d97d34cf33d4d540","bytes":16}]}"#,
)
.unwrap();
// 7. asset list — now shows the seeded assets
let asset_list_out2 = cmd(&["asset", "list", &run_id, "--json"]).success();
let asset_list_stdout2 =
String::from_utf8(asset_list_out2.get_output().stdout.clone()).unwrap();
let assets: Vec<Value> = serde_json::from_str(&asset_list_stdout2)
.expect("asset list --json should produce a JSON array");
assert_eq!(
assets.len(),
2,
"should have two assets: {asset_list_stdout2}"
);
assert_eq!(assets[0]["relative_path"].as_str(), Some("output.txt"));
assert_eq!(assets[0]["node_slug"].as_str(), Some("step1"));
let retry_filtered_out = cmd(&["asset", "list", &run_id, "--retry", "1", "--json"]).success();
let retry_filtered_stdout =
String::from_utf8(retry_filtered_out.get_output().stdout.clone()).unwrap();
let retry_filtered_assets: Vec<Value> = serde_json::from_str(&retry_filtered_stdout)
.expect("asset list --json should produce a JSON array");
assert_eq!(retry_filtered_assets.len(), 1);
assert_eq!(retry_filtered_assets[0]["retry"].as_u64(), Some(1));
// 8. asset cp — ambiguous without --retry when multiple retries captured the same path
let asset_dest = context.temp_dir.join("asset_copy");
cmd(&[
"asset",
"cp",
&format!("{run_id}:output.txt"),
asset_dest.to_str().unwrap(),
])
.failure();
cmd(&[
"asset",
"cp",
&format!("{run_id}:output.txt"),
asset_dest.to_str().unwrap(),
"--retry",
"1",
])
.success();
let copied = std::fs::read_to_string(asset_dest.join("output.txt")).unwrap();
assert_eq!(
copied, "asset-content-42",
"asset cp should copy file content"
);
// 9. cp — download a file from the local sandbox workdir
let sandbox_json: Value = read_json(&run_dir.join("sandbox.json"));
let workdir = sandbox_json["working_directory"]
.as_str()
.expect("sandbox.json should have working_directory");
// Plant a file in the sandbox workdir so we can download it
std::fs::write(
PathBuf::from(workdir).join("cp_test.txt"),
"downloaded-via-cp",
)
.unwrap();
let cp_dest = context.temp_dir.join("cp_download.txt");
cmd(&[
"sandbox",
"cp",
&format!("{run_id}:cp_test.txt"),
cp_dest.to_str().unwrap(),
])
.success();
let cp_content = std::fs::read_to_string(&cp_dest).unwrap();
assert_eq!(
cp_content, "downloaded-via-cp",
"cp should download file from sandbox"
);
// 10. system df — mentions "Runs"
// 6. system df — mentions "Runs"
let df_out = cmd(&["system", "df"]).success();
let df_stdout = String::from_utf8(df_out.get_output().stdout.clone()).unwrap();
assert!(
@ -183,10 +87,10 @@ fn local_run_lifecycle() {
"system df should mention Runs: {df_stdout}"
);
// 11. rm <run_id> — remove the run
// 7. rm <run_id> — remove the run
cmd(&["rm", &run_id]).success();
// 12. ps -a --json — should be empty
// 8. ps -a --json — should be empty
let ps_out2 = cmd(&["ps", "-a", "--json"]).success();
let ps_stdout2 = String::from_utf8(ps_out2.get_output().stdout.clone()).unwrap();
let runs2: Vec<Value> =
@ -196,3 +100,217 @@ fn local_run_lifecycle() {
"runs should be empty after rm: {ps_stdout2}"
);
}
#[test]
fn dry_run_create_start_attach_works_with_default_run_lookup() {
let context = test_context!();
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAJ";
context
.command()
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
run_id,
"../../../test/simple.fabro",
])
.assert()
.success();
context.command().args(["start", run_id]).assert().success();
context
.command()
.args(["attach", run_id])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
let run_dir = context.find_run_dir(run_id);
fabro_json_snapshot!(
context,
serde_json::json!({
"run_json_exists": run_dir.join("run.json").exists(),
"conclusion_json_exists": run_dir.join("conclusion.json").exists(),
}),
@r#"
{
"run_json_exists": true,
"conclusion_json_exists": true
}
"#
);
}
#[test]
fn dry_run_detach_attach_works_with_default_run_lookup() {
let context = test_context!();
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAK";
context
.command()
.args([
"run",
"--detach",
"--dry-run",
"--auto-approve",
"--run-id",
run_id,
"../../../test/simple.fabro",
])
.assert()
.success();
context
.command()
.args(["attach", run_id])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
let run_dir = context.find_run_dir(run_id);
fabro_json_snapshot!(
context,
serde_json::json!({
"run_dir": run_dir,
"conclusion_json_exists": run_dir.join("conclusion.json").exists(),
}),
@r#"
{
"run_dir": "[DRY_RUN_DIR]",
"conclusion_json_exists": true
}
"#
);
}
#[test]
fn completed_run_can_be_attached_by_workflow_slug() {
let context = test_context!();
let project = tempfile::tempdir().unwrap();
let workflow_dir = project.path().join("workflows").join("sluggy");
let workflow_path = workflow_dir.join("workflow.fabro");
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAQ";
std::fs::create_dir_all(&workflow_dir).unwrap();
std::fs::write(
&workflow_path,
"\
digraph BarBaz {
start [shape=Mdiamond, label=\"Start\"]
exit [shape=Msquare, label=\"Exit\"]
start -> exit
}
",
)
.unwrap();
context
.command()
.current_dir(project.path())
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
run_id,
workflow_path.to_str().unwrap(),
])
.assert()
.success();
context
.command()
.current_dir(project.path())
.args(["start", "sluggy"])
.assert()
.success();
context
.command()
.current_dir(project.path())
.args(["attach", run_id])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
context
.command()
.current_dir(project.path())
.args(["attach", "sluggy"])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
let run_record = read_json(&context.find_run_dir(run_id).join("run.json"));
fabro_json_snapshot!(
context,
serde_json::json!({
"graph_name": run_record["graph"]["name"],
"workflow_slug": run_record["workflow_slug"],
}),
@r#"
{
"graph_name": "BarBaz",
"workflow_slug": "sluggy"
}
"#
);
}
#[test]
fn completed_run_can_be_attached_by_file_stem() {
let context = test_context!();
let workflow_dir = tempfile::tempdir().unwrap();
let workflow_path = workflow_dir.path().join("alpha.fabro");
let run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAM";
std::fs::write(
&workflow_path,
"\
digraph FooWorkflow {
start [shape=Mdiamond, label=\"Start\"]
exit [shape=Msquare, label=\"Exit\"]
start -> exit
}
",
)
.unwrap();
context
.command()
.args([
"create",
"--dry-run",
"--auto-approve",
"--run-id",
run_id,
workflow_path.to_str().unwrap(),
])
.assert()
.success();
context
.command()
.args(["start", "alpha"])
.assert()
.success();
context
.command()
.args(["attach", "alpha"])
.timeout(std::time::Duration::from_secs(10))
.assert()
.success();
let run_record = read_json(&context.find_run_dir(run_id).join("run.json"));
fabro_json_snapshot!(
context,
serde_json::json!({
"graph_name": run_record["graph"]["name"],
"workflow_slug": run_record["workflow_slug"],
}),
@r#"
{
"graph_name": "FooWorkflow",
"workflow_slug": "alpha"
}
"#
);
}

1
lib/crates/fabro-cli/tests/it/scenario/mod.rs
View file

@ -1,5 +1,6 @@
mod exec;
mod lifecycle;
mod recovery;
use std::path::{Path, PathBuf};
use std::time::Duration;

278
lib/crates/fabro-cli/tests/it/scenario/recovery.rs Normal file
View file

@ -0,0 +1,278 @@
use std::collections::BTreeSet;
use std::path::Path;
use fabro_checkpoint::branch::BranchStore;
use fabro_checkpoint::git::Store as GitStore;
use fabro_test::{fabro_snapshot, test_context};
use fabro_types::Checkpoint;
use git2::{Repository, Signature};
use crate::support::read_jsonl;
fn list_metadata_run_ids(repo_dir: &Path) -> BTreeSet<String> {
let repo = Repository::discover(repo_dir).unwrap();
repo.references()
.unwrap()
.flatten()
.filter_map(|reference| reference.name().map(ToOwned::to_owned))
.filter_map(|name| {
name.strip_prefix("refs/heads/fabro/meta/")
.map(ToOwned::to_owned)
})
.collect()
}
fn metadata_checkpoints(repo_dir: &Path, run_id: &str) -> Vec<Checkpoint> {
let repo = Repository::discover(repo_dir).unwrap();
let store = GitStore::new(repo);
let sig = Signature::now("Fabro", "noreply@fabro.sh").unwrap();
let branch = format!("fabro/meta/{run_id}");
let bs = BranchStore::new(&store, &branch, &sig);
bs.log(100)
.unwrap()
.iter()
.rev()
.filter(|commit| commit.message.starts_with("checkpoint"))
.map(|commit| {
serde_json::from_slice::<Checkpoint>(
&store
.read_blob_at(commit.oid, "checkpoint.json")
.unwrap()
.unwrap(),
)
.unwrap()
})
.collect()
}
fn latest_metadata_checkpoint(repo_dir: &Path, run_id: &str) -> Checkpoint {
let repo = Repository::discover(repo_dir).unwrap();
let store = GitStore::new(repo);
let tip = store
.resolve_ref(&format!("fabro/meta/{run_id}"))
.unwrap()
.unwrap();
serde_json::from_slice(&store.read_blob_at(tip, "checkpoint.json").unwrap().unwrap()).unwrap()
}
fn run_commit_shas_by_node(run_dir: &Path) -> serde_json::Map<String, serde_json::Value> {
let mut shas_by_node = serde_json::Map::new();
for event in read_jsonl(run_dir.join("progress.jsonl")) {
if !matches!(event["event"].as_str(), Some("git.commit" | "GitCommit")) {
continue;
}
let Some(node_id) = event["node_id"].as_str() else {
continue;
};
let Some(sha) = event
.get("properties")
.and_then(|properties| properties.get("sha"))
.and_then(serde_json::Value::as_str)
.or_else(|| event["sha"].as_str())
else {
continue;
};
shas_by_node
.entry(node_id.to_string())
.or_insert_with(|| serde_json::Value::Array(Vec::new()))
.as_array_mut()
.unwrap()
.push(serde_json::Value::String(sha.to_string()));
}
shas_by_node
}
fn init_repo_with_workflow(repo_dir: &Path) {
std::fs::write(repo_dir.join("README.md"), "recovery test\n").unwrap();
std::fs::write(
repo_dir.join("workflow.fabro"),
"\
digraph Recovery {
start [shape=Mdiamond, label=\"Start\"]
exit [shape=Msquare, label=\"Exit\"]
plan [label=\"Plan\", shape=parallelogram, script=\"echo plan\"]
build [label=\"Build\", shape=parallelogram, script=\"echo build\"]
start -> plan -> build -> exit
}
",
)
.unwrap();
let init = std::process::Command::new("git")
.args(["init"])
.current_dir(repo_dir)
.status()
.unwrap();
assert!(init.success(), "git init should succeed");
let add = std::process::Command::new("git")
.args(["add", "README.md", "workflow.fabro"])
.current_dir(repo_dir)
.status()
.unwrap();
assert!(add.success(), "git add should succeed");
let commit = std::process::Command::new("git")
.args([
"-c",
"user.name=Fabro",
"-c",
"user.email=noreply@fabro.sh",
"commit",
"-m",
"init",
])
.current_dir(repo_dir)
.status()
.unwrap();
assert!(commit.success(), "git commit should succeed");
}
#[test]
fn rewind_and_fork_recover_missing_metadata_from_real_run_state() {
let context = test_context!();
let repo_dir = tempfile::tempdir().unwrap();
let source_run_id = "01ARZ3NDEKTSV4RRFFQ69G5FAN";
init_repo_with_workflow(repo_dir.path());
context
.command()
.current_dir(repo_dir.path())
.args([
"run",
"--dry-run",
"--no-retro",
"--sandbox",
"local",
"--run-id",
source_run_id,
"workflow.fabro",
])
.assert()
.success();
let run_dir = context.find_run_dir(source_run_id);
let run_shas = run_commit_shas_by_node(&run_dir);
let plan_sha = run_shas["plan"][0].as_str().unwrap().to_string();
let build_sha = run_shas["build"][0].as_str().unwrap().to_string();
let mut filters = Vec::new();
for (idx, sha) in [plan_sha.as_str(), build_sha.as_str()].iter().enumerate() {
let replacement = format!("[SHA_{}]", idx + 1);
filters.push((regex::escape(sha), replacement.clone()));
filters.push((regex::escape(&sha[..8]), replacement.clone()));
filters.push((regex::escape(&sha[..7]), replacement));
}
filters.extend(context.filters());
Repository::discover(repo_dir.path())
.unwrap()
.find_reference(&format!("refs/heads/fabro/meta/{source_run_id}"))
.unwrap()
.delete()
.unwrap();
assert!(
list_metadata_run_ids(repo_dir.path()).is_empty(),
"metadata branch should start missing"
);
let mut rewind_list = context.command();
rewind_list.current_dir(repo_dir.path());
rewind_list.args(["rewind", source_run_id, "--list"]);
rewind_list.timeout(std::time::Duration::from_secs(15));
fabro_snapshot!(filters.clone(), rewind_list, @"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
@ Node Details
@1 start (no run commit)
@2 plan
@3 build
");
let rebuilt_checkpoints = metadata_checkpoints(repo_dir.path(), source_run_id);
assert_eq!(rebuilt_checkpoints.len(), 3);
assert_eq!(rebuilt_checkpoints[0].git_commit_sha, None);
assert_eq!(
rebuilt_checkpoints[1].git_commit_sha.as_deref(),
Some(plan_sha.as_str())
);
assert_eq!(
rebuilt_checkpoints[2].git_commit_sha.as_deref(),
Some(build_sha.as_str())
);
let before_child = list_metadata_run_ids(repo_dir.path());
context
.command()
.current_dir(repo_dir.path())
.args(["fork", source_run_id, "--no-push"])
.timeout(std::time::Duration::from_secs(15))
.assert()
.success();
let after_child = list_metadata_run_ids(repo_dir.path());
let child_run_ids: Vec<_> = after_child.difference(&before_child).cloned().collect();
assert_eq!(child_run_ids.len(), 1, "expected one child run");
let child_run_id = &child_run_ids[0];
let child_checkpoint = latest_metadata_checkpoint(repo_dir.path(), child_run_id);
assert_eq!(
child_checkpoint.git_commit_sha.as_deref(),
Some(build_sha.as_str())
);
let mut rewind_filters = filters.clone();
rewind_filters.push((
regex::escape(&source_run_id[..8]),
"[RUN_PREFIX]".to_string(),
));
let mut source_rewind = context.command();
source_rewind.current_dir(repo_dir.path());
source_rewind.args(["rewind", source_run_id, "@2", "--no-push"]);
source_rewind.timeout(std::time::Duration::from_secs(15));
fabro_snapshot!(rewind_filters, source_rewind, @"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
Rewound metadata branch to @2 (plan)
Rewound run branch fabro/run/[ULID] to [SHA_1]
To resume: fabro resume [RUN_PREFIX]
");
let rewound_child = latest_metadata_checkpoint(repo_dir.path(), source_run_id);
assert_eq!(
rewound_child.git_commit_sha.as_deref(),
Some(plan_sha.as_str())
);
let before_grandchild = list_metadata_run_ids(repo_dir.path());
context
.command()
.current_dir(repo_dir.path())
.args(["fork", source_run_id, "--no-push"])
.timeout(std::time::Duration::from_secs(15))
.assert()
.success();
let after_grandchild = list_metadata_run_ids(repo_dir.path());
let grandchild_run_ids: Vec<_> = after_grandchild
.difference(&before_grandchild)
.cloned()
.collect();
assert_eq!(grandchild_run_ids.len(), 1, "expected one grandchild run");
let grandchild_checkpoint = latest_metadata_checkpoint(repo_dir.path(), &grandchild_run_ids[0]);
assert_eq!(
grandchild_checkpoint.git_commit_sha.as_deref(),
Some(plan_sha.as_str())
);
}

82
lib/crates/fabro-cli/tests/it/support/mod.rs Normal file
View file

@ -0,0 +1,82 @@
use std::path::{Path, PathBuf};
use fabro_test::TestContext;
use serde_json::Value;
macro_rules! fabro_json_snapshot {
($context:expr, $value:expr, @$snapshot:literal) => {{
let mut filters = $context.filters();
filters.push((
r"\b\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z\b".to_string(),
"[TIMESTAMP]".to_string(),
));
let filters: Vec<(&str, &str)> = filters
.iter()
.map(|(pattern, replacement)| (pattern.as_str(), replacement.as_str()))
.collect();
let rendered = serde_json::to_string_pretty(&$value).unwrap();
insta::with_settings!({ filters => filters }, {
insta::assert_snapshot!(rendered, @$snapshot);
});
}};
}
pub(crate) use fabro_json_snapshot;
pub(crate) fn example_fixture(name: &str) -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR")).join(format!("../../../test/{name}"))
}
pub(crate) fn read_json(path: impl AsRef<Path>) -> Value {
serde_json::from_str(&std::fs::read_to_string(path).unwrap()).unwrap()
}
pub(crate) fn read_jsonl(path: impl AsRef<Path>) -> Vec<Value> {
std::fs::read_to_string(path)
.unwrap()
.lines()
.map(serde_json::from_str)
.collect::<Result<Vec<_>, _>>()
.unwrap()
}
pub(crate) fn compact_progress_event(event: &Value) -> Value {
fn event_value<'a>(event: &'a Value, key: &str) -> Option<&'a Value> {
event
.get(key)
.or_else(|| {
event
.get("properties")
.and_then(|properties| properties.get(key))
})
.filter(|value| !value.is_null())
}
let mut compact = serde_json::Map::new();
for key in [
"event",
"provider",
"name",
"goal",
"node_id",
"node_label",
"handler_type",
"index",
"status",
"from_node",
"to_node",
"reason",
"artifact_count",
] {
if let Some(value) = event_value(event, key) {
compact.insert(key.to_string(), value.clone());
}
}
Value::Object(compact)
}
pub(crate) fn run_output_filters(context: &TestContext) -> Vec<(String, String)> {
let mut filters = context.filters();
filters.push((r"\b\d+ms\b".to_string(), "[TIME]".to_string()));
filters
}

164
lib/crates/fabro-cli/tests/it/workflow/dry_run_examples.rs Normal file
View file

@ -0,0 +1,164 @@
use fabro_test::{fabro_snapshot, test_context};
use crate::support::{example_fixture, run_output_filters};
#[test]
fn dry_run_branching() {
let context = test_context!();
let mut cmd = context.run_cmd();
cmd.args(["--dry-run", "--auto-approve"]);
cmd.arg(example_fixture("branching.fabro"));
fabro_snapshot!(run_output_filters(&context), cmd, @"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
Workflow: Branch (6 nodes, 6 edges)
Graph: ../../../test/branching.fabro
Goal: Implement and validate a feature
warning [node: implement]: Node 'implement' has goal_gate=true but no retry_target or fallback_retry_target (goal_gate_has_retry)
Sandbox: local (ready in [TIME])
✓ Start [TIME]
✓ Plan [TIME]
✓ Implement [TIME]
✓ Validate [TIME]
✓ Tests passing? [TIME]
✓ Exit [TIME]
=== Run Result ===
Run: [ULID]
Status: SUCCESS
Duration: [DURATION]
Run: [DRY_RUN_DIR]
=== Output ===
[Simulated] Response for stage: validate
");
}
#[test]
fn dry_run_conditions() {
let context = test_context!();
let mut cmd = context.run_cmd();
cmd.args(["--dry-run", "--auto-approve"]);
cmd.arg(example_fixture("conditions.fabro"));
fabro_snapshot!(run_output_filters(&context), cmd, @"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
Workflow: Conditions (5 nodes, 5 edges)
Graph: ../../../test/conditions.fabro
Goal: Test condition evaluation with OR and parentheses
Sandbox: local (ready in [TIME])
✓ start [TIME]
✓ Decide [TIME]
✓ Path B [TIME]
✓ exit [TIME]
=== Run Result ===
Run: [ULID]
Status: SUCCESS
Duration: [DURATION]
Run: [DRY_RUN_DIR]
=== Output ===
[Simulated] Response for stage: path_b
");
}
#[test]
fn dry_run_parallel() {
let context = test_context!();
let mut cmd = context.run_cmd();
cmd.args(["--dry-run", "--auto-approve"]);
cmd.arg(example_fixture("parallel.fabro"));
fabro_snapshot!(run_output_filters(&context), cmd, @"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
Workflow: Parallel (7 nodes, 7 edges)
Graph: ../../../test/parallel.fabro
Goal: Test parallel and fan-in execution
Sandbox: local (ready in [TIME])
✓ start [TIME]
✓ Fork Work [TIME]
✓ Merge Results [TIME]
✓ Review [TIME]
✓ exit [TIME]
=== Run Result ===
Run: [ULID]
Status: SUCCESS
Duration: [DURATION]
Run: [DRY_RUN_DIR]
=== Output ===
[Simulated] Response for stage: review
");
}
#[test]
fn dry_run_styled() {
let context = test_context!();
let mut cmd = context.run_cmd();
cmd.args(["--dry-run", "--auto-approve"]);
cmd.arg(example_fixture("styled.fabro"));
fabro_snapshot!(run_output_filters(&context), cmd, @"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
Workflow: Styled (5 nodes, 4 edges)
Graph: ../../../test/styled.fabro
Goal: Build a styled pipeline
Sandbox: local (ready in [TIME])
✓ start [TIME]
✓ Plan [TIME]
✓ Implement [TIME]
✓ Critical Review [TIME]
✓ exit [TIME]
=== Run Result ===
Run: [ULID]
Status: SUCCESS
Duration: [DURATION]
Run: [DRY_RUN_DIR]
=== Output ===
[Simulated] Response for stage: critical_review
");
}
#[test]
fn dry_run_legacy_tool() {
let context = test_context!();
let mut cmd = context.run_cmd();
cmd.args(["--dry-run", "--auto-approve"]);
cmd.arg(example_fixture("legacy_tool.fabro"));
fabro_snapshot!(run_output_filters(&context), cmd, @"
success: true
exit_code: 0
----- stdout -----
----- stderr -----
Workflow: LegacyTool (3 nodes, 2 edges)
Graph: ../../../test/legacy_tool.fabro
Goal: Verify backwards compatibility with old tool naming
Sandbox: local (ready in [TIME])
✓ Start [TIME]
✓ Echo [TIME]
✓ Exit [TIME]
=== Run Result ===
Run: [ULID]
Status: SUCCESS
Duration: [DURATION]
Run: [DRY_RUN_DIR]
");
}

8
lib/crates/fabro-cli/tests/it/workflow/full_stack.rs
View file

@ -51,12 +51,12 @@ fn scenario_full_stack(sandbox: &str) {
// Progress events
assert!(
has_event(&run_dir, "WorkflowRunStarted"),
"progress should contain WorkflowRunStarted"
has_event(&run_dir, "run.started"),
"progress should contain run.started"
);
assert!(
has_event(&run_dir, "WorkflowRunCompleted"),
"progress should contain WorkflowRunCompleted"
has_event(&run_dir, "run.completed"),
"progress should contain run.completed"
);
// All expected nodes completed

1
lib/crates/fabro-cli/tests/it/workflow/mod.rs
View file

@ -2,6 +2,7 @@ mod agent_linear;
mod command_agent_mixed;
mod command_pipeline;
mod conditional_branching;
mod dry_run_examples;
mod full_stack;
mod human_gate;

7
lib/crates/fabro-git-storage/src/lib.rs
View file

@ -1,7 +0,0 @@
pub mod branchstore;
pub mod error;
pub mod gitobj;
pub mod snapshot;
pub mod trailerlink;
pub use error::{Error, Result};

649
lib/crates/fabro-git-storage/src/snapshot.rs
View file

@ -1,649 +0,0 @@
use std::collections::BTreeMap;
use std::path::PathBuf;
use git2::{Oid, Signature};
use tracing::{debug, warn};
use crate::Result;
use crate::gitobj::{FileMode, Store, TreeEntries};
/// Options for writing a snapshot.
pub struct WriteOptions<'a> {
pub branch: String,
pub base_tree: Oid,
pub changes: FileChanges,
pub metadata: BTreeMap<String, Vec<u8>>,
pub metadata_from_disk: Option<DiskDir>,
pub author: Signature<'a>,
pub message: String,
pub deduplicate: bool,
}
/// File changes to apply from the working directory.
pub struct FileChanges {
pub modified: Vec<String>,
pub new: Vec<String>,
pub deleted: Vec<String>,
pub repo_root: PathBuf,
}
/// A directory on disk to walk and embed into the tree.
pub struct DiskDir {
pub disk_path: PathBuf,
pub tree_prefix: String,
}
/// Result of a snapshot write.
pub struct WriteResult {
pub commit_oid: Oid,
pub tree_oid: Oid,
pub skipped: bool,
}
/// Metadata about a snapshot commit.
#[derive(Debug)]
pub struct SnapshotInfo {
pub commit_oid: Oid,
pub tree_oid: Oid,
pub message: String,
pub time: git2::Time,
}
/// Captures full repo-state on named branches.
pub struct SnapshotStore<'a> {
objects: &'a Store,
}
impl<'a> SnapshotStore<'a> {
pub fn new(objects: &'a Store) -> Self {
Self { objects }
}
/// Write a snapshot to a branch.
pub fn write(&self, opts: &WriteOptions<'_>) -> Result<WriteResult> {
debug!(branch = %opts.branch, "Writing snapshot");
// 1. Resolve existing branch tip or use base_tree
let (base_tree_oid, parent_oid) = match self.objects.resolve_ref(&opts.branch)? {
Some(commit_oid) => {
let commit = self.objects.repo().find_commit(commit_oid)?;
(commit.tree_id(), Some(commit_oid))
}
None => (opts.base_tree, None),
};
// 2. Flatten base tree
let mut entries = self.objects.read_tree(base_tree_oid)?;
// 3. Apply FileChanges
for path in &opts.changes.deleted {
entries.remove(path);
}
for path in opts.changes.modified.iter().chain(opts.changes.new.iter()) {
let full_path = opts.changes.repo_root.join(path);
match self.objects.write_blob_from_file(&full_path) {
Ok((oid, mode)) => {
entries.set(path.clone(), oid, mode);
}
Err(crate::Error::ReadFile { .. }) => {
// File disappeared since detection — treat as deleted
warn!(path = %path, "File disappeared since detection, treating as deleted");
entries.remove(path);
}
Err(e) => return Err(e),
}
}
// 4. Apply in-memory metadata
for (path, content) in &opts.metadata {
let oid = self.objects.write_blob(content)?;
entries.set(path.clone(), oid, FileMode::Blob);
}
// 5. Walk metadata_from_disk
if let Some(disk_dir) = &opts.metadata_from_disk {
self.walk_disk_dir(&mut entries, disk_dir)?;
}
// 6. Write tree
let new_tree_oid = self.objects.write_tree(&entries)?;
// 7. Dedup check
if opts.deduplicate {
if let Some(parent) = parent_oid {
let parent_commit = self.objects.repo().find_commit(parent)?;
if parent_commit.tree_id() == new_tree_oid {
debug!(branch = %opts.branch, "Snapshot skipped (tree unchanged)");
return Ok(WriteResult {
commit_oid: parent,
tree_oid: new_tree_oid,
skipped: true,
});
}
}
}
// 8. Create commit
let parents: Vec<Oid> = parent_oid.into_iter().collect();
let commit_oid =
self.objects
.write_commit(new_tree_oid, &parents, &opts.message, &opts.author)?;
// 9. Update ref
self.objects.update_ref(&opts.branch, commit_oid)?;
debug!(branch = %opts.branch, commit = %commit_oid, "Snapshot written");
Ok(WriteResult {
commit_oid,
tree_oid: new_tree_oid,
skipped: false,
})
}
/// Tip commit of a snapshot branch. `None` if branch doesn't exist.
pub fn latest(&self, branch: &str) -> Result<Option<SnapshotInfo>> {
let Some(commit_oid) = self.objects.resolve_ref(branch)? else {
return Ok(None);
};
let commit = self.objects.repo().find_commit(commit_oid)?;
let tree_oid = commit.tree_id();
let message = commit.message().unwrap_or("").to_string();
let time = commit.author().when();
Ok(Some(SnapshotInfo {
commit_oid,
tree_oid,
message,
time,
}))
}
/// Read a single file from a snapshot commit's tree.
pub fn read_file(&self, commit_oid: Oid, path: &str) -> Result<Option<Vec<u8>>> {
let commit = self.objects.repo().find_commit(commit_oid)?;
let tree = commit.tree()?;
match tree.get_path(std::path::Path::new(path)) {
Ok(entry) => {
let blob = self.objects.repo().find_blob(entry.id())?;
Ok(Some(blob.content().to_vec()))
}
Err(e) if e.code() == git2::ErrorCode::NotFound => Ok(None),
Err(e) => Err(e.into()),
}
}
/// Walk commits on a snapshot branch, newest first.
pub fn list_commits(&self, branch: &str, limit: usize) -> Result<Vec<SnapshotInfo>> {
let Some(commit_oid) = self.objects.resolve_ref(branch)? else {
return Ok(vec![]);
};
let mut revwalk = self.objects.repo().revwalk()?;
revwalk.set_sorting(git2::Sort::TIME | git2::Sort::TOPOLOGICAL)?;
revwalk.push(commit_oid)?;
let mut results = Vec::new();
for oid_result in revwalk.take(limit) {
let oid = oid_result?;
let commit = self.objects.repo().find_commit(oid)?;
results.push(SnapshotInfo {
commit_oid: oid,
tree_oid: commit.tree_id(),
message: commit.message().unwrap_or("").to_string(),
time: commit.author().when(),
});
}
Ok(results)
}
/// Check if a snapshot branch exists.
pub fn exists(&self, branch: &str) -> Result<bool> {
Ok(self.objects.resolve_ref(branch)?.is_some())
}
/// Delete a snapshot branch.
pub fn delete(&self, branch: &str) -> Result<()> {
debug!(branch = %branch, "Deleting snapshot branch");
self.objects.delete_ref(branch)
}
/// Rename a snapshot branch.
pub fn rename(&self, old: &str, new: &str) -> Result<()> {
debug!(old = %old, new = %new, "Renaming snapshot branch");
let oid = self
.objects
.resolve_ref(old)?
.ok_or_else(|| crate::Error::BranchNotFound {
branch: old.to_string(),
})?;
self.objects.update_ref(new, oid)?;
self.objects.delete_ref(old)?;
Ok(())
}
/// List snapshot branches matching a prefix.
pub fn list(&self, prefix: &str) -> Result<Vec<String>> {
let full_prefix = format!("refs/heads/{prefix}");
let mut branches = Vec::new();
for reference in self
.objects
.repo()
.references_glob(&format!("{full_prefix}*"))?
{
let reference = reference?;
if let Some(name) = reference.name() {
if let Some(branch) = name.strip_prefix("refs/heads/") {
branches.push(branch.to_string());
}
}
}
branches.sort();
Ok(branches)
}
/// Walk a directory on disk and add files to tree entries.
fn walk_disk_dir(&self, entries: &mut TreeEntries, disk_dir: &DiskDir) -> Result<()> {
let walker = walkdir::WalkDir::new(&disk_dir.disk_path)
.follow_links(false)
.into_iter()
.filter_map(std::result::Result::ok);
for entry in walker {
// Skip symlinks
if entry.path_is_symlink() {
continue;
}
// Skip directories
if entry.file_type().is_dir() {
continue;
}
let relative = entry
.path()
.strip_prefix(&disk_dir.disk_path)
.unwrap_or(entry.path());
let tree_path = if disk_dir.tree_prefix.is_empty() {
relative.to_string_lossy().to_string()
} else {
format!(
"{}/{}",
disk_dir.tree_prefix.trim_end_matches('/'),
relative.to_string_lossy()
)
};
let (oid, mode) = self.objects.write_blob_from_file(entry.path())?;
entries.set(tree_path, oid, mode);
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use git2::Repository;
fn temp_repo() -> (tempfile::TempDir, Store) {
let dir = tempfile::TempDir::new().unwrap();
let repo = Repository::init(dir.path()).unwrap();
(dir, Store::new(repo))
}
fn test_sig() -> Signature<'static> {
Signature::now("Test", "test@example.com").unwrap()
}
fn empty_changes() -> FileChanges {
FileChanges {
modified: vec![],
new: vec![],
deleted: vec![],
repo_root: PathBuf::from("/tmp"),
}
}
// -- write creates branch + commit from base tree --
#[test]
fn write_creates_branch_from_base_tree() {
let (_dir, store) = temp_repo();
let sig = test_sig();
let snap = SnapshotStore::new(&store);
// Create a base tree with one file
let blob_oid = store.write_blob(b"base content").unwrap();
let mut base_entries = TreeEntries::new();
base_entries.set("existing.txt", blob_oid, FileMode::Blob);
let base_tree = store.write_tree(&base_entries).unwrap();
let result = snap
.write(&WriteOptions {
branch: "snap/test".to_string(),
base_tree,
changes: empty_changes(),
metadata: BTreeMap::new(),
metadata_from_disk: None,
author: sig,
message: "snapshot 1".to_string(),
deduplicate: false,
})
.unwrap();
assert!(!result.skipped);
assert!(store.resolve_ref("snap/test").unwrap().is_some());
// Verify the file is in the snapshot
let content = snap.read_file(result.commit_oid, "existing.txt").unwrap();
assert_eq!(content.unwrap(), b"base content");
}
// -- write applies file changes --
#[test]
fn write_applies_file_changes() {
let (dir, store) = temp_repo();
let sig = test_sig();
let snap = SnapshotStore::new(&store);
// Create files on disk in the repo root
let repo_root = dir.path().to_path_buf();
std::fs::write(repo_root.join("new_file.txt"), b"new content").unwrap();
std::fs::write(repo_root.join("modified.txt"), b"modified content").unwrap();
// Create base tree with a file to delete and one to modify
let old_blob = store.write_blob(b"old content").unwrap();
let delete_blob = store.write_blob(b"delete me").unwrap();
let mut base_entries = TreeEntries::new();
base_entries.set("modified.txt", old_blob, FileMode::Blob);
base_entries.set("to_delete.txt", delete_blob, FileMode::Blob);
let base_tree = store.write_tree(&base_entries).unwrap();
let result = snap
.write(&WriteOptions {
branch: "snap/changes".to_string(),
base_tree,
changes: FileChanges {
modified: vec!["modified.txt".to_string()],
new: vec!["new_file.txt".to_string()],
deleted: vec!["to_delete.txt".to_string()],
repo_root,
},
metadata: BTreeMap::new(),
metadata_from_disk: None,
author: sig,
message: "apply changes".to_string(),
deduplicate: false,
})
.unwrap();
assert_eq!(
snap.read_file(result.commit_oid, "modified.txt")
.unwrap()
.unwrap(),
b"modified content"
);
assert_eq!(
snap.read_file(result.commit_oid, "new_file.txt")
.unwrap()
.unwrap(),
b"new content"
);
assert!(
snap.read_file(result.commit_oid, "to_delete.txt")
.unwrap()
.is_none()
);
}
// -- write embeds in-memory metadata --
#[test]
fn write_embeds_metadata() {
let (_dir, store) = temp_repo();
let sig = test_sig();
let snap = SnapshotStore::new(&store);
let base_tree = store.write_empty_tree().unwrap();
let mut metadata = BTreeMap::new();
metadata.insert(
".meta/transcript.jsonl".to_string(),
b"line1\nline2".to_vec(),
);
let result = snap
.write(&WriteOptions {
branch: "snap/meta".to_string(),
base_tree,
changes: empty_changes(),
metadata,
metadata_from_disk: None,
author: sig,
message: "with metadata".to_string(),
deduplicate: false,
})
.unwrap();
let content = snap
.read_file(result.commit_oid, ".meta/transcript.jsonl")
.unwrap()
.unwrap();
assert_eq!(content, b"line1\nline2");
}
// -- write dedup skips when tree unchanged --
#[test]
fn write_dedup_skips_unchanged() {
let (_dir, store) = temp_repo();
let snap = SnapshotStore::new(&store);
let base_tree = store.write_empty_tree().unwrap();
let sig = test_sig();
// First write
let result1 = snap
.write(&WriteOptions {
branch: "snap/dedup".to_string(),
base_tree,
changes: empty_changes(),
metadata: BTreeMap::new(),
metadata_from_disk: None,
author: sig.clone(),
message: "first".to_string(),
deduplicate: true,
})
.unwrap();
assert!(!result1.skipped);
// Second write with same content — should be skipped
let sig2 = test_sig();
let result2 = snap
.write(&WriteOptions {
branch: "snap/dedup".to_string(),
base_tree,
changes: empty_changes(),
metadata: BTreeMap::new(),
metadata_from_disk: None,
author: sig2,
message: "second".to_string(),
deduplicate: true,
})
.unwrap();
assert!(result2.skipped);
assert_eq!(result2.commit_oid, result1.commit_oid);
}
// -- latest / read_file / list_commits --
#[test]
fn latest_and_list_commits() {
let (_dir, store) = temp_repo();
let snap = SnapshotStore::new(&store);
let base_tree = store.write_empty_tree().unwrap();
// Write two snapshots
let sig1 = test_sig();
snap.write(&WriteOptions {
branch: "snap/history".to_string(),
base_tree,
changes: empty_changes(),
metadata: BTreeMap::from([("a.txt".to_string(), b"a".to_vec())]),
metadata_from_disk: None,
author: sig1,
message: "first".to_string(),
deduplicate: false,
})
.unwrap();
let sig2 = test_sig();
snap.write(&WriteOptions {
branch: "snap/history".to_string(),
base_tree,
changes: empty_changes(),
metadata: BTreeMap::from([("b.txt".to_string(), b"b".to_vec())]),
metadata_from_disk: None,
author: sig2,
message: "second".to_string(),
deduplicate: false,
})
.unwrap();
let latest = snap.latest("snap/history").unwrap().unwrap();
assert_eq!(latest.message, "second");
let commits = snap.list_commits("snap/history", 10).unwrap();
assert_eq!(commits.len(), 2);
assert_eq!(commits[0].message, "second");
assert_eq!(commits[1].message, "first");
}
#[test]
fn latest_nonexistent() {
let (_dir, store) = temp_repo();
let snap = SnapshotStore::new(&store);
assert!(snap.latest("nonexistent").unwrap().is_none());
}
// -- exists / delete / rename / list --
#[test]
fn exists_and_delete() {
let (_dir, store) = temp_repo();
let snap = SnapshotStore::new(&store);
let base_tree = store.write_empty_tree().unwrap();
let sig = test_sig();
snap.write(&WriteOptions {
branch: "snap/del".to_string(),
base_tree,
changes: empty_changes(),
metadata: BTreeMap::new(),
metadata_from_disk: None,
author: sig,
message: "create".to_string(),
deduplicate: false,
})
.unwrap();
assert!(snap.exists("snap/del").unwrap());
snap.delete("snap/del").unwrap();
assert!(!snap.exists("snap/del").unwrap());
}
#[test]
fn rename_branch() {
let (_dir, store) = temp_repo();
let snap = SnapshotStore::new(&store);
let base_tree = store.write_empty_tree().unwrap();
let sig = test_sig();
snap.write(&WriteOptions {
branch: "snap/old".to_string(),
base_tree,
changes: empty_changes(),
metadata: BTreeMap::from([("file.txt".to_string(), b"data".to_vec())]),
metadata_from_disk: None,
author: sig,
message: "create".to_string(),
deduplicate: false,
})
.unwrap();
snap.rename("snap/old", "snap/new").unwrap();
assert!(!snap.exists("snap/old").unwrap());
assert!(snap.exists("snap/new").unwrap());
// Verify data is preserved
let info = snap.latest("snap/new").unwrap().unwrap();
let content = snap.read_file(info.commit_oid, "file.txt").unwrap();
assert_eq!(content.unwrap(), b"data");
}
#[test]
fn list_branches() {
let (_dir, store) = temp_repo();
let snap = SnapshotStore::new(&store);
let base_tree = store.write_empty_tree().unwrap();
// Create several branches
for name in &["snap/a", "snap/b", "other/c"] {
let sig = test_sig();
snap.write(&WriteOptions {
branch: name.to_string(),
base_tree,
changes: empty_changes(),
metadata: BTreeMap::new(),
metadata_from_disk: None,
author: sig,
message: "create".to_string(),
deduplicate: false,
})
.unwrap();
}
let snap_branches = snap.list("snap/").unwrap();
assert_eq!(snap_branches, vec!["snap/a", "snap/b"]);
}
// -- metadata_from_disk --
#[test]
fn write_metadata_from_disk() {
let (_dir, store) = temp_repo();
let snap = SnapshotStore::new(&store);
let base_tree = store.write_empty_tree().unwrap();
// Create a temp directory with files
let meta_dir = tempfile::TempDir::new().unwrap();
std::fs::write(meta_dir.path().join("info.json"), b"{}").unwrap();
std::fs::create_dir(meta_dir.path().join("sub")).unwrap();
std::fs::write(meta_dir.path().join("sub/data.txt"), b"nested").unwrap();
let sig = test_sig();
let result = snap
.write(&WriteOptions {
branch: "snap/disk".to_string(),
base_tree,
changes: empty_changes(),
metadata: BTreeMap::new(),
metadata_from_disk: Some(DiskDir {
disk_path: meta_dir.path().to_path_buf(),
tree_prefix: ".meta".to_string(),
}),
author: sig,
message: "from disk".to_string(),
deduplicate: false,
})
.unwrap();
assert_eq!(
snap.read_file(result.commit_oid, ".meta/info.json")
.unwrap()
.unwrap(),
b"{}"
);
assert_eq!(
snap.read_file(result.commit_oid, ".meta/sub/data.txt")
.unwrap()
.unwrap(),
b"nested"
);
}
}

6
lib/crates/fabro-retro/src/retro.rs
View file

@ -51,14 +51,16 @@ pub fn extract_stage_durations(run_dir: &Path) -> HashMap<String, u64> {
let Ok(envelope) = serde_json::from_str::<serde_json::Value>(line) else {
continue;
};
if envelope.get("event").and_then(|v| v.as_str()) != Some("StageCompleted") {
if envelope.get("event").and_then(|v| v.as_str()) != Some("stage.completed") {
continue;
}
let Some(name) = envelope.get("node_id").and_then(|v| v.as_str()) else {
continue;
};
let Some(duration_ms) = envelope
.get("duration_ms")
.get("properties")
.and_then(serde_json::Value::as_object)
.and_then(|properties| properties.get("duration_ms"))
.and_then(serde_json::Value::as_u64)
else {
continue;

1
lib/crates/fabro-retro/src/retro_agent.rs
View file

@ -649,6 +649,7 @@ mod tests {
event: AgentEvent::SessionStarted,
timestamp: SystemTime::now(),
session_id: "retro-test".into(),
parent_session_id: None,
})
.unwrap();

6
lib/crates/fabro-server/src/server.rs
View file

@ -45,7 +45,7 @@ use crate::sessions::{SessionStore, new_session_store};
use fabro_interview::{Answer, Interviewer, QuestionType, WebInterviewer};
use fabro_retro::RetroExt;
use fabro_workflow::context::Context;
use fabro_workflow::event::{EventEmitter, WorkflowRunEvent};
use fabro_workflow::event::{EventEmitter, RunEventEnvelope};
use fabro_workflow::operations::{self, CreateRunInput, WorkflowInput};
use fabro_workflow::pipeline::Persisted;
use fabro_workflow::records::{Checkpoint, CheckpointExt};
@ -92,7 +92,7 @@ struct ManagedRun {
created_at: chrono::DateTime<chrono::Utc>,
// Populated when running:
interviewer: Option<Arc<WebInterviewer>>,
event_tx: Option<broadcast::Sender<WorkflowRunEvent>>,
event_tx: Option<broadcast::Sender<RunEventEnvelope>>,
context: Option<Context>,
checkpoint: Option<Checkpoint>,
cancel_tx: Option<oneshot::Sender<()>>,
@ -627,7 +627,7 @@ async fn execute_run(state: Arc<AppState>, run_id: RunId) {
// Create interviewer and event plumbing (this is the "provisioning" phase)
let interviewer = Arc::new(WebInterviewer::new());
let context = Context::new();
let emitter = EventEmitter::new();
let emitter = EventEmitter::new(run_id);
if let Some(tx_clone) = event_tx {
emitter.on_event(move |event| {
let _ = tx_clone.send(event.clone());

1
lib/crates/fabro-store/src/disk_projecting.rs
View file

@ -512,6 +512,7 @@ mod tests {
fn event_payload(run_id: &str, ts: &str, event: &str) -> EventPayload {
EventPayload::new(
serde_json::json!({
"id": format!("evt-{run_id}-{event}"),
"ts": ts,
"run_id": test_run_id(run_id).to_string(),
"event": event,

3
lib/crates/fabro-store/src/memory.rs
View file

@ -986,6 +986,7 @@ mod tests {
assert!(matches!(err, StoreError::InvalidEvent(_)));
let invalid_run_id: EventPayload = serde_json::from_value(serde_json::json!({
"id": "evt-invalid-run",
"ts": "2026-03-27T12:00:00Z",
"run_id": "other-run",
"event": "StageStarted"
@ -1022,6 +1023,7 @@ mod tests {
.unwrap();
let first = EventPayload::new(
serde_json::json!({
"id": "evt-1",
"ts": "2026-03-27T12:00:00.000Z",
"run_id": test_run_id("run-1").to_string(),
"event": "WorkflowRunStarted"
@ -1031,6 +1033,7 @@ mod tests {
.unwrap();
let second = EventPayload::new(
serde_json::json!({
"id": "evt-2",
"ts": "2026-03-27T12:00:01.000Z",
"run_id": test_run_id("run-1").to_string(),
"event": "StageCompleted"

1
lib/crates/fabro-store/src/slate/mod.rs
View file

@ -496,6 +496,7 @@ mod tests {
fn event_payload(run_id: &str, ts: &str, event: &str) -> EventPayload {
EventPayload::new(
serde_json::json!({
"id": format!("evt-{run_id}-{event}"),
"ts": ts,
"run_id": test_run_id(run_id).to_string(),
"event": event

2
lib/crates/fabro-store/src/types.rs
View file

@ -81,7 +81,7 @@ impl EventPayload {
StoreError::InvalidEvent("event payload must be a JSON object".into())
})?;
for field in ["ts", "run_id", "event"] {
for field in ["id", "ts", "run_id", "event"] {
match obj.get(field) {
Some(serde_json::Value::String(_)) => {}
_ => {

60
lib/crates/fabro-test/src/lib.rs
View file

@ -180,13 +180,6 @@ impl TestContext {
cmd
}
/// Build a `sandbox cp` subcommand.
pub fn cp(&self) -> Command {
let mut cmd = self.command();
cmd.args(["sandbox", "cp"]);
cmd
}
/// Build an `init` subcommand.
pub fn init_cmd(&self) -> Command {
let mut cmd = self.command();
@ -208,13 +201,6 @@ impl TestContext {
cmd
}
/// Build a `sandbox preview` subcommand.
pub fn preview(&self) -> Command {
let mut cmd = self.command();
cmd.args(["sandbox", "preview"]);
cmd
}
/// Build a `repo` subcommand.
pub fn repo(&self) -> Command {
let mut cmd = self.command();
@ -222,13 +208,6 @@ impl TestContext {
cmd
}
/// Build a `sandbox ssh` subcommand.
pub fn ssh(&self) -> Command {
let mut cmd = self.command();
cmd.args(["sandbox", "ssh"]);
cmd
}
/// Build a `system` subcommand.
pub fn system(&self) -> Command {
let mut cmd = self.command();
@ -277,6 +256,45 @@ impl TestContext {
std::fs::write(&full, content).expect("failed to write file");
self
}
/// Find a run directory whose name ends with `run_id_suffix`.
pub fn find_run_dir(&self, run_id_suffix: &str) -> PathBuf {
let runs_dir = self.storage_dir.join("runs");
std::fs::read_dir(&runs_dir)
.expect("runs directory should exist")
.flatten()
.map(|entry| entry.path())
.find(|path| {
path.is_dir()
&& path
.file_name()
.is_some_and(|name| name.to_string_lossy().ends_with(run_id_suffix))
})
.unwrap_or_else(|| {
panic!(
"expected run directory for {run_id_suffix} under {}",
runs_dir.display()
)
})
}
/// Return the only run directory currently present under storage.
pub fn single_run_dir(&self) -> PathBuf {
let runs_dir = self.storage_dir.join("runs");
let entries: Vec<_> = std::fs::read_dir(&runs_dir)
.expect("runs directory should exist")
.flatten()
.map(|entry| entry.path())
.filter(|path| path.is_dir())
.collect();
assert_eq!(
entries.len(),
1,
"expected exactly one run directory under {}",
runs_dir.display()
);
entries.into_iter().next().unwrap()
}
}
/// Execute a command and format the output for snapshot testing.

2
lib/crates/fabro-workflow/Cargo.toml
View file

@ -30,7 +30,7 @@ fabro-mcp = { path = "../fabro-mcp" }
fabro-github = { path = "../fabro-github" }
fabro-interview = { path = "../fabro-interview" }
fabro-util = { path = "../fabro-util" }
fabro-git-storage = { path = "../fabro-git-storage" }
fabro-checkpoint = { path = "../fabro-checkpoint" }
fabro-llm = { path = "../fabro-llm" }
fabro-model = { path = "../fabro-model" }
fabro-retro = { path = "../fabro-retro" }

56
lib/crates/fabro-workflow/src/devcontainer_bridge.rs
View file

@ -348,7 +348,7 @@ mod tests {
#[tokio::test]
async fn shell_command_executed() {
let sandbox = TestSandbox::new();
let emitter = EventEmitter::new();
let emitter = EventEmitter::default();
let commands = vec![fabro_devcontainer::Command::Shell("echo hi".to_string())];
run_devcontainer_lifecycle(&sandbox, &emitter, "on_create", &commands, 300_000)
.await
@ -361,7 +361,7 @@ mod tests {
#[tokio::test]
async fn args_command_joins() {
let sandbox = TestSandbox::new();
let emitter = EventEmitter::new();
let emitter = EventEmitter::default();
let commands = vec![fabro_devcontainer::Command::Args(vec![
"echo".to_string(),
"hi".to_string(),
@ -380,8 +380,8 @@ mod tests {
#[tokio::test]
async fn emits_started_and_completed_events() {
let emitter = EventEmitter::new();
let events = Arc::new(Mutex::new(Vec::new()));
let emitter = EventEmitter::default();
let events = Arc::new(Mutex::new(Vec::<crate::event::RunEventEnvelope>::new()));
let events_clone = Arc::clone(&events);
emitter.on_event(move |event| {
events_clone.lock().unwrap().push(event.clone());
@ -392,28 +392,27 @@ mod tests {
.await
.unwrap();
let events = events.lock().unwrap();
assert!(matches!(
&events[0],
WorkflowRunEvent::DevcontainerLifecycleStarted { phase, command_count } if phase == "on_create" && *command_count == 1
));
assert!(matches!(
&events[1],
WorkflowRunEvent::DevcontainerLifecycleCommandStarted { phase, index, .. } if phase == "on_create" && *index == 0
));
assert!(matches!(
&events[2],
WorkflowRunEvent::DevcontainerLifecycleCommandCompleted { phase, index, exit_code, .. } if phase == "on_create" && *index == 0 && *exit_code == 0
));
assert!(matches!(
&events[3],
WorkflowRunEvent::DevcontainerLifecycleCompleted { phase, .. } if phase == "on_create"
));
assert_eq!(events[0].event, "devcontainer.lifecycle.started");
assert_eq!(events[0].properties["phase"], "on_create");
assert_eq!(events[0].properties["command_count"], 1);
assert_eq!(events[1].event, "devcontainer.lifecycle.command.started");
assert_eq!(events[1].properties["phase"], "on_create");
assert_eq!(events[1].properties["index"], 0);
assert_eq!(events[2].event, "devcontainer.lifecycle.command.completed");
assert_eq!(events[2].properties["phase"], "on_create");
assert_eq!(events[2].properties["index"], 0);
assert_eq!(events[2].properties["exit_code"], 0);
assert_eq!(events[3].event, "devcontainer.lifecycle.completed");
assert_eq!(events[3].properties["phase"], "on_create");
}
#[tokio::test]
async fn failed_command_emits_failed_and_returns_error() {
let emitter = EventEmitter::new();
let events = Arc::new(Mutex::new(Vec::new()));
let emitter = EventEmitter::default();
let events = Arc::new(Mutex::new(Vec::<crate::event::RunEventEnvelope>::new()));
let events_clone = Arc::clone(&events);
emitter.on_event(move |event| {
events_clone.lock().unwrap().push(event.clone());
@ -424,15 +423,16 @@ mod tests {
run_devcontainer_lifecycle(&sandbox, &emitter, "on_create", &commands, 300_000).await;
assert!(result.is_err());
let events = events.lock().unwrap();
assert!(events.iter().any(|e| matches!(
e,
WorkflowRunEvent::DevcontainerLifecycleFailed { phase, exit_code, .. } if phase == "on_create" && *exit_code == 1
)));
assert!(events.iter().any(|event| {
event.event == "devcontainer.lifecycle.failed"
&& event.properties["phase"] == "on_create"
&& event.properties["exit_code"] == 1
}));
}
#[tokio::test]
async fn empty_commands_is_noop() {
let emitter = EventEmitter::new();
let emitter = EventEmitter::default();
let events = Arc::new(Mutex::new(Vec::new()));
let events_clone = Arc::clone(&events);
emitter.on_event(move |event| {
@ -448,7 +448,7 @@ mod tests {
#[tokio::test]
async fn parallel_commands_run() {
let sandbox = TestSandbox::new();
let emitter = EventEmitter::new();
let emitter = EventEmitter::default();
let mut map = HashMap::new();
map.insert("install".to_string(), "npm install".to_string());
map.insert("build".to_string(), "npm run build".to_string());

46
lib/crates/fabro-workflow/src/error.rs
View file

@ -343,12 +343,26 @@ impl From<fabro_validate::ValidationError> for FabroError {
}
}
impl From<fabro_checkpoint::MetadataError> for FabroError {
fn from(err: fabro_checkpoint::MetadataError) -> Self {
let message = err.to_string();
match err {
fabro_checkpoint::MetadataError::Deserialize {
entity: "checkpoint",
..
} => Self::Checkpoint(message),
_ => Self::engine(message),
}
}
}
pub type Result<T> = std::result::Result<T, FabroError>;
#[cfg(test)]
mod tests {
use super::*;
use crate::outcome::OutcomeExt;
use fabro_checkpoint::MetadataError;
use fabro_llm::error::ProviderErrorDetail;
#[test]
@ -419,6 +433,38 @@ mod tests {
assert!(err.is_err());
}
#[test]
fn metadata_checkpoint_deserialize_error_preserves_source_detail() {
let source = serde_json::from_str::<serde_json::Value>("not json").unwrap_err();
let source_message = source.to_string();
let fabro_error = FabroError::from(MetadataError::Deserialize {
entity: "checkpoint",
branch: "fabro/meta/run-1".to_string(),
source,
});
assert!(matches!(fabro_error, FabroError::Checkpoint(_)));
let message = fabro_error.to_string();
assert!(message.contains("deserialize checkpoint on branch fabro/meta/run-1"));
assert!(message.contains(&source_message));
}
#[test]
fn metadata_non_checkpoint_deserialize_error_maps_to_engine_with_source_detail() {
let source = serde_json::from_str::<serde_json::Value>("not json").unwrap_err();
let source_message = source.to_string();
let fabro_error = FabroError::from(MetadataError::Deserialize {
entity: "run record",
branch: "fabro/meta/run-1".to_string(),
source,
});
assert!(matches!(fabro_error, FabroError::Engine { .. }));
let message = fabro_error.to_string();
assert!(message.contains("deserialize run record on branch fabro/meta/run-1"));
assert!(message.contains(&source_message));
}
#[test]
fn cancelled_error_display() {
let err = FabroError::Cancelled;

2378
lib/crates/fabro-workflow/src/event.rs
View file

File diff suppressed because it is too large Load diff

402
lib/crates/fabro-workflow/src/git.rs
View file

@ -1,77 +1,20 @@
use std::fmt::Write;
use std::path::Path;
use std::process::Command;
use fabro_checkpoint::git::Store;
use fabro_config::FabroSettings;
use fabro_config::server::GitAuthorSettings;
use fabro_git_storage::branchstore::BranchStore;
use fabro_git_storage::gitobj::Store;
use git2::{Repository, Signature};
use crate::error::{FabroError, Result};
use crate::records::{Checkpoint, RunRecord, StartRecord};
use tokio::task::{JoinError, spawn_blocking};
use tokio::time::timeout;
pub use fabro_checkpoint::META_BRANCH_PREFIX;
pub use fabro_checkpoint::author::GitAuthor;
pub use fabro_checkpoint::metadata::MetadataStore;
/// Branch prefix for workflow run branches (e.g. `fabro/run/{run_id}`).
pub const RUN_BRANCH_PREFIX: &str = "fabro/run/";
/// Branch prefix for metadata branches (e.g. `fabro/meta/{run_id}`).
pub const META_BRANCH_PREFIX: &str = "fabro/meta/";
/// Resolved git author identity for checkpoint commits.
#[derive(Debug, Clone, PartialEq)]
pub struct GitAuthor {
pub name: String,
pub email: String,
}
impl Default for GitAuthor {
fn default() -> Self {
Self {
name: "Fabro".into(),
email: "noreply@fabro.sh".into(),
}
}
}
impl GitAuthor {
/// Create a `GitAuthor` from optional name/email, falling back to defaults.
pub fn from_options(name: Option<String>, email: Option<String>) -> Self {
let defaults = Self::default();
Self {
name: name.unwrap_or(defaults.name),
email: email.unwrap_or(defaults.email),
}
}
/// Returns true when this identity matches the default Fabro identity.
pub fn is_default(&self) -> bool {
let defaults = Self::default();
self.name == defaults.name && self.email == defaults.email
}
/// Append the Fabro footer (and Co-Authored-By when the author is not the
/// default identity) to a commit message.
pub fn append_footer(&self, message: &mut String) {
message.push_str("\n\u{2692}\u{fe0f} Generated with [Fabro](https://fabro.sh)\n");
if !self.is_default() {
let defaults = Self::default();
let _ = write!(
message,
"\nCo-Authored-By: {} <{}>\n",
defaults.name, defaults.email
);
}
}
}
impl From<&GitAuthorSettings> for GitAuthor {
fn from(value: &GitAuthorSettings) -> Self {
Self::from_options(value.name.clone(), value.email.clone())
}
}
pub fn git_author_from_settings(settings: &FabroSettings) -> GitAuthor {
settings
.git_author()
@ -409,160 +352,11 @@ pub fn scan_node_files(run_dir: &Path) -> Vec<(String, Vec<u8>)> {
result
}
/// Git-native metadata storage for pipeline runs.
///
/// Stores checkpoint data, run records, and metadata on an orphan branch
/// (`fabro/meta/{run_id}`) so that runs can be resumed from git alone.
pub struct MetadataStore {
repo_path: std::path::PathBuf,
author: GitAuthor,
}
impl MetadataStore {
pub fn new(repo_path: impl Into<std::path::PathBuf>, author: &GitAuthor) -> Self {
Self {
repo_path: repo_path.into(),
author: author.clone(),
}
}
/// Returns the branch name for a run: `fabro/meta/{run_id}`.
pub fn branch_name(run_id: &str) -> String {
format!("{META_BRANCH_PREFIX}{run_id}")
}
/// Format a commit message with the standard Fabro footer appended.
fn commit_message(&self, subject: &str) -> String {
let mut msg = format!("{subject}\n");
self.author.append_footer(&mut msg);
msg
}
fn open_store(&self) -> Result<(Store, Signature<'static>)> {
let repo = Repository::discover(&self.repo_path)
.map_err(|e| git_error(format!("failed to open repo: {e}")))?;
let store = Store::new(repo);
let sig = Signature::now(&self.author.name, &self.author.email)
.map_err(|e| git_error(format!("failed to create signature: {e}")))?;
Ok((store, sig))
}
/// Initialize a run's metadata branch with the given files.
///
/// Callers pass all files (run.json, start.json, sandbox.json, etc.)
/// via the `files` slice.
pub fn init_run(&self, run_id: &str, files: &[(&str, &[u8])]) -> Result<()> {
let (store, sig) = self.open_store()?;
let branch = Self::branch_name(run_id);
let bs = BranchStore::new(&store, &branch, &sig);
bs.ensure_branch()
.map_err(|e| git_error(format!("ensure_branch failed: {e}")))?;
let msg = self.commit_message("init run");
bs.write_entries(files, &msg)
.map_err(|e| git_error(format!("write_entries failed: {e}")))?;
Ok(())
}
/// Write arbitrary files to the metadata branch without overwriting checkpoint.json.
pub fn write_files(
&self,
run_id: &str,
entries: &[(&str, &[u8])],
message: &str,
) -> Result<()> {
let (store, sig) = self.open_store()?;
let branch = Self::branch_name(run_id);
let bs = BranchStore::new(&store, &branch, &sig);
let msg = self.commit_message(message);
bs.write_entries(entries, &msg)
.map_err(|e| git_error(format!("write_entries failed: {e}")))?;
Ok(())
}
/// Write checkpoint data (and optional artifacts) to the metadata branch.
/// Returns the SHA of the new commit on the shadow branch.
pub fn write_checkpoint(
&self,
run_id: &str,
checkpoint_json: &[u8],
artifacts: &[(&str, &[u8])],
) -> Result<String> {
let (store, sig) = self.open_store()?;
let branch = Self::branch_name(run_id);
let bs = BranchStore::new(&store, &branch, &sig);
let mut entries: Vec<(&str, &[u8])> = vec![("checkpoint.json", checkpoint_json)];
entries.extend_from_slice(artifacts);
let msg = self.commit_message("checkpoint");
let oid = bs
.write_entries(&entries, &msg)
.map_err(|e| git_error(format!("write_entries failed: {e}")))?;
Ok(oid.to_string())
}
/// Read a single file from the metadata branch. Returns `None` if branch or path doesn't exist.
fn read_file(repo_path: &Path, run_id: &str, path: &str) -> Result<Option<Vec<u8>>> {
let Ok(repo) = Repository::discover(repo_path) else {
return Ok(None);
};
let store = Store::new(repo);
let sig = Signature::now("Fabro", "noreply@fabro.sh")
.map_err(|e| git_error(format!("failed to create signature: {e}")))?;
let branch = Self::branch_name(run_id);
let bs = BranchStore::new(&store, &branch, &sig);
bs.read_entry(path)
.map_err(|e| git_error(format!("read_entry failed: {e}")))
}
/// Read a checkpoint from the metadata branch. Returns `None` if branch or file doesn't exist.
pub fn read_checkpoint(repo_path: &Path, run_id: &str) -> Result<Option<Checkpoint>> {
match Self::read_file(repo_path, run_id, "checkpoint.json")? {
Some(bytes) => {
let cp: Checkpoint = serde_json::from_slice(&bytes)
.map_err(|e| FabroError::Checkpoint(format!("deserialize failed: {e}")))?;
Ok(Some(cp))
}
None => Ok(None),
}
}
/// Read the run record from the metadata branch. Returns `None` if not found.
pub fn read_run_record(repo_path: &Path, run_id: &str) -> Result<Option<RunRecord>> {
match Self::read_file(repo_path, run_id, "run.json")? {
Some(bytes) => {
let record: RunRecord = serde_json::from_slice(&bytes)
.map_err(|e| git_error(format!("run record deserialize failed: {e}")))?;
Ok(Some(record))
}
None => Ok(None),
}
}
/// Read the start record from the metadata branch. Returns `None` if not found.
pub fn read_start_record(repo_path: &Path, run_id: &str) -> Result<Option<StartRecord>> {
match Self::read_file(repo_path, run_id, "start.json")? {
Some(bytes) => {
let record: StartRecord = serde_json::from_slice(&bytes)
.map_err(|e| git_error(format!("start record deserialize failed: {e}")))?;
Ok(Some(record))
}
None => Ok(None),
}
}
/// Read an artifact from the metadata branch. Returns `None` if not found.
pub fn read_artifact(repo_path: &Path, run_id: &str, key: &str) -> Result<Option<Vec<u8>>> {
Self::read_file(repo_path, run_id, &format!("artifacts/{key}.json"))
}
}
#[cfg(test)]
mod tests {
use super::*;
use fabro_types::fixtures;
use std::fs;
use crate::records::{CheckpointExt, RunRecordExt};
/// Create a temporary git repo with an initial commit.
fn init_repo(dir: &Path) {
Command::new("git")
@ -647,142 +441,6 @@ mod tests {
assert!(!wt_path.exists());
}
// --- MetadataStore tests ---
#[test]
fn metadata_store_init_run_and_read() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
let run_id = fixtures::RUN_1.to_string();
let run_record = format!(
r#"{{"run_id":"{run_id}","created_at":"2025-01-01T00:00:00Z","settings":{{}},"graph":{{"name":"test","nodes":{{}},"edges":[],"attrs":{{}}}},"working_directory":"/tmp"}}"#
);
store
.init_run(&run_id, &[("run.json", run_record.as_bytes())])
.unwrap();
let read_record = MetadataStore::read_run_record(dir.path(), &run_id)
.unwrap()
.unwrap();
assert_eq!(read_record.run_id, fixtures::RUN_1);
assert_eq!(read_record.workflow_name(), "test");
}
#[test]
fn metadata_store_write_and_read_checkpoint() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
store.init_run("RUN2", &[]).unwrap();
let ctx = crate::context::Context::new();
ctx.set("goal", serde_json::json!("test"));
let cp = crate::records::Checkpoint::from_context(
&ctx,
"node_a",
vec!["start".to_string()],
std::collections::HashMap::new(),
std::collections::HashMap::new(),
Some("node_b".to_string()),
std::collections::HashMap::new(),
std::collections::HashMap::new(),
std::collections::HashMap::new(),
);
let cp_json = serde_json::to_vec_pretty(&cp).unwrap();
store.write_checkpoint("RUN2", &cp_json, &[]).unwrap();
let loaded = MetadataStore::read_checkpoint(dir.path(), "RUN2")
.unwrap()
.unwrap();
assert_eq!(loaded.current_node, "node_a");
assert_eq!(loaded.completed_nodes, vec!["start"]);
assert_eq!(loaded.next_node_id.as_deref(), Some("node_b"));
assert_eq!(
loaded.context_values.get("goal"),
Some(&serde_json::json!("test"))
);
}
#[test]
fn metadata_store_write_checkpoint_overwrites() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
store.init_run("RUN3", &[]).unwrap();
let ctx = crate::context::Context::new();
let cp1 = crate::records::Checkpoint::from_context(
&ctx,
"node_a",
vec!["start".to_string()],
std::collections::HashMap::new(),
std::collections::HashMap::new(),
None,
std::collections::HashMap::new(),
std::collections::HashMap::new(),
std::collections::HashMap::new(),
);
let cp1_json = serde_json::to_vec_pretty(&cp1).unwrap();
store.write_checkpoint("RUN3", &cp1_json, &[]).unwrap();
let cp2 = crate::records::Checkpoint::from_context(
&ctx,
"node_b",
vec!["start".to_string(), "node_a".to_string()],
std::collections::HashMap::new(),
std::collections::HashMap::new(),
Some("node_c".to_string()),
std::collections::HashMap::new(),
std::collections::HashMap::new(),
std::collections::HashMap::new(),
);
let cp2_json = serde_json::to_vec_pretty(&cp2).unwrap();
store.write_checkpoint("RUN3", &cp2_json, &[]).unwrap();
let loaded = MetadataStore::read_checkpoint(dir.path(), "RUN3")
.unwrap()
.unwrap();
assert_eq!(loaded.current_node, "node_b");
assert_eq!(loaded.completed_nodes.len(), 2);
}
#[test]
fn metadata_store_read_checkpoint_missing_branch() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let result = MetadataStore::read_checkpoint(dir.path(), "NONEXISTENT").unwrap();
assert!(result.is_none());
}
#[test]
fn metadata_store_artifact_roundtrip() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
store.init_run("RUN4", &[]).unwrap();
let artifact_data = br#"{"large_output":"some data"}"#;
let cp_json = b"{}"; // minimal checkpoint for the test
store
.write_checkpoint(
"RUN4",
cp_json,
&[("artifacts/response.plan.json", artifact_data.as_slice())],
)
.unwrap();
let read_back = MetadataStore::read_artifact(dir.path(), "RUN4", "response.plan")
.unwrap()
.unwrap();
assert_eq!(read_back, artifact_data);
}
#[test]
fn scan_node_files_picks_up_allowlisted() {
let dir = tempfile::tempdir().unwrap();
@ -834,56 +492,6 @@ mod tests {
assert!(files.is_empty());
}
#[test]
fn metadata_store_write_files() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
let run_id = fixtures::RUN_5.to_string();
let run_record = format!(
r#"{{"run_id":"{run_id}","created_at":"2025-01-01T00:00:00Z","settings":{{}},"graph":{{"name":"test","nodes":{{}},"edges":[],"attrs":{{}}}},"working_directory":"/tmp"}}"#
);
store
.init_run(&run_id, &[("run.json", run_record.as_bytes())])
.unwrap();
store
.write_files(
&run_id,
&[("retro.json", b"{\"status\":\"ok\"}")],
"finalize",
)
.unwrap();
let data = MetadataStore::read_file(dir.path(), &run_id, "retro.json")
.unwrap()
.unwrap();
assert_eq!(data, b"{\"status\":\"ok\"}");
// Original files still present
let record = MetadataStore::read_run_record(dir.path(), &run_id)
.unwrap()
.unwrap();
assert_eq!(record.run_id, fixtures::RUN_5);
}
#[test]
fn metadata_store_init_run_with_extra_files() {
let dir = tempfile::tempdir().unwrap();
init_repo(dir.path());
let store = MetadataStore::new(dir.path(), &GitAuthor::default());
store
.init_run("RUN6", &[("sandbox.json", b"{\"type\":\"local\"}")])
.unwrap();
let data = MetadataStore::read_file(dir.path(), "RUN6", "sandbox.json")
.unwrap()
.unwrap();
assert_eq!(data, b"{\"type\":\"local\"}");
}
#[test]
fn sanitize_ref_component_lowercases() {
assert_eq!(sanitize_ref_component("Hello"), "hello");

114
lib/crates/fabro-workflow/src/handler/llm/api.rs
View file

@ -48,8 +48,6 @@ struct FileTracking {
last: Option<String>,
}
/// Recursively extract file-tracking events from agent events, including
/// those wrapped in one or more layers of `SubAgentEvent`.
fn track_file_event(event: &AgentEvent, state: &mut FileTracking) {
match event {
AgentEvent::ToolCallStarted {
@ -75,9 +73,6 @@ fn track_file_event(event: &AgentEvent, state: &mut FileTracking) {
}
}
}
AgentEvent::SubAgentEvent { event: inner, .. } => {
track_file_event(inner, state);
}
_ => {}
}
}
@ -116,6 +111,8 @@ fn spawn_event_forwarder(
emitter.emit(&WorkflowRunEvent::Agent {
stage: node_id.clone(),
event: event.event.clone(),
session_id: Some(event.session_id.clone()),
parent_session_id: event.parent_session_id.clone(),
});
}
}
@ -261,7 +258,7 @@ impl AgentApiBackend {
manager_for_callback
.lock()
.await
.set_event_callback(session.event_callback());
.set_event_callback(session.sub_agent_event_callback());
Ok(session)
}
@ -730,7 +727,7 @@ mod tests {
}
#[test]
fn track_file_event_unwraps_sub_agent_edit() {
fn track_file_event_tracks_edit_file() {
let mut state = new_file_tracking();
let mut args = serde_json::Map::new();
@ -739,32 +736,22 @@ mod tests {
serde_json::Value::String("/src/lib.rs".to_string()),
);
// ToolCallStarted wrapped in SubAgentEvent
track_file_event(
&AgentEvent::SubAgentEvent {
agent_id: "sub-1".to_string(),
depth: 1,
event: Box::new(AgentEvent::ToolCallStarted {
tool_name: "edit_file".to_string(),
tool_call_id: "tc-sub".to_string(),
arguments: serde_json::Value::Object(args),
}),
&AgentEvent::ToolCallStarted {
tool_name: "edit_file".to_string(),
tool_call_id: "tc-sub".to_string(),
arguments: serde_json::Value::Object(args),
},
&mut state,
);
assert_eq!(state.pending.get("tc-sub").unwrap(), "/src/lib.rs");
// ToolCallCompleted wrapped in SubAgentEvent
track_file_event(
&AgentEvent::SubAgentEvent {
agent_id: "sub-1".to_string(),
depth: 1,
event: Box::new(AgentEvent::ToolCallCompleted {
tool_call_id: "tc-sub".to_string(),
tool_name: "edit_file".to_string(),
is_error: false,
output: serde_json::Value::String("ok".to_string()),
}),
&AgentEvent::ToolCallCompleted {
tool_call_id: "tc-sub".to_string(),
tool_name: "edit_file".to_string(),
is_error: false,
output: serde_json::Value::String("ok".to_string()),
},
&mut state,
);
@ -772,55 +759,6 @@ mod tests {
assert_eq!(state.last.as_deref(), Some("/src/lib.rs"));
}
#[test]
fn track_file_event_unwraps_nested_sub_sub_agent() {
let mut state = new_file_tracking();
let mut args = serde_json::Map::new();
args.insert(
"file_path".to_string(),
serde_json::Value::String("/deep/file.rs".to_string()),
);
// Double-wrapped SubAgentEvent → SubAgentEvent → ToolCallStarted
track_file_event(
&AgentEvent::SubAgentEvent {
agent_id: "sub-outer".to_string(),
depth: 1,
event: Box::new(AgentEvent::SubAgentEvent {
agent_id: "sub-inner".to_string(),
depth: 2,
event: Box::new(AgentEvent::ToolCallStarted {
tool_name: "write_file".to_string(),
tool_call_id: "tc-deep".to_string(),
arguments: serde_json::Value::Object(args),
}),
}),
},
&mut state,
);
assert!(state.pending.contains_key("tc-deep"));
track_file_event(
&AgentEvent::SubAgentEvent {
agent_id: "sub-outer".to_string(),
depth: 1,
event: Box::new(AgentEvent::SubAgentEvent {
agent_id: "sub-inner".to_string(),
depth: 2,
event: Box::new(AgentEvent::ToolCallCompleted {
tool_call_id: "tc-deep".to_string(),
tool_name: "write_file".to_string(),
is_error: false,
output: serde_json::Value::String("ok".to_string()),
}),
}),
},
&mut state,
);
assert!(state.touched.contains("/deep/file.rs"));
}
#[test]
fn track_file_event_error_removes_pending() {
let mut state = new_file_tracking();
@ -832,28 +770,20 @@ mod tests {
);
track_file_event(
&AgentEvent::SubAgentEvent {
agent_id: "sub-1".to_string(),
depth: 1,
event: Box::new(AgentEvent::ToolCallStarted {
tool_name: "edit_file".to_string(),
tool_call_id: "tc-err".to_string(),
arguments: serde_json::Value::Object(args),
}),
&AgentEvent::ToolCallStarted {
tool_name: "edit_file".to_string(),
tool_call_id: "tc-err".to_string(),
arguments: serde_json::Value::Object(args),
},
&mut state,
);
track_file_event(
&AgentEvent::SubAgentEvent {
agent_id: "sub-1".to_string(),
depth: 1,
event: Box::new(AgentEvent::ToolCallCompleted {
tool_call_id: "tc-err".to_string(),
tool_name: "edit_file".to_string(),
is_error: true,
output: serde_json::Value::String("failed".to_string()),
}),
&AgentEvent::ToolCallCompleted {
tool_call_id: "tc-err".to_string(),
tool_name: "edit_file".to_string(),
is_error: true,
output: serde_json::Value::String("failed".to_string()),
},
&mut state,
);

6
lib/crates/fabro-workflow/src/handler/llm/cli.rs
View file

@ -969,7 +969,7 @@ mod tests {
#[allow(unsafe_code)]
async fn ensure_cli_skips_install_when_present() {
let sandbox: Arc<dyn Sandbox> = Arc::new(CliMockSandbox::new(vec![ok_result()]));
let emitter = Arc::new(EventEmitter::new());
let emitter = Arc::new(EventEmitter::default());
let result = ensure_cli(AgentCli::Claude, Provider::Anthropic, &sandbox, &emitter).await;
assert!(result.is_ok());
@ -988,7 +988,7 @@ mod tests {
fail_result(127), // claude --version
ok_result(), // combined node + npm install
]));
let emitter = Arc::new(EventEmitter::new());
let emitter = Arc::new(EventEmitter::default());
let result = ensure_cli(AgentCli::Claude, Provider::Anthropic, &sandbox, &emitter).await;
assert!(result.is_ok());
@ -1005,7 +1005,7 @@ mod tests {
fail_result(127), // claude --version
fail_result(1), // combined install fails
]));
let emitter = Arc::new(EventEmitter::new());
let emitter = Arc::new(EventEmitter::default());
let result = ensure_cli(AgentCli::Claude, Provider::Anthropic, &sandbox, &emitter).await;
assert!(result.is_err());

13
lib/crates/fabro-workflow/src/handler/manager_loop.rs
View file

@ -4,12 +4,6 @@ use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use async_trait::async_trait;
use chrono::Utc;
use fabro_config::FabroSettings;
use fabro_store::{InMemoryStore, Store};
use fabro_types::RunId;
use crate::condition::evaluate_condition;
use crate::context::keys;
use crate::context::{Context, WorkflowContext};
@ -20,7 +14,11 @@ use crate::pipeline;
use crate::pipeline::types::Initialized;
use crate::run_dir::visit_from_context;
use crate::run_options::RunOptions;
use async_trait::async_trait;
use chrono::Utc;
use fabro_config::FabroSettings;
use fabro_graphviz::graph::{AttrValue, Graph, Node};
use fabro_store::{InMemoryStore, Store};
use tokio::time::{sleep, timeout};
use super::{EngineServices, Handler};
@ -166,7 +164,8 @@ impl Handler for SubWorkflowHandler {
settings: fabro_config::FabroSettings::default(),
run_dir: child_logs,
cancel_token: Some(cancel_token),
run_id: RunId::new(),
// Child workflows are part of the parent run's event stream.
run_id: services.emitter.run_id(),
labels: HashMap::new(),
workflow_slug: None,
github_app: None,

2
lib/crates/fabro-workflow/src/handler/mod.rs
View file

@ -71,7 +71,7 @@ impl EngineServices {
pub fn test_default() -> Self {
Self {
registry: Arc::new(HandlerRegistry::new(Box::new(start::StartHandler))),
emitter: Arc::new(EventEmitter::new()),
emitter: Arc::new(EventEmitter::default()),
sandbox: Arc::new(fabro_agent::LocalSandbox::new(
std::env::current_dir().unwrap_or_else(|_| std::path::PathBuf::from(".")),
)),

9
lib/crates/fabro-workflow/src/lib.rs
View file

@ -113,13 +113,18 @@ pub fn extract_stage_durations_from_events(events: &[EventEnvelope]) -> HashMap<
let mut durations = HashMap::new();
for envelope in events {
let value = envelope.payload.as_value();
if value.get("event").and_then(serde_json::Value::as_str) != Some("StageCompleted") {
if value.get("event").and_then(serde_json::Value::as_str) != Some("stage.completed") {
continue;
}
let Some(node_id) = value.get("node_id").and_then(serde_json::Value::as_str) else {
continue;
};
let Some(duration_ms) = value.get("duration_ms").and_then(serde_json::Value::as_u64) else {
let Some(duration_ms) = value
.get("properties")
.and_then(serde_json::Value::as_object)
.and_then(|properties| properties.get("duration_ms"))
.and_then(serde_json::Value::as_u64)
else {
continue;
};
durations.insert(node_id.to_string(), duration_ms);

4
lib/crates/fabro-workflow/src/operations/fork.rs
View file

@ -1,6 +1,6 @@
use anyhow::{Context, Result};
use fabro_git_storage::branchstore::BranchStore;
use fabro_git_storage::gitobj::Store;
use fabro_checkpoint::branch::BranchStore;
use fabro_checkpoint::git::Store;
use fabro_types::RunId;
use git2::{Oid, Signature};

9
lib/crates/fabro-workflow/src/operations/hydrate.rs
View file

@ -227,7 +227,7 @@ mod tests {
use fabro_types::{Conclusion, RunStatus, RunStatusRecord, StageStatus, fixtures};
use super::open_or_hydrate_run;
use crate::event::{WorkflowRunEvent, append_progress_event};
use crate::event::{WorkflowRunEvent, append_progress_event, canonicalize_event};
use crate::records::{Checkpoint, CheckpointExt, ConclusionExt, RunRecord, RunRecordExt};
use crate::run_status::RunStatusRecordExt;
@ -286,16 +286,15 @@ mod tests {
has_pricing: false,
};
conclusion.save(&run_dir.join("conclusion.json")).unwrap();
append_progress_event(
run_dir,
let envelope = canonicalize_event(
&test_run_id(),
&WorkflowRunEvent::RunNotice {
level: crate::event::RunNoticeLevel::Info,
code: "hydrated".to_string(),
message: "hello".to_string(),
},
)
.unwrap();
);
append_progress_event(run_dir, &envelope).unwrap();
}
#[tokio::test]

4
lib/crates/fabro-workflow/src/operations/rebuild_meta.rs
View file

@ -3,8 +3,8 @@ use std::fmt::Write;
use std::path::PathBuf;
use anyhow::{Context, Result, bail};
use fabro_git_storage::branchstore::BranchStore;
use fabro_git_storage::gitobj::Store as GitStore;
use fabro_checkpoint::branch::BranchStore;
use fabro_checkpoint::git::Store as GitStore;
use fabro_store::{
ListRunsQuery, NodeVisitRef, RunStore as DurableRunStore, Store as DurableStore,
};

4
lib/crates/fabro-workflow/src/operations/rewind.rs
View file

@ -3,8 +3,8 @@ use std::fmt::Write;
use std::str::FromStr;
use anyhow::{Context, Result, bail};
use fabro_git_storage::branchstore::{BranchStore, CommitInfo};
use fabro_git_storage::gitobj::Store;
use fabro_checkpoint::branch::{BranchStore, CommitInfo};
use fabro_checkpoint::git::Store;
use fabro_types::RunId;
use git2::{Oid, Repository, Signature};

131
lib/crates/fabro-workflow/src/operations/start.rs
View file

@ -19,7 +19,8 @@ use crate::context::Context;
use crate::error::FabroError;
use crate::event::{
EventEmitter, RunNoticeLevel, StoreProgressLogger, WorkflowRunEvent, append_progress_event,
build_redacted_event_payload,
append_progress_event_with_line, canonicalize_event, event_payload_from_redacted_json,
redacted_event_json,
};
use crate::git::MetadataStore;
use crate::handler::HandlerRegistry;
@ -128,8 +129,7 @@ pub(super) async fn execute_persisted_run(
// Write directly to progress.jsonl/live.json so projection failures do not
// recurse back through the decorated store.append_event() path.
let _ = append_progress_event(
&projection_run_dir,
let envelope = canonicalize_event(
&run_id,
&WorkflowRunEvent::RunNotice {
level: RunNoticeLevel::Warn,
@ -146,6 +146,7 @@ pub(super) async fn execute_persisted_run(
),
},
);
let _ = append_progress_event(&projection_run_dir, &envelope);
}),
),
);
@ -468,23 +469,38 @@ impl RunSession {
{
let sha_clone = Arc::clone(&last_git_sha);
self.emitter.on_event(move |event| match event {
WorkflowRunEvent::CheckpointCompleted {
git_commit_sha: Some(sha),
..
envelope if envelope.event == "checkpoint.completed" => {
if let Some(sha) = envelope
.properties
.get("git_commit_sha")
.and_then(serde_json::Value::as_str)
{
*sha_clone.lock().unwrap() = Some(sha.to_string());
}
}
| WorkflowRunEvent::WorkflowRunCompleted {
final_git_commit_sha: Some(sha),
..
envelope if envelope.event == "run.completed" => {
if let Some(sha) = envelope
.properties
.get("final_git_commit_sha")
.and_then(serde_json::Value::as_str)
{
*sha_clone.lock().unwrap() = Some(sha.to_string());
}
}
| WorkflowRunEvent::GitCommit { sha, .. } => {
*sha_clone.lock().unwrap() = Some(sha.clone());
envelope if envelope.event == "git.commit" => {
if let Some(sha) = envelope
.properties
.get("sha")
.and_then(serde_json::Value::as_str)
{
*sha_clone.lock().unwrap() = Some(sha.to_string());
}
}
_ => {}
});
}
let store_progress_logger =
StoreProgressLogger::new(Arc::clone(&self.run_store), record.run_id);
let store_progress_logger = StoreProgressLogger::new(Arc::clone(&self.run_store));
store_progress_logger.register(self.emitter.as_ref());
let init_options = InitOptions {
@ -703,9 +719,8 @@ impl Drop for DetachedRunCompletionGuard {
if !self.run_dir.join("conclusion.json").exists() {
let _ = write_failure_conclusion(&self.run_dir, message, Some(reason));
}
if let Some(run_id) = load_run_id(&self.run_dir) {
let _ = append_progress_event(
&self.run_dir,
let serialized_notice = load_run_id(&self.run_dir).and_then(|run_id| {
let envelope = canonicalize_event(
&run_id,
&WorkflowRunEvent::RunNotice {
level: RunNoticeLevel::Error,
@ -713,7 +728,19 @@ impl Drop for DetachedRunCompletionGuard {
message: message.to_string(),
},
);
}
let line = match redacted_event_json(&envelope) {
Ok(line) => line,
Err(err) => {
tracing::warn!(error = %err, "Failed to serialize post-run abort event");
return None;
}
};
if let Err(err) = append_progress_event_with_line(&self.run_dir, &envelope, &line) {
tracing::warn!(error = %err, "Failed to append post-run abort event");
return None;
}
Some((run_id, line))
});
let run_store = Arc::clone(&self.run_store);
let run_id = self.run_id;
if let Ok(handle) = Handle::try_current() {
@ -733,13 +760,23 @@ impl Drop for DetachedRunCompletionGuard {
"Failed to save post-run abort conclusion to store"
);
}
if let Some(run_id) = run_id {
let event = WorkflowRunEvent::RunNotice {
level: RunNoticeLevel::Error,
code: code.to_string(),
message: message.to_string(),
};
match build_redacted_event_payload(&event, &run_id) {
if let Some((run_id, line)) = serialized_notice.or(run_id
.map(|run_id| {
let envelope = canonicalize_event(
&run_id,
&WorkflowRunEvent::RunNotice {
level: RunNoticeLevel::Error,
code: code.to_string(),
message: message.to_string(),
},
);
redacted_event_json(&envelope)
.ok()
.map(|line| (run_id, line))
})
.flatten())
{
match event_payload_from_redacted_json(&line, &run_id) {
Ok(payload) => {
let _ = run_store.append_event(&payload).await;
}
@ -814,22 +851,16 @@ async fn persist_detached_failure(
}
if let Some(run_id) = load_run_id(run_dir) {
append_progress_event(
run_dir,
&run_id,
&WorkflowRunEvent::RunNotice {
level: RunNoticeLevel::Error,
code: format!("{phase}_failed"),
message: message.clone(),
},
)
.map_err(|err| FabroError::Io(err.to_string()))?;
let event = WorkflowRunEvent::RunNotice {
level: RunNoticeLevel::Error,
code: format!("{phase}_failed"),
message,
message: message.clone(),
};
match build_redacted_event_payload(&event, &run_id) {
let envelope = canonicalize_event(&run_id, &event);
let line = redacted_event_json(&envelope).map_err(|err| FabroError::Io(err.to_string()))?;
append_progress_event_with_line(run_dir, &envelope, &line)
.map_err(|err| FabroError::Io(err.to_string()))?;
match event_payload_from_redacted_json(&line, &run_id) {
Ok(payload) => {
if let Err(err) = run_store.append_event(&payload).await {
tracing::warn!(error = %err, "Failed to append detached failure event to store");
@ -956,7 +987,7 @@ mod tests {
async fn start_captures_checkpoint_git_sha_in_conclusion() {
let temp = tempfile::tempdir().unwrap();
let run_dir = temp.path().join("run");
let emitter = Arc::new(EventEmitter::new());
let emitter = Arc::new(EventEmitter::new(fixtures::RUN_1));
let registry = Arc::new(test_registry());
let injected = Arc::new(AtomicBool::new(false));
@ -967,15 +998,13 @@ mod tests {
if injected.load(Ordering::SeqCst) {
return;
}
if let WorkflowRunEvent::StageStarted { node_id, .. } = event {
if node_id == "start" {
injected.store(true, Ordering::SeqCst);
emitter_for_injection.emit(&WorkflowRunEvent::CheckpointCompleted {
node_id: node_id.clone(),
status: "success".to_string(),
git_commit_sha: Some("sha-test".to_string()),
});
}
if event.event == "stage.started" && event.node_id.as_deref() == Some("start") {
injected.store(true, Ordering::SeqCst);
emitter_for_injection.emit(&WorkflowRunEvent::CheckpointCompleted {
node_id: "start".to_string(),
status: "success".to_string(),
git_commit_sha: Some("sha-test".to_string()),
});
}
});
}
@ -1000,7 +1029,7 @@ mod tests {
async fn start_loads_persisted_from_run_dir() {
let temp = tempfile::tempdir().unwrap();
let run_dir = temp.path().join("run");
let emitter = Arc::new(EventEmitter::new());
let emitter = Arc::new(EventEmitter::new(fixtures::RUN_1));
let registry = Arc::new(test_registry());
persisted_workflow(MINIMAL_DOT, &run_dir);
@ -1020,7 +1049,7 @@ mod tests {
async fn start_invokes_on_node_callback_before_execution() {
let temp = tempfile::tempdir().unwrap();
let run_dir = temp.path().join("run");
let emitter = Arc::new(EventEmitter::new());
let emitter = Arc::new(EventEmitter::new(fixtures::RUN_1));
let registry = Arc::new(test_registry());
let visited = Arc::new(Mutex::new(Vec::new()));
@ -1049,7 +1078,7 @@ mod tests {
async fn start_errors_when_checkpoint_exists() {
let temp = tempfile::tempdir().unwrap();
let run_dir = temp.path().join("run");
let emitter = Arc::new(EventEmitter::new());
let emitter = Arc::new(EventEmitter::new(fixtures::RUN_1));
let registry = Arc::new(test_registry());
persisted_workflow(MINIMAL_DOT, &run_dir);
@ -1086,7 +1115,7 @@ mod tests {
async fn resume_errors_when_checkpoint_missing() {
let temp = tempfile::tempdir().unwrap();
let run_dir = temp.path().join("run");
let emitter = Arc::new(EventEmitter::new());
let emitter = Arc::new(EventEmitter::new(fixtures::RUN_1));
let registry = Arc::new(test_registry());
persisted_workflow(MINIMAL_DOT, &run_dir);
@ -1108,7 +1137,7 @@ mod tests {
async fn resume_errors_when_run_already_finished_successfully() {
let temp = tempfile::tempdir().unwrap();
let run_dir = temp.path().join("run");
let emitter = Arc::new(EventEmitter::new());
let emitter = Arc::new(EventEmitter::new(fixtures::RUN_1));
let registry = Arc::new(test_registry());
persisted_workflow(MINIMAL_DOT, &run_dir);

2
lib/crates/fabro-workflow/src/operations/test_support.rs
View file

@ -1,6 +1,6 @@
use std::collections::HashMap;
use fabro_git_storage::gitobj::Store;
use fabro_checkpoint::git::Store;
use git2::{Repository, Signature};
pub(super) fn temp_repo() -> (tempfile::TempDir, Store) {

Some files were not shown because too many files have changed in this diff Show more