mirror of
https://github.com/fabro-sh/fabro.git
synced 2026-10-17 04:37:03 +00:00
## Summary
Adds `output_schema` and `output_retries` node attributes that validate
structured LLM output and perform corrective repair turns inside the
same conversation context before failing the node. Also adds sortable
columns (Repo, Title, Workflow, Changes) to the runs list view and hides
the pager when the result set is small.
### Plan Summary
- **Task 1**: `Node::output_schema()` / `Node::output_retries()`
accessors in `fabro-types`, with `@`-prefix file-reference support in
static validation and file inlining.
- **Task 2**: New `handler/structured_output.rs` module —
`OutputSchemaKind` (Routing / JsonSchema), balanced JSON scanning,
validation, repair-message generation, `apply_validated_output`, and
`exhausted_failure_outcome`.
- **Task 3**: `extract_status_fields` moved to `structured_output.rs`;
agent routing fallback chain (response → `status.json` → last file
touched) preserved and delegated to `validate_agent_output_sources`.
- **Task 4/5**: `one_shot` (prompt) and `run` (agent) both loop over LLM
calls, appending the prior assistant response and a corrective user turn
on validation failure, up to `output_retries` times.
- **Task 6**: ACP backend rejects `output_schema` immediately with a
clear error before launching any process.
- **Task 7**: `outputs.mdx` and `dot-language.mdx` updated with
attribute docs, repair semantics, and `output.{node_id}` context key.
## What changed and why
```mermaid
TB
graph
A[Node attrs\noutput_schema / output_retries] --> B[structured_output.rs\nparse / validate / repair]
B --> C{OutputSchemaKind}
C -->|Routing| D[validate routing fields\n→ outcome routing]
C -->|JsonSchema| E[jsonschema validator\n→ context_updates.output.node_id]
B --> F[exhausted_failure_outcome\nterminal, non-retryable]
G[prompt handler\none_shot loop] --> B
H[agent handler\nrun loop + session.process_input] --> B
I[ACP backend] -->|output_schema present| J[Validation error\nno process launched]
```
**`output_schema="routing"`** tightens existing loose routing
extraction: malformed fields now fail validation and trigger a repair
turn rather than being silently ignored. The fallback priority (response
text → `status.json` → last file touched) is preserved but only for the
`NoJsonObject`/`NoRelevantJsonObject` error kinds that allow it.
**Custom schemas** (`@path` inlined to JSON Schema) validate the last
JSON object in the response against a precompiled
`jsonschema::Validator`. On success, the parsed value is stored at
`output.{node_id}` in `context_updates` for downstream nodes.
**Repair loop** — prompt nodes keep the prior assistant response in the
message list and append a corrective user message; agent API sessions
call `session.process_input` on the live session. Both paths aggregate
token usage across all turns. Exhausting `output_retries` returns a
terminal `OutputSchemaValidation` error (non-retryable, deterministic
failure category) that does not consume `max_retries`.
**ACP guardrail** rejects `output_schema` before spawning any
subprocess, with a clear `"output_schema is not supported with
backend=\"acp\" in this release"` message.
The `one_shot` refactor also extracted `complete_one_shot_request` and
`OneShotCompletion` to separate fallback-chain logic from the repair
loop, removing duplication.
### Fabro Details
<details>
<summary>Ran 9 stages in 74m 9s for $31.92</summary>
| Stage | Duration | Cost | Retries |
|---|---|---|---|
| start | 0s | – | 0 |
| toolchain | 1s | – | 0 |
| preflight_compile | 2m 16s | – | 0 |
| preflight_lint | 2m 28s | – | 0 |
| implement | 24m 59s | $17.96 | 0 |
| simplify_opus | 16m 36s | $9.95 | 0 |
| simplify_gpt | 3m 20s | $1.75 | 0 |
| verify | 6m 29s | – | 0 |
| fixup | 17m 13s | $2.26 | 0 |
| **Total** | **74m 9s** | **$31.92** | **0** |
</details>
<details>
<summary>Ran <code>ImplementPlan.fabro</code> (11 nodes and 14
edges)</summary>
```dot
digraph ImplementPlan {
graph [
goal="Implement and simplify",
model_stylesheet="
* { model: claude-opus-4-7; }
"
]
rankdir=LR
start [shape=Mdiamond, label="Start"]
exit [shape=Msquare, label="Exit"]
toolchain [label="Toolchain", shape=parallelogram, script="command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1", max_retries=0]
preflight_compile [label="Preflight Compile", shape=parallelogram, script="cargo check -q --workspace 2>&1", max_retries=0]
preflight_lint [label="Preflight Lint", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1", max_retries=0]
fix_lints [label="Fix Lints", prompt="The preflight lint step failed. Read the build output from context and fix all clippy lint warnings.", max_visits=3]
implement [label="Implement", prompt="Read the plan file referenced in the goal and implement every step. Make all the code changes described in the plan. Use red/green TDD.", model="gpt-55", reasoning_effort="xhigh"]
simplify_opus [label="Simplify (Opus)", prompt="@prompts/simplify.md"]
simplify_gpt [label="Simplify (GPT-55)", prompt="@prompts/simplify.md", model="gpt-55"]
verify [label="Verify", shape=parallelogram, script="git fetch origin main 2>&1 && git merge --no-edit --no-stat origin/main 2>&1 && cargo +nightly-2026-04-14 fmt --all 2>&1 && cargo dev docs refresh 2>&1 && cargo +nightly-2026-04-14 fmt --check --all 2>&1 && ! rg -n 'AuthMode::Disabled|RunAuthMethod|RunSubjectProvenance|\bActorRef\b|\bActorKind\b|AuthenticatedSubject|AuthenticatedService|AuthorizeRunScoped|AuthorizeRunBlob|AuthorizeStageArtifact|AuthorizeCommandLog|auth_method\s*==\s*\"disabled\"' lib/crates apps lib/packages docs/public/api-reference/fabro-api.yaml 2>&1 && cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --workspace --status-level slow --profile ci 2>&1 && cargo dev docs check 2>&1 && bun install --frozen-lockfile 2>&1 && (cd apps/fabro-web && bun run typecheck) 2>&1 && (cd apps/fabro-web && bun run test) 2>&1 && (cd lib/packages/fabro-api-client && bun run typecheck) 2>&1 && cargo dev build -- -p fabro-cli --release 2>&1", goal_gate=true, retry_target="fixup"]
fixup [label="Fixup", prompt="The verify step failed. Read the build output from context and fix all format, clippy, Rust test, docs, TypeScript typecheck/test, and build failures.", max_visits=3]
start -> toolchain
toolchain -> preflight_compile [condition="outcome=succeeded"]
toolchain -> exit
preflight_compile -> preflight_lint [condition="outcome=succeeded"]
preflight_compile -> exit
preflight_lint -> implement [condition="outcome=succeeded"]
preflight_lint -> fix_lints
fix_lints -> preflight_lint
implement -> simplify_opus -> simplify_gpt -> verify
verify -> exit [condition="outcome=succeeded"]
verify -> fixup
fixup -> verify
}
```
</details>
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Fabro <fabro@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
289 lines
14 KiB
Text
289 lines
14 KiB
Text
---
|
|
title: "Outputs & Artifacts"
|
|
description: "How Fabro captures agent responses, tracks file changes, and collects test assets"
|
|
---
|
|
|
|
When an agent or prompt node finishes, Fabro captures its response text and produces an **outcome** that feeds into context, transition logic, and downstream nodes. Fabro also tracks every file change per stage, offloads large outputs into content-addressed blob storage, and automatically collects test artifacts like screenshots and reports.
|
|
|
|
## Response capture
|
|
|
|
After an agent or prompt node completes, Fabro captures the full response text and persists it to `stages/{rank:03}-{node_id}@{visit}/response.md` in metadata snapshots and `fabro dump` output. It also writes the final outcome (status, context updates, routing directives) to `stages/{rank:03}-{node_id}@{visit}/status.json`.
|
|
|
|
## Context updates
|
|
|
|
Every agent and prompt node sets three context keys from its response:
|
|
|
|
| Key | Value |
|
|
|---|---|
|
|
| `last_stage` | The node ID of the stage that just completed |
|
|
| `last_response` | The response text, truncated to the first 200 characters |
|
|
| `response.{node_id}` | The full response text |
|
|
|
|
These keys are available to downstream nodes via the [context](/execution/context). The `last_response` key provides a quick preview, while `response.{node_id}` preserves the complete output for nodes that need it.
|
|
|
|
```dot
|
|
plan -> implement -> review -> exit
|
|
|
|
// In the review node's prompt, you can reference prior outputs:
|
|
// The context key response.plan contains the full plan text
|
|
// The context key response.implement contains the full implementation
|
|
```
|
|
|
|
## Routing directives
|
|
|
|
Agent and prompt nodes can influence which edge is taken after they complete by including a JSON object with routing fields in their response. Fabro scans the LLM output for the **last** JSON object containing any recognized routing field:
|
|
|
|
```json
|
|
{
|
|
"outcome": "failed",
|
|
"failure_reason": "tests failed",
|
|
"preferred_next_label": "fix",
|
|
"suggested_next_ids": ["implement", "review"],
|
|
"context_updates": { "tests_passed": true, "coverage": 85 }
|
|
}
|
|
```
|
|
|
|
| Field | Effect |
|
|
|---|---|
|
|
| `outcome` | Sets the node outcome: `succeeded`, `failed`, `partially_succeeded`, or `skipped` |
|
|
| `failure_reason` | When `outcome` is `failed`, provides a structured failure message |
|
|
| `preferred_next_label` | Matched against edge labels to select the next node |
|
|
| `suggested_next_ids` | Ordered list of preferred target node IDs |
|
|
| `context_updates` | Key-value pairs merged into the run context |
|
|
|
|
### How extraction works
|
|
|
|
Fabro finds all balanced `{...}` JSON objects in the response text, parses each one, and uses the **last** object that contains at least one recognized field (`preferred_next_label`, `outcome`, `failure_reason`, `suggested_next_ids`, `context_updates`). The JSON can appear anywhere in the response -- inside a fenced code block, inline with natural language, or at the end.
|
|
|
|
```markdown
|
|
I've reviewed the code and found several issues that need fixing.
|
|
The test coverage is below the threshold.
|
|
|
|
{"preferred_next_label": "fix", "context_updates": {"coverage": 72}}
|
|
```
|
|
|
|
JSON objects without recognized fields are ignored.
|
|
|
|
### Validated routing output
|
|
|
|
Set `output_schema="routing"` on an agent or prompt node to require Fabro's built-in routing directive schema:
|
|
|
|
```dot
|
|
review [
|
|
shape=tab,
|
|
prompt="Review the implementation and return routing JSON.",
|
|
output_schema="routing",
|
|
output_retries=2
|
|
]
|
|
```
|
|
|
|
With `output_schema="routing"`, the routing JSON must be an object with at least one recognized routing field (`preferred_next_label`, `outcome`, `failure_reason`, `suggested_next_ids`, or `context_updates`) and those fields must have the expected types. Malformed routing JSON fails validation instead of being silently ignored.
|
|
|
|
Fabro repairs invalid structured output inside the same LLM context before failing the node. For prompt nodes, Fabro appends the invalid assistant response and a corrective user message to the same message list. For agent nodes using the API backend, Fabro sends the corrective message to the same live agent session. `output_retries` controls these repair turns and defaults to `2`; `output_retries=0` validates once and fails without a repair turn. These repair turns are separate from workflow `max_retries` and do not consume node retry attempts.
|
|
|
|
### Fallback: status.json file
|
|
|
|
If no routing directives are found in the response text, Fabro checks whether the agent wrote a `status.json` file into the sandbox working directory. If the file exists, Fabro extracts routing directives from it using the same logic. This is useful for agents that write structured output to files rather than including JSON in their response text.
|
|
|
|
Response text directives always take priority -- `status.json` is only read as a fallback when the response contains no recognized routing fields.
|
|
|
|
If neither source provides routing directives, the transition falls through to condition matching, unconditional edges, or weight-based tiebreaking as described in [Transitions](/workflows/transitions).
|
|
|
|
### Instructing the agent
|
|
|
|
Fabro does not automatically instruct agents to emit routing JSON. You must include instructions in your prompt:
|
|
|
|
```dot
|
|
review [
|
|
label="Review",
|
|
shape=tab,
|
|
prompt="Review the implementation. If changes are needed, \
|
|
respond with: {\"preferred_next_label\": \"fix\"}. \
|
|
If everything looks good, respond with: \
|
|
{\"preferred_next_label\": \"approve\"}."
|
|
]
|
|
|
|
review -> fix [label="Fix"]
|
|
review -> approve [label="Approve"]
|
|
```
|
|
|
|
## Custom structured outputs
|
|
|
|
Agent and prompt nodes can also validate their final JSON object against a JSON Schema file:
|
|
|
|
```dot
|
|
audit [
|
|
shape=tab,
|
|
prompt="Audit the change and return JSON that matches the schema.",
|
|
output_schema="@schemas/audit-result.schema.json",
|
|
output_retries=2
|
|
]
|
|
```
|
|
|
|
`output_schema="@path/to/schema.json"` uses the same workflow file-reference rules as prompt files: the schema is loaded relative to the workflow file and inlined before execution. The final JSON object in the LLM response is validated with `jsonschema`.
|
|
|
|
When custom schema validation succeeds, Fabro stores the parsed JSON value in context at:
|
|
|
|
| Key | Value |
|
|
|---|---|
|
|
| `output.{node_id}` | The parsed JSON object that matched the custom schema |
|
|
|
|
For example, node `audit` writes its parsed custom output to `output.audit`. Fabro still stores the raw response text at `response.audit`.
|
|
|
|
If custom schema validation fails, Fabro sends concise validation feedback to the same prompt conversation or agent session and asks for corrected JSON. After `output_retries` repair turns are exhausted, the node fails terminally with `output schema validation failed after N repair attempt(s)`.
|
|
|
|
<Note>
|
|
Structured output validation currently applies to agent and prompt nodes. `backend="acp"` does not support `output_schema` in this release. Custom schemas update `output.{node_id}`; routing schemas update routing fields and `context_updates` instead.
|
|
</Note>
|
|
|
|
## Output logging
|
|
|
|
Fabro writes several files per stage to `stages/{rank:03}-{node_id}@{visit}/` in metadata snapshots and `fabro dump` output:
|
|
|
|
| File | Contents |
|
|
|---|---|
|
|
| `prompt.md` | The assembled prompt (preamble + expanded prompt text) |
|
|
| `response.md` | The full LLM response text |
|
|
| `status.json` | The outcome: status, context updates, routing directives, usage stats |
|
|
|
|
These files are written for every agent and prompt node execution, including retries. Use them for debugging unexpected agent behavior or verifying that routing directives were extracted correctly.
|
|
|
|
## File tracking
|
|
|
|
Fabro records which files each stage touches. When an agent calls `write_file` or `edit_file`, Fabro tracks the file path. When using a CLI-based agent backend, Fabro snapshots the Git working tree before and after execution and diffs the results.
|
|
|
|
The tracked paths are stored as `files_touched` on the stage outcome:
|
|
|
|
```json
|
|
{
|
|
"status": "succeeded",
|
|
"files_touched": ["src/main.rs", "tests/api_test.rs", "README.md"]
|
|
}
|
|
```
|
|
|
|
### Where files_touched appears
|
|
|
|
| Location | How it's used |
|
|
|---|---|
|
|
| `StageCompleted` event | Emitted with `files_touched` in the event stream and surfaced by `fabro events` / exported event streams |
|
|
| Preambles | Listed under each completed stage so downstream agents know what changed |
|
|
| `status.json` | Written to the stage's logs directory after each node completes |
|
|
|
|
### How tracking works
|
|
|
|
For the **API backend**, Fabro subscribes to agent session events. When a `ToolCallStarted` event fires for `write_file` or `edit_file`, Fabro records the `file_path` argument as pending. When the corresponding `ToolCallCompleted` arrives without an error, the path is confirmed as touched. Failed tool calls are discarded.
|
|
|
|
For the **CLI** and **ACP** backends, Fabro takes a different approach: it runs `git diff --name-only` and `git ls-files --others --exclude-standard` before and after the external agent session, then computes the difference. Any files that appear in the "after" snapshot but not "before" are recorded as touched.
|
|
|
|
## Artifact offloading
|
|
|
|
When a stage produces a large context value -- an LLM response or any context update -- Fabro automatically offloads it into a global content-addressed blob store instead of leaving the full value inline in durable context. Command output is streamed to a stage log file while the command runs, then finalized into a durable blob ref after completion.
|
|
|
|
### How offloading works
|
|
|
|
After each node completes, Fabro checks every context update. If the serialized JSON of a value exceeds **100KB**, it is stored once by SHA-256 hash and replaced with a durable blob ref. Command output is always stored this way after completion, even when it is small or empty:
|
|
|
|
```
|
|
response.plan --> blob://sha256/2cf24dba5fb0...
|
|
command.output --> blob://sha256/a4f3c1d9c2e1...
|
|
```
|
|
|
|
Non-command values under 100KB remain inline.
|
|
|
|
Checkpoints, checkpoint-completed events, forks, and resumes persist these `blob://` refs, not host-specific file paths.
|
|
|
|
### Preamble rendering
|
|
|
|
When Fabro builds a [preamble](/execution/context#preamble-construction) for a downstream stage, it first materializes any blob refs into execution-local files and then renders a reference instead of inlining the full content:
|
|
|
|
```markdown
|
|
## Completed stages
|
|
- **plan**: success
|
|
- Model: claude-sonnet-4-5, 12.4k tokens in / 3.2k out
|
|
- Files: src/main.rs, tests/api_test.rs
|
|
- Response: See: /path/to/runtime/blobs/<blob_id>.json
|
|
- **test**: success
|
|
- Script: `cargo test 2>&1 || true`
|
|
- Stdout: See: /path/to/runtime/blobs/<blob_id>.json
|
|
```
|
|
|
|
This keeps preambles concise while still giving agents a path to read the full output if needed.
|
|
|
|
## Git storage
|
|
|
|
Large offloaded context values are not stored on the Git [metadata branch](/execution/checkpoints#metadata-branch). The metadata branch keeps checkpoint JSON and stage metadata; blob payloads live in the durable blob store and are referenced by `blob://sha256/...`.
|
|
|
|
Captured stage artifacts such as screenshots, videos, reports, and traces still use the artifact store and metadata export paths described below.
|
|
|
|
## Remote sandbox syncing
|
|
|
|
For remote sandboxes (Docker, Daytona), execution-time file access happens inside the sandbox filesystem.
|
|
|
|
- Blob refs are materialized into `{working_directory}/.fabro/blobs/{blob_id}.json`
|
|
- Explicit non-blob `file://` refs keep the existing copy-on-demand behavior and are copied into `{working_directory}/.fabro/artifacts/{filename}` when needed
|
|
|
|
In both cases, downstream handlers and agents continue to consume ordinary `file://` pointers during execution.
|
|
|
|
<Note>
|
|
For local sandboxes, syncing is a no-op since the agent can already access the host filesystem directly.
|
|
</Note>
|
|
|
|
## Automatic asset capture
|
|
|
|
After each node executes a command, Fabro automatically scans the sandbox for test artifacts — screenshots, videos, reports, and traces — and copies any new or changed files to the run's directory. This happens without any agent or workflow configuration.
|
|
|
|
### How asset capture works
|
|
|
|
1. **Before** the command runs, Fabro takes a baseline snapshot of known artifact paths in the sandbox
|
|
2. **After** the command completes, Fabro re-scans and diffs against the baseline
|
|
3. Files that are new or modified since the command started are downloaded to the stage's artifact directory
|
|
|
|
Only files modified after the command started are collected. Files that match the baseline fingerprint (same size and mtime) are skipped. Individual files over 10 MB and total collections over 50 MB are also skipped.
|
|
|
|
### What gets captured
|
|
|
|
Fabro looks for files inside these directories:
|
|
|
|
| Directory | Typical contents |
|
|
|---|---|
|
|
| `playwright-report/` | Playwright HTML reports |
|
|
| `test-results/` | Playwright screenshots, videos, and traces |
|
|
| `cypress/videos/` | Cypress test recordings |
|
|
| `cypress/screenshots/` | Cypress failure screenshots |
|
|
|
|
And files matching these filename patterns anywhere in the tree:
|
|
|
|
| Pattern | Typical contents |
|
|
|---|---|
|
|
| `junit*.xml` | JUnit XML test reports |
|
|
| `*.trace.zip` | Playwright trace archives |
|
|
|
|
Tool caches and dependency directories (`node_modules`, `.cache/ms-playwright`, `.yarn/cache`, etc.) are excluded from scanning.
|
|
|
|
### Asset storage layout
|
|
|
|
Collected assets are written to the run's directory, organized by node and retry attempt:
|
|
|
|
```
|
|
~/.fabro/scratch/{run_id}/
|
|
cache/
|
|
artifacts/
|
|
files/
|
|
{node_slug}/
|
|
retry_1/
|
|
test-results/
|
|
screenshot.png
|
|
video.webm
|
|
```
|
|
|
|
## Observability
|
|
|
|
Outputs and artifacts appear in several observability surfaces:
|
|
|
|
| Surface | What's reported |
|
|
|---|---|
|
|
| `StageCompleted` event | `files_touched` list for the stage |
|
|
| `WorkflowRunCompleted` event | `artifact_count` -- total number of offloaded artifacts across the run |
|
|
| Web UI | Run stage output, stage artifacts, and downloadable artifact files |
|
|
| [Preambles](/execution/context#preamble-construction) | File list and artifact pointer references for completed stages |
|
|
| Stage logs | `status.json` in each stage's run directory contains the full outcome including `files_touched` |
|