mirror of
https://github.com/fabro-sh/fabro.git
synced 2026-08-28 05:27:41 +00:00
ed68c8b1d4
Rewrite every docs/ reference and integration guide example that previously showed legacy flat TOML (`[llm]`, `[vars]`, `[sandbox]`, `[setup]`, `[exec]`, `[fabro]`, `[pull_request]`, `[mcp_servers]`, `[git]`, `[web]`, `[api]`, `[features] retros`, `version = 1`, top-level `storage_dir`) to use the v2 namespaced schema. Also update the surrounding prose to describe v2 merge semantics (R22 run.inputs wholesale replacement, R71 sticky sandbox.env/labels, R30 whole-list prepare.steps replacement, hook id-based replacement). Files touched: - docs/reference/user-configuration.mdx (complete rewrite around [cli.*] ownership, [run.*] run-scoped defaults, [cli.target] / [cli.exec] / [cli.output] / [cli.updates] / [cli.logging], and [run.agent.mcps.<name>] with durations like "10s") - docs/reference/cli.mdx (settings.toml example uses [cli.exec.*], [run.model], [cli.target]) - docs/execution/run-configuration.mdx (full run-config example rewritten to use [workflow].graph, [run].goal/working_dir, [run.model], [run.prepare.steps], [run.sandbox.daytona.snapshot] with Size values, [run.inputs], [run.artifacts], [run.agent.mcps], [run.pull_request], [[run.hooks]] with optional id and duration timeout; section docs explain the new merge semantics) - docs/execution/environments.mdx and devcontainers.mdx (sandbox examples now use [run.sandbox.*]) - docs/execution/retros.mdx (retros moved to [run.execution] retros = true per R31) - docs/execution/failures.mdx (fallbacks now a single ordered array under [run.model].fallbacks) - docs/workflows/variables.mdx ([vars] → [run.inputs], wholesale replacement semantics explained) - docs/administration/server-configuration.mdx (full reference rewritten around [server.listen]/[server.api]/[server.web]/ [server.auth]/[server.storage]/[server.scheduler]/[server.logging]/ [server.integrations]) - docs/api-reference/overview.mdx (auth strategies now enabled via [server.auth.api.jwt].enabled and [server.auth.api.mtls].enabled; listener TLS moved to [server.listen.tls]) - docs/integrations/daytona.mdx, github.mdx (provider config now nested under [run.sandbox.daytona] / [server.integrations.github]) - docs/human-tools/ssh-access.mdx (sandbox examples to v2) - docs/agents/mcp.mdx (Playwright sandbox example to [run.agent.mcps]) - docs/core-concepts/models.mdx (model config and fallbacks array to [run.model]) Canonical fabro-cli overrides and server run_manifest now emit verbose via [cli.output].verbosity = verbose rather than the prior run.metadata staging. No code changes beyond those Stage 4 fixes that were already in flight.
146 lines
6.5 KiB
Text
146 lines
6.5 KiB
Text
---
|
||
title: "Retros"
|
||
description: "Automatic retrospectives that analyze every workflow run"
|
||
---
|
||
|
||
<Warning>
|
||
**Experimental feature.** Retros are disabled by default. Enable them by setting `retros = true` under `[run.execution]` in your project or workflow config.
|
||
</Warning>
|
||
|
||
After every workflow run, Fabro can generate a **retro** — a structured retrospective that captures what happened, what went well, and what didn't. Retros combine deterministic metrics extracted from the run's checkpoint with a qualitative narrative produced by an LLM agent that analyzes the full event stream.
|
||
|
||
The goal is continuous improvement. Retros give you a searchable history of how your workflows perform over time, surface friction patterns that would otherwise go unnoticed, and identify follow-up work before it falls through the cracks.
|
||
|
||
<Frame caption="The Retros page shows all retrospectives with smoothness ratings, duration, and friction counts.">
|
||
<img src="/images/web/retros-list.png" alt="Fabro web UI Retros list showing runs with Smooth, Bumpy, Effortless, and Struggled ratings" />
|
||
</Frame>
|
||
|
||
## What's in a retro
|
||
|
||
A retro has two layers: **quantitative stats** derived cheaply from checkpoint data, and an **agent-generated narrative** that interprets the run holistically.
|
||
|
||
### Quantitative layer
|
||
|
||
The quantitative layer is extracted directly from the [checkpoint](/execution/checkpoints) and event stream — no LLM calls required:
|
||
|
||
| Field | Description |
|
||
|---|---|
|
||
| **Per-stage breakdown** | Duration, retry count, cost, files touched, status, and failure reason for each stage |
|
||
| **Aggregate stats** | Total duration, total cost, total retries, all files touched, stages completed vs. failed |
|
||
|
||
### Narrative layer
|
||
|
||
An LLM agent reads the run's full event stream and produces a structured analysis:
|
||
|
||
| Field | Description |
|
||
|---|---|
|
||
| **Smoothness** | Overall rating on a 5-point scale (see below) |
|
||
| **Intent** | What the run was trying to accomplish |
|
||
| **Outcome** | What actually happened |
|
||
| **Learnings** | What was discovered about the repo, code, workflow, or tools |
|
||
| **Friction points** | Where things got stuck and why |
|
||
| **Open items** | Follow-up work, tech debt, test gaps, or investigations identified |
|
||
|
||
The agent has tool access to grep and read the event stream, so it can inspect actual tool call patterns, error messages, and approach pivots — not just pass/fail signals.
|
||
|
||
## Smoothness ratings
|
||
|
||
Every retro includes a smoothness rating that grades the overall quality of the run's execution:
|
||
|
||
| Rating | Meaning |
|
||
|---|---|
|
||
| **Effortless** | Goal achieved on the first try. No retries, no wrong approaches. Agent moved efficiently from start to finish. |
|
||
| **Smooth** | Goal achieved with minor hiccups — 1–2 retries or a brief wrong approach quickly corrected. No human intervention needed. |
|
||
| **Bumpy** | Goal achieved but with notable friction: multiple retries, at least one significant wrong approach, or substantial time on dead ends. |
|
||
| **Struggled** | Goal achieved only with difficulty: many retries, major approach changes, human intervention, or partial failures requiring recovery. |
|
||
| **Failed** | Run did not achieve its stated goal. Some stages may have completed, but the overall intent was not fulfilled. |
|
||
|
||
The rating considers the full context visible in agent events — tool call patterns, error recovery sequences, approach pivots — not just stage pass/fail counts.
|
||
|
||
## Learnings, friction points, and open items
|
||
|
||
### Learnings
|
||
|
||
Learnings capture what was discovered during the run, categorized by type:
|
||
|
||
| Category | Examples |
|
||
|---|---|
|
||
| `repo` | Repository structure, build system quirks, CI configuration |
|
||
| `code` | Bug root causes, module boundaries, API contracts |
|
||
| `workflow` | Node ordering issues, missing stages, prompt improvements |
|
||
| `tool` | Tool limitations, MCP server behavior, command output parsing |
|
||
|
||
### Friction points
|
||
|
||
Friction points identify where the run got stuck and what caused the slowdown:
|
||
|
||
| Kind | Description |
|
||
|---|---|
|
||
| `retry` | A stage needed multiple attempts |
|
||
| `timeout` | A stage or tool call hit a time limit |
|
||
| `wrong_approach` | The agent pursued a dead end before pivoting |
|
||
| `tool_failure` | A tool or command failed unexpectedly |
|
||
| `ambiguity` | Unclear requirements or conflicting signals caused confusion |
|
||
|
||
Each friction point can optionally reference the `stage_id` where it occurred.
|
||
|
||
### Open items
|
||
|
||
Open items capture follow-up work identified during the run:
|
||
|
||
| Kind | Description |
|
||
|---|---|
|
||
| `tech_debt` | Code quality issues worth addressing later |
|
||
| `follow_up` | Work that's related but out of scope for this run |
|
||
| `investigation` | Unknowns that need further research |
|
||
| `test_gap` | Missing test coverage discovered during the run |
|
||
|
||
## How retros are generated
|
||
|
||
Retro generation happens in two phases after a run completes:
|
||
|
||
1. **Derive** — Fabro extracts stage durations from durable run events and builds a retro from the checkpoint data. This is deterministic, fast, and produces the quantitative layer.
|
||
|
||
2. **Narrate** — An LLM agent session analyzes the run data. The agent receives temp files named `progress.jsonl`, `checkpoint.json`, `run.json`, and `start.json` inside its sandbox so it can grep and read the event stream and run state. The narrative fields are merged back into durable retro state.
|
||
|
||
Both phases run automatically at the end of every CLI run. The API server derives the quantitative layer but does not currently run the narrative agent.
|
||
|
||
<Frame caption="A run's Retro tab shows the smoothness rating, stats, learnings, and open items.">
|
||
<img src="/images/web/run-retro.png" alt="Fabro web UI run retro showing Smooth rating, duration, cost, learnings, and follow-up items" />
|
||
</Frame>
|
||
|
||
## Accessing retros
|
||
|
||
### CLI
|
||
|
||
To enable retros for your project, set `retros = true` under `[run.execution]` in your `fabro.toml`:
|
||
|
||
```toml title="fabro.toml"
|
||
_version = 1
|
||
|
||
[run.execution]
|
||
retros = true
|
||
```
|
||
|
||
To skip retro generation for a single run when retros are enabled, pass `--no-retro`:
|
||
|
||
```bash
|
||
fabro run workflow.fabro --no-retro
|
||
```
|
||
|
||
Retros can also be enabled server-wide in `settings.toml`:
|
||
|
||
```toml title="settings.toml"
|
||
_version = 1
|
||
|
||
[run.execution]
|
||
retros = true
|
||
```
|
||
|
||
### API
|
||
|
||
Retros are also available via the REST API. See the [list retros](/api-reference/retros/list-retros) and [retrieve retro](/api-reference/retros/retrieve-retro) API reference pages.
|
||
|
||
## Storage
|
||
|
||
Retros are stored in durable run state. If you need files on disk, `fabro store dump` materializes the retro as `retro.json` alongside other exported run data.
|