mirror of
https://github.com/fabro-sh/fabro.git
synced 2026-08-28 05:27:41 +00:00
142 lines
4.8 KiB
Text
142 lines
4.8 KiB
Text
---
|
|
title: "Observability"
|
|
description: "How to monitor, inspect, and analyze workflow runs"
|
|
---
|
|
|
|
Fabro captures a structured event for every significant action during a workflow run. These events cover stage execution, agent tool calls, retries, routing, sandbox lifecycle, git checkpoints, retro generation, and more.
|
|
|
|
## Event stream
|
|
|
|
Every workflow run emits a sequence of canonical **run event envelopes** that are:
|
|
|
|
- Written to `progress.jsonl` in the run directory
|
|
- Broadcast over SSE to connected API clients
|
|
- Stored for later analysis and retro generation
|
|
- Rendered by CLI progress and log tooling
|
|
|
|
### Event names
|
|
|
|
Event names use lowercase dot notation, for example:
|
|
|
|
- `run.started`
|
|
- `stage.started`
|
|
- `stage.completed`
|
|
- `agent.tool.started`
|
|
- `agent.tool.completed`
|
|
- `sandbox.ready`
|
|
- `parallel.branch.completed`
|
|
|
|
### Envelope format
|
|
|
|
Each line in `progress.jsonl` is a JSON object with a stable envelope:
|
|
|
|
```json
|
|
{
|
|
"id": "01960d0c-5d16-7d6e-8f61-9fd6f4a532b5",
|
|
"ts": "2026-03-30T12:00:01.000Z",
|
|
"run_id": "01JQ...",
|
|
"event": "agent.tool.started",
|
|
"session_id": "ses_child",
|
|
"parent_session_id": "ses_parent",
|
|
"node_id": "implement",
|
|
"node_label": "Implement",
|
|
"properties": {
|
|
"tool_name": "shell",
|
|
"tool_call_id": "call_1",
|
|
"arguments": {"command": "cargo test"}
|
|
}
|
|
}
|
|
```
|
|
|
|
Envelope fields:
|
|
|
|
| Field | Description |
|
|
|---|---|
|
|
| `id` | Unique event id |
|
|
| `ts` | UTC timestamp |
|
|
| `run_id` | Workflow run id |
|
|
| `event` | Event name |
|
|
| `session_id` | Session that emitted the event, when applicable |
|
|
| `parent_session_id` | Immediate parent session for forwarded child events |
|
|
| `node_id` | Node or branch id, when applicable |
|
|
| `node_label` | Human-facing label for `node_id`, when applicable |
|
|
| `properties` | Event-specific payload |
|
|
|
|
Only `id`, `ts`, `run_id`, and `event` are always present. Optional fields are omitted when they do not apply.
|
|
|
|
## Reading `progress.jsonl`
|
|
|
|
Because event payload lives in `properties`, most shell queries should look there.
|
|
|
|
```bash
|
|
# Count tool calls in a run
|
|
jq -r 'select(.event == "agent.tool.started") | .properties.tool_name' \
|
|
~/.fabro/runs/01JKXYZ.../progress.jsonl | wc -l
|
|
|
|
# Find stage failures
|
|
jq 'select(.event == "stage.failed")' \
|
|
~/.fabro/runs/01JKXYZ.../progress.jsonl
|
|
|
|
# See which edges were taken
|
|
jq '{from: .properties.from_node, to: .properties.to_node, label: .properties.label}' \
|
|
~/.fabro/runs/01JKXYZ.../progress.jsonl | head
|
|
```
|
|
|
|
`live.json` is still a pretty-printed copy of the most recent event envelope.
|
|
|
|
## Event categories
|
|
|
|
Common categories include:
|
|
|
|
| Category | Example events |
|
|
|---|---|
|
|
| Run lifecycle | `run.started`, `run.completed`, `run.failed`, `run.notice` |
|
|
| Stage lifecycle | `stage.started`, `stage.completed`, `stage.failed`, `stage.retrying` |
|
|
| Agent activity | `agent.message`, `agent.tool.started`, `agent.warning`, `agent.sub.spawned` |
|
|
| Routing | `edge.selected`, `loop.restart`, `parallel.started` |
|
|
| Git and checkpoints | `checkpoint.completed`, `git.commit`, `git.push` |
|
|
| Setup and sandbox | `sandbox.initializing`, `sandbox.ready`, `setup.started` |
|
|
| Retro | `retro.started`, `retro.completed`, `retro.failed` |
|
|
|
|
## Sub-agent visibility
|
|
|
|
Sub-agent activity now appears as normal agent events with session linkage:
|
|
|
|
- `session_id` identifies the child session
|
|
- `parent_session_id` identifies its immediate parent
|
|
|
|
Lifecycle events such as `agent.sub.spawned` and `agent.sub.completed` are emitted by the parent session. Tool calls and other child activity are forwarded with their original `session_id`.
|
|
|
|
## Real-time monitoring
|
|
|
|
### API: Server-Sent Events
|
|
|
|
When running workflows through the API server, subscribe to the [run events endpoint](/api-reference/runs/stream-run-events). Each SSE payload is a serialized run event envelope in the same shape used by `progress.jsonl`.
|
|
|
|
### Web UI
|
|
|
|
The web frontend consumes the SSE stream automatically and shows stage progress, tool calls, and human interaction as they happen.
|
|
|
|
<Frame caption="The Stages tab shows the full agent conversation including tool calls and responses.">
|
|
<img src="/images/web/run-stages.png" alt="Fabro web UI run stages showing agent conversation with tool calls" />
|
|
</Frame>
|
|
|
|
### CLI progress
|
|
|
|
The CLI renders live progress from the same envelope format. This is written to stderr so stdout remains pipe-friendly.
|
|
|
|
## Post-run analysis
|
|
|
|
Run artifacts still include:
|
|
|
|
| File | Description |
|
|
|---|---|
|
|
| `run.json` | Run metadata and graph |
|
|
| `start.json` | Start record |
|
|
| `progress.jsonl` | Full event envelope stream |
|
|
| `live.json` | Latest event envelope |
|
|
| `checkpoint.json` | Final execution state |
|
|
| `retro.json` | Retrospective, when enabled |
|
|
| `conclusion.json` | Terminal summary |
|
|
|
|
See [retros](/execution/retros), [stages](/api-reference/run-internals/list-run-stages), and [turns](/api-reference/run-internals/list-stage-turns) for higher-level analysis views built on top of this event stream.
|