fabro/docs/public/reference/server-operations.mdx
Scott Werner bccc5750a4 Size the blob activation disk preflight to the remaining import work
The preflight demanded ~1.5x the full legacy inventory bytes free on
every startup, with no credit for rows already imported. Because the
first activation itself consumes about twice the legacy bytes (the
SQLite copy plus the retained backup) and the legacy keyspace stays in
place for the whole retention window, a successfully activated server
could fall below the requirement and become unable to restart until an
operator freed space the server would never write.

The legacy inventory now checks each row's hash against the SQLite blobs
table and reports pending rows and bytes, and the preflight requires
1.5x only the pending bytes plus the backup reserve and fixed headroom.
A warm restart with nothing left to import needs only the headroom.
Also updates the server operations doc for this and for the
verification pass now running only on boots that import rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 14:02:35 -04:00

201 lines
11 KiB
Text

---
title: "Server Operations"
description: "Operate the Fabro server: starting, install wizard, auth, web UI, and pointing the CLI at it"
---
<Warning>
The server interface is in private early access. Contact [bryan@qlty.sh](mailto:bryan@qlty.sh) if you're interested in trying it.
</Warning>
This page covers operating the Fabro server once it's running, whether locally on your laptop or self-hosted in a container. For where to run it, see [Deployment](/administration/deployment).
## Starting the server
```bash
fabro server start
```
This starts the server on a Unix socket at `~/.fabro/fabro.sock` by default. Use `--bind 127.0.0.1` for TCP.
### First run: web install wizard
If `~/.fabro/settings.toml` does not yet exist, `fabro server start` enters **install mode**: it prints an install URL and a one-time install token, attempts to open the URL in your default browser, and serves a web wizard that walks you through configuring your server URL, shared object store, LLM provider, and GitHub integration.
The LLM step can be completed with one or more provider keys, or explicitly skipped so you can finish server setup first and add model credentials later. A skipped LLM step writes no LLM vault credentials; LLM-dependent workflows keep failing with provider-not-configured errors until credentials are added. Optional integration secrets collected by install mode, including LLM keys and GitHub App secrets, are written to the server vault rather than `server.env`.
When Fabro can construct a direct install URL, the token is embedded in the URL and also printed on its own line for copying. If you open the server root through a reverse proxy or another machine, paste the printed install token when prompted.
The `Object store` step offers two wizard-managed modes:
- `Local disk` for a host-local object-store root, detected by default and editable before continuing
- `AWS S3` for one shared bucket with fixed `slatedb/` and `artifacts/` prefixes
The wizard's manual-credential path stores only `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in `server.env`. It does not collect STS/session tokens or S3-compatible endpoint settings. If you need MinIO, Cloudflare R2, path-style options, or custom endpoints, finish install with local defaults and then edit `[server.slatedb]` / `[server.artifacts]` in `settings.toml` manually.
When you finish the wizard, the server writes `~/.fabro/settings.toml` and exits cleanly. Start it again to boot in configured mode:
```bash
fabro server start
```
Under a process supervisor with a restart policy (for example docker-compose `restart: unless-stopped`, systemd, or Railway's restart-on-exit) this second start happens automatically.
For headless or scripted environments where no browser is available, run `fabro install` instead — it's the same wizard as a CLI prompt flow.
Common flags:
| Flag | Default | Description |
|---|---|---|
| `--bind` | `~/.fabro/fabro.sock` | Address to bind: `IP` or `IP:port` for TCP, or a path for Unix socket |
| `--model` | — | Override default LLM model |
| `--environment` | — | Override default environment slug |
| `--max-concurrent-runs` | `5` | Maximum concurrent run executions |
See [Server Configuration](/administration/server-configuration) for the full `settings.toml` reference.
### SQLite blob storage activation
On startup, Fabro activates SQLite as the only live content-addressed blob
store before it opens routes, schedulers, workers, webhooks, reapers, or the
ready callback. The activation inventories the exact legacy SlateDB blob
prefix, checks disk headroom sized to the rows not yet imported (a warm
restart with nothing left to import only needs a small fixed headroom),
imports in bounded transactions, compares every legacy blob byte-for-byte
with SQLite, runs a live SQLite integrity check, and completes a final WAL
checkpoint. Boots that import new rows additionally re-verify every legacy
blob against SQLite and validate every SQLite blob row independently. Any
failure stops startup. Rows committed by an interrupted import are retained so the
next startup can resume, but the legacy source is never modified and there is
no fallback or dual read/write path.
For a non-empty legacy inventory, the first activation also creates the
private sibling backup
`fabro.sqlite3.pre-blob-activation.bak`. Fabro writes and validates a staging
database before publishing the backup without overwriting an existing file.
A valid retained backup is revalidated on every warm restart and is preserved
as the original pre-activation safety artifact. It is not a promise that an
older binary can safely resume after the activated server has accepted new
work; recovery after that boundary is forward-only. Empty legacy inventories
do not need this backup, but startup still validates every SQLite blob row.
Keep both the unchanged legacy `blobs/sha256` prefix and the private activation
backup for at least 30 consecutive calendar days after the first successful
production activation. Cleanup is eligible only after a successful cold
activation, a later fully verified warm restart, and 30 days of production
observation with no unresolved inventory, import, verification, integrity,
backup, or checkpoint failure. Scott must review that evidence and explicitly
authorize a separate cleanup change. Day 30 is only the earliest eligibility
date; nothing is deleted automatically, and incomplete evidence extends the
support window.
## Submitting runs
Workflows are submitted via the REST API and executed in the background. The exact request body is documented in the API reference:
```bash
curl -X POST http://localhost:3000/api/v1/runs
```
The server returns immediately with a run ID. After a start request, a background scheduler promotes `runnable` runs to `running` in FIFO order, up to the concurrency limit. Parent-generated [child runs](/execution/child-runs) can remain `pending` until a user approves them.
## Run lifecycle
1. **Submit** — `POST /api/v1/runs` creates the run with status `submitted`.
2. **Start request** — `POST /api/v1/runs/{id}/start` makes normal runs `runnable`; parent-generated [child runs](/execution/child-runs) may become `pending` with `approval_required`.
3. **Approve if needed** — Approving a pending child run makes it `runnable`; denying it fails with `approval_denied`.
4. **Schedule** — The scheduler picks up `runnable` runs up to `max_concurrent_runs`.
5. **Execute** — The engine walks the graph, streaming events to all subscribers.
6. **Complete** — The run transitions to `succeeded`, `failed`, or `dead`.
## Web UI
The web UI connects to the API server and provides:
- **Runs board** — Monitor all active runs organized by status
- **Run detail** — Real-time stage progress, event stream, diffs, and usage stats
- **Files Changed** — Browse changed files with a searchable tree, per-file status, aggregate diff stats, and split or stacked diffs
- **Settings** — Inspect server configuration, enabled integrations, storage, auth, and capacity settings
- **Start new run** — Submit workflows from the browser
- **Human-in-the-loop** — Answer agent questions through the web interface
- **Workflows** — Browse available workflows, view their graphs, and see run history
- **Insights** — SQL-based analysis across runs via DuckDB
<Frame caption="The Runs board shows all active runs organized by status.">
<img src="/images/web/runs-board.png" alt="Fabro web UI Runs board with Working, Pending, Verify, and Merge columns" />
</Frame>
<Frame caption="The run detail view shows stage progress alongside the workflow graph.">
<img src="/images/web/run-overview.png" alt="Fabro web UI run detail showing stages and workflow graph" />
</Frame>
## Event streaming
The API streams run events via [Server-Sent Events (SSE)](/api-reference/runs/stream-run-events). Every stage start, LLM call, tool invocation, and edge selection is emitted as a structured JSON event. Any HTTP client that supports SSE can subscribe — the web UI is just one consumer.
## Human-in-the-loop
Human-in-the-loop questions are served over HTTP. The engine blocks the current stage until an answer is submitted, then continues execution. See the [list questions](/api-reference/human-in-the-loop/list-run-questions) and [submit answer](/api-reference/human-in-the-loop/submit-run-answer) API reference pages.
## Authentication
The server configures auth with `server.auth.methods`:
- **`dev-token`** — Operators can call the API directly with `Authorization: Bearer fabro_dev_...`, and the web login page can accept the dev token too.
- **`github`** — End users sign in through GitHub OAuth and receive a browser session cookie.
Both methods can be enabled simultaneously:
```toml title="settings.toml"
[server.auth]
methods = ["dev-token", "github"]
[server.auth.github]
allowed_usernames = ["alice", "bob"]
```
## Demo mode
Send the `X-Fabro-Demo: 1` header on any API request to get static mock data. To enable demo mode in the web UI, set the `fabro-demo=1` cookie in your browser devtools (Application → Cookies). This lets you explore the UI without API keys or real workflow execution.
## Pointing the CLI at a server
The CLI can target a running Fabro server for commands that support a remote API. Configure `~/.fabro/settings.toml`:
```toml title="settings.toml"
[cli.target]
type = "http"
url = "https://fabro.example.com/api/v1"
```
Or use the `--server` flag:
```bash
fabro model list --server https://fabro.example.com/api/v1
```
For dev-token servers, save the token in the CLI auth store instead of exporting it for every command:
```bash
fabro auth login --server https://fabro.example.com/api/v1 --dev-token fabro_dev_...
```
`fabro model list` and `fabro model test` honor `[cli.target]` by default unless you explicitly pass `--storage-dir`. `fabro exec` remains a local agent session and only uses the server when you pass `--server`.
See [User Configuration](/reference/user-configuration#cli-target-section) for the full connection options, including client certificates for proxy-terminated HTTPS endpoints.
## Next steps
<Columns cols={2}>
<Card title="Deployment" icon="server" href="/administration/deployment">
Choose where the server runs: laptop or self-hosted Docker container.
</Card>
<Card title="Server Configuration" icon="gear" href="/administration/server-configuration">
Full settings.toml reference — authentication, reverse-proxy TLS, run defaults, and more.
</Card>
<Card title="API Reference" icon="code" href="/api-reference/overview">
REST API for submitting runs, streaming events, and managing resources.
</Card>
<Card title="How Fabro Works" icon="lightbulb" href="/core-concepts/how-fabro-works">
The workflow engine and architecture.
</Card>
</Columns>