mirror of
https://github.com/fabro-sh/fabro.git
synced 2026-10-16 04:33:16 +00:00
0f36304d2c
## Summary
Introduces a `small_default` catalog role for identifying each
provider's small/cheap utility model, and uses that model to
asynchronously generate human-readable run titles when the caller
doesn't supply one explicitly.
### Plan Summary
- **Catalog**: Add `small_default: Option<bool>` to
`ModelCatalogSettings` and `small_default: bool` to the `Model` type.
Mark built-in small defaults: `claude-haiku-4-5` (Anthropic),
`gpt-5.4-mini` (OpenAI), `gemini-3.1-flash-lite-preview` (Gemini).
Validate that each provider has at most one small default; zero is
allowed with fallback to the provider's regular default.
- **Helpers**: Add `small_default_for_provider` and
`small_default_for_configured_ids` on `Catalog`, mirroring the existing
`default_for_provider` / `default_for_configured_ids` /
`probe_for_provider` pattern.
- **Title generation**: New `run_title_generation` module in
`fabro-server` builds a prompt from workflow identity, goal, and raw run
inputs, calls `generate_object` with `max_tokens(64)` and a 10 s
timeout, normalizes output (trim, reject blank/control, truncate to 100
chars), and falls back to the deterministic title on any failure.
- **Server integration**: In the create-run handler, if no explicit
`RunManifest.title` was supplied and at least one LLM provider is ready,
spawn a detached task that generates a title and appends
`run.title.updated` — but only if the title hasn't been changed by a
concurrent user PATCH.
## What changed and why
**`small_default` vs `default`** — the existing `default` role drives
normal model selection for workflow execution and must not be disturbed.
`small_default` is a separate, additive role for lightweight metadata
work. The two roles are intentionally independent so teams can promote a
newer large model to `default` without accidentally routing title
generation there.
**Best-effort, async title enrichment** — run creation is kept
synchronous and reliable. The title task is fire-and-forget: LLM errors,
timeouts, and validation failures all silently leave the deterministic
title in place. The stale-title guard (`current.title !=
deterministic_title`) prevents the async task from clobbering a
concurrent user edit via `PATCH /runs/{id}`.
**No redaction** — per the design goal, raw input values are forwarded
to the model. This is noted explicitly in the prompt and in the module
docs.
**Prompt size bounding** — each of the three prompt sections (workflow
identity, run inputs, workflow summary) is independently capped at 4 000
characters with a `...[truncated]` marker so pathological inputs can't
produce enormous requests.
## Public interface changes
- `Model` gains `small_default: bool` in the Rust type, OpenAPI schema,
and generated TypeScript client.
- `MAX_RUN_TITLE_CHARS` is now `pub` in `fabro-types` so the
title-generation module can reuse the same limit.
- Config docs (`models.mdx`, `litellm.mdx`) document `small_default =
true` alongside `default` and `probe`.
### Fabro Details
<details>
<summary>Ran 9 stages in 61m 23s for $32.17</summary>
| Stage | Duration | Cost | Retries |
|---|---|---|---|
| start | 0s | – | 0 |
| toolchain | 2s | – | 0 |
| preflight_compile | 1m 52s | – | 0 |
| preflight_lint | 2m 5s | – | 0 |
| implement | 29m 16s | $23.26 | 0 |
| simplify_opus | 17m 15s | $6.28 | 0 |
| simplify_gpt | 7m 1s | $2.63 | 0 |
| verify | 3m 7s | – | 0 |
| fmt | 3s | – | 0 |
| **Total** | **61m 23s** | **$32.17** | **0** |
</details>
<details>
<summary>Ran <code>ImplementPlan.fabro</code> (12 nodes and 15
edges)</summary>
```dot
digraph ImplementPlan {
graph [
goal="Implement and simplify",
model_stylesheet="
* { model: claude-opus-4-7; }
"
]
rankdir=LR
start [shape=Mdiamond, label="Start"]
exit [shape=Msquare, label="Exit"]
toolchain [label="Toolchain", shape=parallelogram, script="command -v cargo >/dev/null || { curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y && sudo ln -sf $HOME/.cargo/bin/* /usr/local/bin/; }; cargo --version 2>&1", max_retries=0]
preflight_compile [label="Preflight Compile", shape=parallelogram, script="cargo check -q --workspace 2>&1", max_retries=0]
preflight_lint [label="Preflight Lint", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1", max_retries=0]
fix_lints [label="Fix Lints", prompt="The preflight lint step failed. Read the build output from context and fix all clippy lint warnings.", max_visits=3]
implement [label="Implement", prompt="Read the plan file referenced in the goal and implement every step. Make all the code changes described in the plan. Use red/green TDD.", model="gpt-55", reasoning_effort="xhigh"]
simplify_opus [label="Simplify (Opus)", prompt="@prompts/simplify.md"]
simplify_gpt [label="Simplify (GPT-55)", prompt="@prompts/simplify.md", model="gpt-55"]
verify [label="Verify", shape=parallelogram, script="cargo +nightly-2026-04-14 clippy -q --workspace --all-targets -- -D warnings 2>&1 && cargo nextest run --cargo-quiet --workspace --status-level fail 2>&1 && cargo dev docs refresh 2>&1 && cargo dev docs check 2>&1", goal_gate=true, retry_target="fixup"]
fixup [label="Fixup", prompt="The verify step failed. Read the build output from context and fix all clippy lint warnings, test failures, and generated docs errors.", max_visits=3]
fmt [label="Format", shape=parallelogram, script="cargo +nightly-2026-04-14 fmt --all 2>&1", max_retries=0]
start -> toolchain
toolchain -> preflight_compile [condition="outcome=succeeded"]
toolchain -> exit
preflight_compile -> preflight_lint [condition="outcome=succeeded"]
preflight_compile -> exit
preflight_lint -> implement [condition="outcome=succeeded"]
preflight_lint -> fix_lints
fix_lints -> preflight_lint
implement -> simplify_opus -> simplify_gpt -> verify
verify -> fmt [condition="outcome=succeeded"]
verify -> fixup
fixup -> verify
fmt -> exit
}
```
</details>
⚒️ Generated with [Fabro](https://fabro.sh)
---------
Co-authored-by: Fabro <noreply@fabro.sh>
Co-authored-by: Bryan Helmkamp <bryan@brynary.com>
133 lines
4 KiB
Text
133 lines
4 KiB
Text
---
|
|
title: "LiteLLM"
|
|
description: "Route Fabro models through a LiteLLM proxy"
|
|
---
|
|
|
|
[LiteLLM](https://docs.litellm.ai/) can run as an OpenAI-compatible proxy in front of many model providers. Fabro includes a disabled `litellm` provider entry so you can opt in from `settings.toml` without changing Fabro code.
|
|
|
|
## Prerequisites
|
|
|
|
- A running LiteLLM proxy reachable from the Fabro process
|
|
- At least one LiteLLM model name you want Fabro to route to
|
|
- A LiteLLM key or placeholder key available to Fabro
|
|
|
|
Fabro's built-in LiteLLM provider points at `http://localhost:4000/v1`. Change `base_url` if your proxy is hosted elsewhere.
|
|
|
|
## Enable the provider
|
|
|
|
Add the provider override and one or more model entries to `~/.fabro/settings.toml`:
|
|
|
|
```toml title="settings.toml"
|
|
_version = 1
|
|
|
|
[llm.providers.litellm]
|
|
enabled = true
|
|
base_url = "http://localhost:4000/v1"
|
|
|
|
[llm.models."litellm-gpt-5"]
|
|
provider = "litellm"
|
|
api_id = "gpt-5"
|
|
display_name = "LiteLLM GPT-5"
|
|
family = "litellm"
|
|
default = true
|
|
|
|
[llm.models."litellm-gpt-5".limits]
|
|
context_window = 128000
|
|
max_output = 8192
|
|
|
|
[llm.models."litellm-gpt-5".features]
|
|
tools = true
|
|
vision = false
|
|
reasoning = false
|
|
```
|
|
|
|
`api_id` is the model name Fabro sends to LiteLLM. It should match a model name configured in your LiteLLM proxy.
|
|
|
|
## Configure credentials
|
|
|
|
The LiteLLM provider checks `LITELLM_API_KEY` from the Fabro process environment first, then the `vault:LITELLM_API_KEY` server secret.
|
|
|
|
For a server-owned secret:
|
|
|
|
```bash
|
|
fabro secret set LITELLM_API_KEY sk-proxy-key
|
|
```
|
|
|
|
For a process environment variable:
|
|
|
|
```bash
|
|
export LITELLM_API_KEY=sk-proxy-key
|
|
```
|
|
|
|
If your local LiteLLM proxy does not enforce authentication, use a placeholder value such as `anything`; the OpenAI-compatible client still needs a credential value.
|
|
|
|
## Use LiteLLM models
|
|
|
|
Once the provider is enabled and at least one model is declared, use the Fabro model ID like any other catalog model:
|
|
|
|
```bash
|
|
fabro model list --provider litellm
|
|
fabro model test --model litellm-gpt-5
|
|
fabro run workflow.fabro --model litellm-gpt-5
|
|
```
|
|
|
|
In workflow stylesheets:
|
|
|
|
```dot title="workflow.fabro"
|
|
digraph Example {
|
|
graph [
|
|
model_stylesheet="
|
|
* { model: litellm-gpt-5; }
|
|
"
|
|
]
|
|
|
|
start [shape=Mdiamond, label="Start"]
|
|
work [label="Work", prompt="Use the configured LiteLLM model."]
|
|
exit [shape=Msquare, label="Exit"]
|
|
|
|
start -> work -> exit
|
|
}
|
|
```
|
|
|
|
## Declaring more models
|
|
|
|
Declare each LiteLLM-routed model explicitly so Fabro knows its provider, context window, tool support, and routing defaults:
|
|
|
|
```toml title="settings.toml"
|
|
[llm.models."litellm-fast"]
|
|
provider = "litellm"
|
|
api_id = "fast-model"
|
|
display_name = "LiteLLM Fast"
|
|
family = "litellm"
|
|
aliases = ["fast"]
|
|
|
|
[llm.models."litellm-fast".limits]
|
|
context_window = 64000
|
|
max_output = 4096
|
|
|
|
[llm.models."litellm-fast".features]
|
|
tools = true
|
|
vision = false
|
|
reasoning = false
|
|
```
|
|
|
|
Only one model for a provider should set `default = true`. You may also mark one small/cheap utility model with `small_default = true`; Fabro uses it for metadata tasks such as generated run titles and falls back to the provider default when it is omitted.
|
|
|
|
## Troubleshooting
|
|
|
|
**"No API key configured"** — Set `vault:LITELLM_API_KEY` with `fabro secret set LITELLM_API_KEY ...` or export `LITELLM_API_KEY` in the Fabro process environment.
|
|
|
|
**Connection refused** — Confirm the LiteLLM proxy is running and that `base_url` is reachable from the Fabro process. For Docker deployments, `localhost` means the Fabro container unless you point it at a host or service name.
|
|
|
|
**Unknown model from LiteLLM** — Check that the model's `api_id` matches the model name configured in LiteLLM, then run `fabro model test --model <fabro-model-id>`.
|
|
|
|
## Further reading
|
|
|
|
<Columns cols={2}>
|
|
<Card title="Models" icon="microchip" href="/core-concepts/models">
|
|
How Fabro routes model IDs, providers, and fallbacks.
|
|
</Card>
|
|
<Card title="Settings Configuration" icon="gear" href="/reference/user-configuration">
|
|
Full reference for `[llm.providers.<id>]` and `[llm.models.<id>]`.
|
|
</Card>
|
|
</Columns>
|