mirror of
https://github.com/fabro-sh/fabro.git
synced 2026-09-11 22:53:00 +00:00
7ef09968f2
The `[llm]` reference now describes `enabled`, `api_key_url`,
`stands_in_for`, `small_default`, `probe`, `family`, and the cutoffs as
lithos fields, the coding harness under `metadata.agent`, and the secret
names lithos derives for operator-defined providers. Secret-bearing
headers go in `default_headers` as `{{ secrets.NAME }}` tokens. The
integration guides enable a provider with `enabled = true` on its table.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
115 lines
4.1 KiB
Text
115 lines
4.1 KiB
Text
---
|
|
title: "LiteLLM"
|
|
description: "Route Fabro models through a LiteLLM proxy"
|
|
---
|
|
|
|
[LiteLLM](https://docs.litellm.ai/) can run as an OpenAI-compatible proxy in front of many model providers. Fabro includes a disabled `litellm` provider entry so you can opt in from `settings.toml` without changing Fabro code.
|
|
|
|
## Prerequisites
|
|
|
|
- A running LiteLLM proxy reachable from the Fabro process
|
|
- At least one LiteLLM model name you want Fabro to route to
|
|
- A LiteLLM key or placeholder key available to Fabro
|
|
|
|
Fabro's built-in LiteLLM provider points at `http://localhost:4000/v1`. Change `base_url` if your proxy is hosted elsewhere.
|
|
|
|
## Enable the provider
|
|
|
|
Add the provider override and one or more model entries to `~/.fabro/settings.toml`:
|
|
|
|
```toml title="settings.toml"
|
|
_version = 1
|
|
|
|
[llm.providers.litellm]
|
|
base_url = "http://localhost:4000/v1"
|
|
default_model = "litellm-gpt-5"
|
|
enabled = true
|
|
|
|
[llm.providers.litellm.models."litellm-gpt-5"]
|
|
display_name = "LiteLLM GPT-5"
|
|
api_model = "gpt-5"
|
|
limits = { context_tokens = 128000, max_output_tokens = 8192 }
|
|
capabilities = { text = true, tools = true }
|
|
```
|
|
|
|
`api_model` is the model name Fabro sends to LiteLLM. It should match a model name configured in your LiteLLM proxy.
|
|
|
|
## Configure credentials
|
|
|
|
For server-backed runs, store `LITELLM_API_KEY` in the Fabro server vault.
|
|
|
|
For a server-owned secret:
|
|
|
|
```bash
|
|
fabro secret set LITELLM_API_KEY sk-proxy-key
|
|
```
|
|
|
|
`fabro exec` and direct `fabro-llm` SDK usage can use an env-backed credential source explicitly:
|
|
|
|
```bash
|
|
export LITELLM_API_KEY=sk-proxy-key
|
|
```
|
|
|
|
If your local LiteLLM proxy does not enforce authentication, use a placeholder value such as `anything`; the OpenAI-compatible client still needs a credential value.
|
|
|
|
## Use LiteLLM models
|
|
|
|
Once the provider is enabled and at least one model is declared, use the Fabro model ID like any other catalog model:
|
|
|
|
```bash
|
|
fabro model list --provider litellm
|
|
fabro model test --model litellm-gpt-5
|
|
fabro run workflow.fabro --model litellm-gpt-5
|
|
```
|
|
|
|
In workflow stylesheets:
|
|
|
|
```dot title="workflow.fabro"
|
|
digraph Example {
|
|
graph [
|
|
model_stylesheet="
|
|
* { model: litellm-gpt-5; }
|
|
"
|
|
]
|
|
|
|
start [shape=Mdiamond, label="Start"]
|
|
work [label="Work", prompt="Use the configured LiteLLM model."]
|
|
exit [shape=Msquare, label="Exit"]
|
|
|
|
start -> work -> exit
|
|
}
|
|
```
|
|
|
|
## Declaring more models
|
|
|
|
Declare each LiteLLM-routed model explicitly so Fabro knows its provider, context window, tool support, and routing defaults:
|
|
|
|
```toml title="settings.toml"
|
|
[llm.providers.litellm.models."litellm-fast"]
|
|
display_name = "LiteLLM Fast"
|
|
aliases = ["fast"]
|
|
api_model = "fast-model"
|
|
limits = { context_tokens = 64000, max_output_tokens = 4096 }
|
|
capabilities = { text = true, tools = true }
|
|
```
|
|
|
|
The provider's `default_model` names its default. You may also mark one small utility model with `small_default = true`; Fabro uses it for metadata tasks such as generated run titles and falls back to the provider default when it is omitted.
|
|
|
|
## Troubleshooting
|
|
|
|
**"No API key configured"** — For runs, set `vault:LITELLM_API_KEY` with `fabro secret set LITELLM_API_KEY ...`. Exporting it in the server's shell has no effect on runs: workers start from a cleared environment and provider keys are not inherited. For `fabro exec` or direct SDK usage, export `LITELLM_API_KEY` in the invoking shell and use an env-backed credential source.
|
|
|
|
**Connection refused** — Confirm the LiteLLM proxy is running and that `base_url` is reachable from the Fabro process. For Docker deployments, `localhost` means the Fabro container unless you point it at a host or service name.
|
|
|
|
**Unknown model from LiteLLM** — Check that the model's `api_model` matches the model name configured in LiteLLM, then run `fabro model test --model <fabro-model-id>`.
|
|
|
|
## Further reading
|
|
|
|
<Columns cols={2}>
|
|
<Card title="Models" icon="microchip" href="/core-concepts/models">
|
|
How Fabro routes model IDs, providers, and fallbacks.
|
|
</Card>
|
|
<Card title="Settings Configuration" icon="gear" href="/reference/user-configuration">
|
|
Full reference for provider settings and provider-scoped model offerings.
|
|
</Card>
|
|
</Columns>
|