fabro/docs/public/integrations/litellm.mdx

---
title: "LiteLLM"
description: "Route Fabro models through a LiteLLM proxy"
---

[LiteLLM](https://docs.litellm.ai/) can run as an OpenAI-compatible proxy in front of many model providers. Fabro includes a disabled `litellm` provider entry so you can opt in from `settings.toml` without changing Fabro code.

## Prerequisites

- A running LiteLLM proxy reachable from the Fabro process
- At least one LiteLLM model name you want Fabro to route to
- A LiteLLM key or placeholder key available to Fabro

Fabro's built-in LiteLLM provider points at `http://localhost:4000/v1`. Change `base_url` if your proxy is hosted elsewhere.

## Enable the provider

Add the provider override and one or more model entries to `~/.fabro/settings.toml`:

```toml title="settings.toml"
_version = 1

[llm.providers.litellm]
enabled = true
base_url = "http://localhost:4000/v1"

[llm.models."litellm-gpt-5"]
provider = "litellm"
api_id = "gpt-5"
display_name = "LiteLLM GPT-5"
family = "litellm"
default = true

[llm.models."litellm-gpt-5".limits]
context_window = 128000
max_output = 8192

[llm.models."litellm-gpt-5".features]
tools = true
vision = false
reasoning = false
```

`api_id` is the model name Fabro sends to LiteLLM. It should match a model name configured in your LiteLLM proxy.

## Configure credentials

For server-backed runs, store `LITELLM_API_KEY` in the Fabro server vault.

For a server-owned secret:

```bash
fabro secret set LITELLM_API_KEY sk-proxy-key
```

Standalone local SDK/CLI runs can still use an env-backed credential source explicitly:

```bash
export LITELLM_API_KEY=sk-proxy-key
```

If your local LiteLLM proxy does not enforce authentication, use a placeholder value such as `anything`; the OpenAI-compatible client still needs a credential value.

## Use LiteLLM models

Once the provider is enabled and at least one model is declared, use the Fabro model ID like any other catalog model:

```bash
fabro model list --provider litellm
fabro model test --model litellm-gpt-5
fabro run workflow.fabro --model litellm-gpt-5
```

In workflow stylesheets:

```dot title="workflow.fabro"
digraph Example {
    graph [
        model_stylesheet="
            * { model: litellm-gpt-5; }
        "
    ]

    start [shape=Mdiamond, label="Start"]
    work  [label="Work", prompt="Use the configured LiteLLM model."]
    exit  [shape=Msquare, label="Exit"]

    start -> work -> exit
}
```

## Declaring more models

Declare each LiteLLM-routed model explicitly so Fabro knows its provider, context window, tool support, and routing defaults:

```toml title="settings.toml"
[llm.models."litellm-fast"]
provider = "litellm"
api_id = "fast-model"
display_name = "LiteLLM Fast"
family = "litellm"
aliases = ["fast"]

[llm.models."litellm-fast".limits]
context_window = 64000
max_output = 4096

[llm.models."litellm-fast".features]
tools = true
vision = false
reasoning = false
```

Only one model for a provider should set `default = true`. You may also mark one small/cheap utility model with `small_default = true`; Fabro uses it for metadata tasks such as generated run titles and falls back to the provider default when it is omitted.

## Troubleshooting

**"No API key configured"** — For server-backed runs, set `vault:LITELLM_API_KEY` with `fabro secret set LITELLM_API_KEY ...`. For standalone local usage, export `LITELLM_API_KEY` in the invoking shell and use an env-backed credential source.

**Connection refused** — Confirm the LiteLLM proxy is running and that `base_url` is reachable from the Fabro process. For Docker deployments, `localhost` means the Fabro container unless you point it at a host or service name.

**Unknown model from LiteLLM** — Check that the model's `api_id` matches the model name configured in LiteLLM, then run `fabro model test --model <fabro-model-id>`.

## Further reading

<Columns cols={2}>
  <Card title="Models" icon="microchip" href="/core-concepts/models">
    How Fabro routes model IDs, providers, and fallbacks.
  </Card>
  <Card title="Settings Configuration" icon="gear" href="/reference/user-configuration">
    Full reference for `[llm.providers.<id>]` and `[llm.models.<id>]`.
  </Card>
</Columns>