mirror of
https://github.com/fabro-sh/fabro.git
synced 2026-10-07 03:00:29 +00:00
feat: add opt-in LiteLLM TOML provider (#269)
## Summary - Add a disabled built-in `litellm` provider fragment backed by the OpenAI-compatible adapter and local proxy defaults. - Document how to enable LiteLLM in `settings.toml`, configure credentials, and declare explicit LiteLLM-routed models. - Register the LiteLLM integration page and cross-link it from the model and settings docs. ## Validation - `cargo test -p fabro-model` - `cargo test -p fabro-config` - `jq empty docs/public/docs.json` - `rg -n 'aliases = \["openai_compatible", "openai-compatible"\]|llm\.discovery|FABRO_LITELLM|litellm_api_key_env|x-litellm-' lib/crates/fabro-model/src/catalog/providers/litellm.toml docs/public/integrations/litellm.mdx docs/public/core-concepts/models.mdx docs/public/reference/user-configuration.mdx` returned no matches --------- Co-authored-by: Mark Ferraz <mferraz@netwoven.com>
This commit is contained in:
parent
9768651b52
commit
d52a2ccbe2
4 changed files with 187 additions and 0 deletions
|
|
@ -90,6 +90,30 @@ output_cost_per_mtok = 16.00
|
|||
cache_input_cost_per_mtok = 0.60
|
||||
```
|
||||
|
||||
For [LiteLLM](/integrations/litellm), Fabro ships a disabled provider entry. Enable it in settings and declare the models your proxy exposes:
|
||||
|
||||
```toml title="settings.toml"
|
||||
[llm.providers.litellm]
|
||||
enabled = true
|
||||
base_url = "http://localhost:4000/v1"
|
||||
|
||||
[llm.models."litellm-gpt-5"]
|
||||
provider = "litellm"
|
||||
api_id = "gpt-5"
|
||||
display_name = "LiteLLM GPT-5"
|
||||
family = "litellm"
|
||||
default = true
|
||||
|
||||
[llm.models."litellm-gpt-5".limits]
|
||||
context_window = 128000
|
||||
max_output = 8192
|
||||
|
||||
[llm.models."litellm-gpt-5".features]
|
||||
tools = true
|
||||
vision = false
|
||||
reasoning = false
|
||||
```
|
||||
|
||||
`api_id` is the model name sent to the provider API. Omit it when the Fabro model ID and provider model ID are the same.
|
||||
|
||||
Header values and credentials are typed references, not raw secrets. Use `env:<NAME>` or `credential:<id>` for provider credentials, and `{ env = "NAME" }`, `{ credential = "id" }`, or `{ literal = "value" }` for extra headers.
|
||||
|
|
|
|||
|
|
@ -93,6 +93,7 @@
|
|||
"pages": [
|
||||
"integrations/github",
|
||||
"integrations/daytona",
|
||||
"integrations/litellm",
|
||||
"integrations/slack",
|
||||
"integrations/brave-search"
|
||||
]
|
||||
|
|
|
|||
133
docs/public/integrations/litellm.mdx
Normal file
133
docs/public/integrations/litellm.mdx
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
---
|
||||
title: "LiteLLM"
|
||||
description: "Route Fabro models through a LiteLLM proxy"
|
||||
---
|
||||
|
||||
[LiteLLM](https://docs.litellm.ai/) can run as an OpenAI-compatible proxy in front of many model providers. Fabro includes a disabled `litellm` provider entry so you can opt in from `settings.toml` without changing Fabro code.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A running LiteLLM proxy reachable from the Fabro process
|
||||
- At least one LiteLLM model name you want Fabro to route to
|
||||
- A LiteLLM key or placeholder key available to Fabro
|
||||
|
||||
Fabro's built-in LiteLLM provider points at `http://localhost:4000/v1`. Change `base_url` if your proxy is hosted elsewhere.
|
||||
|
||||
## Enable the provider
|
||||
|
||||
Add the provider override and one or more model entries to `~/.fabro/settings.toml`:
|
||||
|
||||
```toml title="settings.toml"
|
||||
_version = 1
|
||||
|
||||
[llm.providers.litellm]
|
||||
enabled = true
|
||||
base_url = "http://localhost:4000/v1"
|
||||
|
||||
[llm.models."litellm-gpt-5"]
|
||||
provider = "litellm"
|
||||
api_id = "gpt-5"
|
||||
display_name = "LiteLLM GPT-5"
|
||||
family = "litellm"
|
||||
default = true
|
||||
|
||||
[llm.models."litellm-gpt-5".limits]
|
||||
context_window = 128000
|
||||
max_output = 8192
|
||||
|
||||
[llm.models."litellm-gpt-5".features]
|
||||
tools = true
|
||||
vision = false
|
||||
reasoning = false
|
||||
```
|
||||
|
||||
`api_id` is the model name Fabro sends to LiteLLM. It should match a model name configured in your LiteLLM proxy.
|
||||
|
||||
## Configure credentials
|
||||
|
||||
The LiteLLM provider checks `credential:litellm` first, then `LITELLM_API_KEY` from the Fabro process environment.
|
||||
|
||||
For a server-owned secret:
|
||||
|
||||
```bash
|
||||
fabro secret set litellm sk-proxy-key
|
||||
```
|
||||
|
||||
For a process environment variable:
|
||||
|
||||
```bash
|
||||
export LITELLM_API_KEY=sk-proxy-key
|
||||
```
|
||||
|
||||
If your local LiteLLM proxy does not enforce authentication, use a placeholder value such as `anything`; the OpenAI-compatible client still needs a credential value.
|
||||
|
||||
## Use LiteLLM models
|
||||
|
||||
Once the provider is enabled and at least one model is declared, use the Fabro model ID like any other catalog model:
|
||||
|
||||
```bash
|
||||
fabro model list --provider litellm
|
||||
fabro model test --model litellm-gpt-5
|
||||
fabro run workflow.fabro --model litellm-gpt-5
|
||||
```
|
||||
|
||||
In workflow stylesheets:
|
||||
|
||||
```dot title="workflow.fabro"
|
||||
digraph Example {
|
||||
graph [
|
||||
model_stylesheet="
|
||||
* { model: litellm-gpt-5; }
|
||||
"
|
||||
]
|
||||
|
||||
start [shape=Mdiamond, label="Start"]
|
||||
work [label="Work", prompt="Use the configured LiteLLM model."]
|
||||
exit [shape=Msquare, label="Exit"]
|
||||
|
||||
start -> work -> exit
|
||||
}
|
||||
```
|
||||
|
||||
## Declaring more models
|
||||
|
||||
Declare each LiteLLM-routed model explicitly so Fabro knows its provider, context window, tool support, and routing defaults:
|
||||
|
||||
```toml title="settings.toml"
|
||||
[llm.models."litellm-fast"]
|
||||
provider = "litellm"
|
||||
api_id = "fast-model"
|
||||
display_name = "LiteLLM Fast"
|
||||
family = "litellm"
|
||||
aliases = ["fast"]
|
||||
|
||||
[llm.models."litellm-fast".limits]
|
||||
context_window = 64000
|
||||
max_output = 4096
|
||||
|
||||
[llm.models."litellm-fast".features]
|
||||
tools = true
|
||||
vision = false
|
||||
reasoning = false
|
||||
```
|
||||
|
||||
Only one model for a provider should set `default = true`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"No API key configured"** — Set `credential:litellm` with `fabro secret set litellm ...` or export `LITELLM_API_KEY` in the Fabro process environment.
|
||||
|
||||
**Connection refused** — Confirm the LiteLLM proxy is running and that `base_url` is reachable from the Fabro process. For Docker deployments, `localhost` means the Fabro container unless you point it at a host or service name.
|
||||
|
||||
**Unknown model from LiteLLM** — Check that the model's `api_id` matches the model name configured in LiteLLM, then run `fabro model test --model <fabro-model-id>`.
|
||||
|
||||
## Further reading
|
||||
|
||||
<Columns cols={2}>
|
||||
<Card title="Models" icon="microchip" href="/core-concepts/models">
|
||||
How Fabro routes model IDs, providers, and fallbacks.
|
||||
</Card>
|
||||
<Card title="Settings Configuration" icon="gear" href="/reference/user-configuration">
|
||||
Full reference for `[llm.providers.<id>]` and `[llm.models.<id>]`.
|
||||
</Card>
|
||||
</Columns>
|
||||
29
lib/crates/fabro-model/src/catalog/providers/litellm.toml
Normal file
29
lib/crates/fabro-model/src/catalog/providers/litellm.toml
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
[providers.litellm]
|
||||
display_name = "LiteLLM"
|
||||
adapter = "openai_compatible"
|
||||
base_url = "http://localhost:4000/v1"
|
||||
credentials = ["credential:litellm", "env:LITELLM_API_KEY"]
|
||||
priority = 50
|
||||
enabled = false
|
||||
|
||||
# To enable LiteLLM, add entries like these to settings.toml:
|
||||
#
|
||||
# [llm.providers.litellm]
|
||||
# enabled = true
|
||||
# base_url = "http://localhost:4000/v1"
|
||||
#
|
||||
# [llm.models."litellm-gpt-5"]
|
||||
# provider = "litellm"
|
||||
# api_id = "gpt-5"
|
||||
# display_name = "LiteLLM GPT-5"
|
||||
# family = "litellm"
|
||||
# default = true
|
||||
#
|
||||
# [llm.models."litellm-gpt-5".limits]
|
||||
# context_window = 128000
|
||||
# max_output = 8192
|
||||
#
|
||||
# [llm.models."litellm-gpt-5".features]
|
||||
# tools = true
|
||||
# vision = false
|
||||
# reasoning = false
|
||||
Loading…
Add table
Reference in a new issue