From a27a716d11e90388bb8801bd8c1c9c61c03dd07b Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 20 Oct 2026 18:29:11 -0700 Subject: [PATCH 1/9] feat(skills): add a litellm-setup agent skill --- skills/litellm-setup/SKILL.md | 175 +++++++++++++++++++++++++++ tests/unit/test_repo_agent_skills.py | 27 +++++ 2 files changed, 202 insertions(+) create mode 100644 skills/litellm-setup/SKILL.md create mode 100644 tests/unit/test_repo_agent_skills.py diff --git a/skills/litellm-setup/SKILL.md b/skills/litellm-setup/SKILL.md new file mode 100644 index 00000000000..709972f6517 --- /dev/null +++ b/skills/litellm-setup/SKILL.md @@ -0,0 +1,175 @@ +--- +name: litellm-setup +description: Set up a self-hosted LiteLLM proxy, the OpenAI-compatible LLM gateway, on this machine, from install through provider keys, config.yaml, a master key, optional Postgres, and a verified chat completion. Use when the user asks to set up LiteLLM, install the LiteLLM proxy, self-host an LLM gateway, or put one endpoint in front of OpenAI, Anthropic, Gemini, Azure, Bedrock, Ollama, or other providers. Not for using the LiteLLM Python SDK inside an app +--- + +# Set up a LiteLLM proxy + +Take the user from nothing to a running LiteLLM proxy that has answered one chat completion. Run every command yourself unless a step says the user does it + +Never print a secret. Provider keys, the master key, the salt key, the database password, and virtual keys live only in `.env`, and `config.yaml` reads them through `os.environ/NAME`. Check that a value is there with `grep -c '^NAME=.' .env`, never with `cat .env` or `echo "$NAME"` + +## 1. Ask once + +Ask everything in one message, with defaults the user can accept by saying "go": + +- Providers and models. Default is OpenAI with its newest flagship model +- uv (default) or Docker +- Whether they want virtual keys, teams, spend tracking, or the Admin UI. Those need Postgres, plain routing does not. Default is no +- Folder and port. Defaults are `./litellm-gateway` and 4000 + +In the same message, say that someone who would rather answer prompts can run `litellm --setup` in their own terminal after step 3. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy. When the user is done, skip to step 7 and run it against their proxy, reading the master key from that file into a shell variable without printing it + +## 2. Check prerequisites + +```bash +uv --version; docker info --format '{{.ServerVersion}}'; curl --version | head -1; openssl version +``` + +The uv path needs uv, which picks or downloads a supported Python on its own (LiteLLM supports 3.10 to 3.14). If uv is missing, offer to install it the way https://docs.astral.sh/uv/getting-started/installation/ shows for their OS, and run the installer only after the user says yes. The Docker path needs a running Docker daemon. Postgres in step 5 also uses Docker unless the user already has a database + +## 3. Install + +uv: + +```bash +uv tool install 'litellm[proxy]' +litellm --version +``` + +With Postgres, install `'litellm[proxy,extra_proxy]'` instead, which adds the Prisma client the proxy uses for the database. Running `uv tool install` again with the new extras replaces the old install. If `litellm` is not found, run `uv tool update-shell` for new shells and call `"$(uv tool dir --bin)/litellm"` in this one + +Docker: + +```bash +docker pull docker.litellm.ai/berriai/litellm:main-stable +``` + +## 4. Write `.env` and `config.yaml` + +Run every command from here on in the folder from step 1. Create `.env` readable only by the user, keep it out of git, and add a master key unless it already has one. The proxy refuses to start without one, and `sk-` is the format of every key it issues: + +```bash +mkdir -p litellm-gateway && cd litellm-gateway +(umask 077; touch .env); grep -qx '.env' .gitignore 2>/dev/null || echo '.env' >> .gitignore +grep -q '^LITELLM_MASTER_KEY=.' .env || printf 'LITELLM_MASTER_KEY=sk-%s\n' "$(openssl rand -hex 32)" >> .env +``` + +For each provider key, if it is already in your environment (`[ -n "$OPENAI_API_KEY" ] && echo set`), copy it in without printing it, as in `printf 'OPENAI_API_KEY=%s\n' "$OPENAI_API_KEY" >> .env`. Otherwise ask the user to add the line `OPENAI_API_KEY=` to `.env` themselves and tell you when it is there. If they paste a key into the chat anyway, write it to `.env` and never repeat it. Keep one `NAME=value` per line with no quotes, so the shell and `docker --env-file` read the same value + +Confirm each model's current LiteLLM name in the cost map instead of trusting memory, since model names go stale fast. This lists the chat models for one provider (`openai`, `anthropic`, `gemini`, `azure`, `bedrock`, `ollama`, and so on): + +```bash +curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json \ + | python3 -c 'import json,sys; [print(k) for k,v in json.load(sys.stdin).items() if isinstance(v,dict) and v.get("litellm_provider")=="openai" and v.get("mode")=="chat"]' +``` + +In `config.yaml`, `model` is `/`, except that names which already start with the provider (like `gemini/...`) are used as they are. `model_name` is what clients will send. This example uses the newest GPT model in the cost map when this skill was written, so replace it with what the user picked: + +```yaml +model_list: + - model_name: gpt-6.1-sol + litellm_params: + model: openai/gpt-6.1-sol + api_key: os.environ/OPENAI_API_KEY + +general_settings: + master_key: os.environ/LITELLM_MASTER_KEY +``` + +Add one entry per model. Providers that need more than a key, like Azure or Bedrock, list their `litellm_params` on their page under https://docs.litellm.ai/docs/providers. Point every secret at `.env` through `os.environ/NAME` the same way + +## 5. Postgres, only when the user wants it + +Skip this step for plain routing. If the user already has a Postgres, ask them to add `DATABASE_URL=postgresql://...` to `.env` themselves and go to the uv-only part at the end of this step + +Otherwise add a database password, its URL, and a salt key to `.env`. The salt key encrypts the provider credentials the proxy stores in the database and falls back to the master key when unset, so set it now and never change it. Use host `localhost` for the uv path and `litellm-db` for the Docker path: + +```bash +pw=$(openssl rand -hex 16); host=localhost +printf 'POSTGRES_PASSWORD=%s\nDATABASE_URL=postgresql://litellm:%s@%s:5432/litellm\nLITELLM_SALT_KEY=sk-%s\n' \ + "$pw" "$pw" "$host" "$(openssl rand -hex 32)" >> .env +``` + +Start Postgres in Docker. On the uv path the proxy reaches it through the published port, so check `lsof -nP -iTCP:5432 -sTCP:LISTEN` prints nothing first, and on the Docker path drop the `-p` flag: + +```bash +docker network create litellm +set -a; . ./.env; set +a +docker run -d --name litellm-db --network litellm -p 127.0.0.1:5432:5432 \ + -e POSTGRES_USER=litellm -e POSTGRES_PASSWORD -e POSTGRES_DB=litellm \ + -v litellm-db:/var/lib/postgresql/data postgres:16 +for i in $(seq 1 60); do docker exec litellm-db pg_isready -h 127.0.0.1 -U litellm >/dev/null 2>&1 && break; sleep 1; done +``` + +Keep `-h 127.0.0.1` in the readiness check. On first boot Postgres answers on its socket during setup and then restarts, so a check without it passes too early + +On the uv path only, generate the Prisma client once inside the tool's environment. A pip or uv install doesn't ship it generated (Docker images do): + +```bash +tool="$(uv tool dir)/litellm" +schema=$("$tool/bin/python" -I -c 'import importlib.util, os; print(os.path.join(importlib.util.find_spec("litellm").submodule_search_locations[0], "proxy", "schema.prisma"))') +PATH="$tool/bin:$PATH" prisma generate --schema "$schema" +``` + +## 6. Start the proxy + +Check the port is free with `lsof -nP -iTCP:4000 -sTCP:LISTEN`, which prints nothing when it is. If it is taken, pick another port and use it everywhere below, because a proxy asked for a busy 4000 silently moves to a random port. Bind to `127.0.0.1` unless the user wants other machines to reach it + +uv, in one shell command, since the installed `litellm` does not read `.env` by itself: + +```bash +set -a; . ./.env; set +a +nohup litellm --config config.yaml --host 127.0.0.1 --port 4000 > litellm.log 2>&1 & +echo $! > litellm.pid +``` + +Docker, adding `--network litellm` when Postgres runs in Docker: + +```bash +docker run -d --name litellm --env-file .env -v "$PWD/config.yaml:/app/config.yaml:ro" -p 127.0.0.1:4000:4000 \ + docker.litellm.ai/berriai/litellm:main-stable --config /app/config.yaml --port 4000 +``` + +Wait on the health route. The first start with Postgres applies the database migrations and takes longer: + +```bash +for i in $(seq 1 180); do curl -fsS http://127.0.0.1:4000/health/liveliness >/dev/null 2>&1 && break; sleep 1; done +curl -sS http://127.0.0.1:4000/health/readiness +``` + +Readiness reports `"db":"connected"` with Postgres and `"db":"Not connected"` without it. If the proxy never comes up, read the end of `litellm.log` (or `docker logs litellm`) and check the troubleshooting list + +## 7. Verify with a chat completion + +Use a `model_name` from `config.yaml`: + +```bash +set -a; . ./.env; set +a +curl -sS http://127.0.0.1:4000/v1/chat/completions \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \ + -d '{"model": "gpt-6.1-sol", "messages": [{"role": "user", "content": "Say hello in five words."}]}' +``` + +Show the user the reply in `choices[0].message.content`. An error here means the setup is not done, so fix it before handing off + +## 8. Hand off + +Tell the user, without printing any secret: + +- The base URL is `http://127.0.0.1:4000`. Any OpenAI-compatible client works with it as the base URL and the master key or a virtual key as the API key +- The master key is `LITELLM_MASTER_KEY` in the folder's `.env` +- To stop it, `kill "$(cat litellm.pid)"` (uv) or `docker stop litellm` (Docker). To start it again, rerun the step 6 command or `docker start litellm` +- With Postgres, the Admin UI is at `http://127.0.0.1:4000/ui` (user `admin`, password the master key), and `POST /key/generate` with the master key creates virtual keys with their own budgets and model lists. Without Postgres, both need step 5 first +- To manage the running proxy from a coding agent (keys, teams, users, models, MCP servers), install `npx skills add BerriAI/litellm-skills -g` + +## Troubleshooting + +- `refused to start: no master key is set` means `.env` was not exported into the shell that started `litellm`, or `--env-file .env` is missing from `docker run` +- `a database URL is set but the prisma CLI is neither on PATH nor importable` means the install lacks `extra_proxy`, so rerun step 3 with it +- `The Client hasn't been generated yet` or `Unable to find Prisma binaries` means the Prisma generate command in step 5 has not run +- `Incorrect API key provided` or another provider `AuthenticationError` means a provider key in `.env` is wrong. Fix it and restart, since the proxy reads the environment only at startup +- `Invalid model name passed in model=...` means the request's `model` is not a `model_name` in `config.yaml`. `GET /v1/models` lists the ones the key can use +- `No connected db.` on a proxy without Postgres means the bearer token is not the master key +- `password authentication failed` from Postgres means an old `litellm-db` volume kept the password it was created with. Put that password back in `.env`, or with the user's OK run `docker rm -f litellm-db && docker volume rm litellm-db` and redo step 5 +- Inside Docker, `localhost` in `config.yaml` (an Ollama `api_base`, say) is the container itself. Use `host.docker.internal`, and on Linux also pass `--add-host=host.docker.internal:host-gateway` diff --git a/tests/unit/test_repo_agent_skills.py b/tests/unit/test_repo_agent_skills.py new file mode 100644 index 00000000000..afeb498bd7b --- /dev/null +++ b/tests/unit/test_repo_agent_skills.py @@ -0,0 +1,27 @@ +from pathlib import Path +from typing import Final + +import pytest +import yaml + +_SKILLS_DIR: Final = Path(__file__).resolve().parents[2] / "skills" +_SKILL_DIRECTORIES: Final = tuple(sorted(skill_file.parent for skill_file in _SKILLS_DIR.glob("*/SKILL.md"))) + + +def _frontmatter(skill_directory: Path) -> object: + opening, frontmatter, _body = (skill_directory / "SKILL.md").read_text(encoding="utf-8").split("---\n", 2) + assert opening == "", f"{skill_directory.name}/SKILL.md must open with a --- frontmatter block" + return yaml.safe_load(frontmatter) + + +def test_litellm_setup_skill_sits_where_skill_installers_look() -> None: + assert _SKILLS_DIR / "litellm-setup" in _SKILL_DIRECTORIES + + +@pytest.mark.parametrize("skill_directory", _SKILL_DIRECTORIES, ids=lambda path: path.name) +def test_skill_frontmatter_has_string_name_matching_its_directory_and_a_description(skill_directory: Path) -> None: + frontmatter: Final = _frontmatter(skill_directory) + assert isinstance(frontmatter, dict), frontmatter + assert frontmatter.get("name") == skill_directory.name, frontmatter + description: Final = frontmatter.get("description") + assert isinstance(description, str) and description.strip(), frontmatter From c27e3d2949a4e4bebe82f40324ae1297a7986ceb Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 20 Oct 2026 19:07:07 -0700 Subject: [PATCH 2/9] fix(skills): run the quickstart for Docker with Postgres in litellm-setup Use scripts/quickstart.sh for the Docker plus Postgres path instead of a hand-rolled Postgres container, keep uv with the user's own Postgres, load .env as data instead of sourcing it, pin the uv Python range, chmod an existing .env, list bedrock_converse and vertex_ai rows in the cost map lookup, ask before editing the shell profile, and stop the health wait when the proxy exits --- skills/litellm-setup/SKILL.md | 123 ++++++++++++++++++---------------- 1 file changed, 66 insertions(+), 57 deletions(-) diff --git a/skills/litellm-setup/SKILL.md b/skills/litellm-setup/SKILL.md index 709972f6517..59b26afdf95 100644 --- a/skills/litellm-setup/SKILL.md +++ b/skills/litellm-setup/SKILL.md @@ -1,43 +1,71 @@ --- name: litellm-setup -description: Set up a self-hosted LiteLLM proxy, the OpenAI-compatible LLM gateway, on this machine, from install through provider keys, config.yaml, a master key, optional Postgres, and a verified chat completion. Use when the user asks to set up LiteLLM, install the LiteLLM proxy, self-host an LLM gateway, or put one endpoint in front of OpenAI, Anthropic, Gemini, Azure, Bedrock, Ollama, or other providers. Not for using the LiteLLM Python SDK inside an app +description: Set up a self-hosted LiteLLM proxy, the OpenAI-compatible LLM gateway, on this machine, from install through provider keys, config.yaml, a master key, optional Postgres and Admin UI, and a verified chat completion. Use when the user asks to set up LiteLLM, install the LiteLLM proxy, self-host an LLM gateway, or put one endpoint in front of OpenAI, Anthropic, Gemini, Azure, Bedrock, Ollama, or other providers. Not for using the LiteLLM Python SDK inside an app --- # Set up a LiteLLM proxy Take the user from nothing to a running LiteLLM proxy that has answered one chat completion. Run every command yourself unless a step says the user does it -Never print a secret. Provider keys, the master key, the salt key, the database password, and virtual keys live only in `.env`, and `config.yaml` reads them through `os.environ/NAME`. Check that a value is there with `grep -c '^NAME=.' .env`, never with `cat .env` or `echo "$NAME"` +Never print a secret. Provider keys, the master key, the salt key, the database URL, and virtual keys live only in `.env`, and `config.yaml` reads them through `os.environ/NAME`. Check that a value is there with `grep -c '^NAME=.' .env`, never with `cat .env` or `echo "$NAME"`. Load `.env` into a shell with this line, which exports each line as it is, so a `&` or `$` in a value stays literal. The steps that need it start with it: + +```bash +while IFS= read -r line || [ -n "$line" ]; do case "$line" in [A-Za-z_]*=*) export "$line" ;; esac; done < .env +``` ## 1. Ask once Ask everything in one message, with defaults the user can accept by saying "go": - Providers and models. Default is OpenAI with its newest flagship model -- uv (default) or Docker - Whether they want virtual keys, teams, spend tracking, or the Admin UI. Those need Postgres, plain routing does not. Default is no -- Folder and port. Defaults are `./litellm-gateway` and 4000 +- uv (default) or Docker. With Postgres, Docker runs the repo's quickstart, which brings its own database, and uv needs a Postgres they already have +- Folder and port. Defaults are `./litellm-proxy` (`./litellm-gateway` for the quickstart) and 4000 -In the same message, say that someone who would rather answer prompts can run `litellm --setup` in their own terminal after step 3. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy. When the user is done, skip to step 7 and run it against their proxy, reading the master key from that file into a shell variable without printing it +Then follow the steps for their answers. Plain routing skips steps 3 and 6. The quickstart is steps 2, 3, 8, and 9. uv with their own Postgres skips step 3 + +In the same message, say that someone who would rather answer prompts can run `litellm --setup` in their own terminal after step 4. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy. When the user is done, skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it ## 2. Check prerequisites ```bash -uv --version; docker info --format '{{.ServerVersion}}'; curl --version | head -1; openssl version +uv --version; docker info --format '{{.ServerVersion}}'; docker compose version; curl --version | head -1; openssl version ``` -The uv path needs uv, which picks or downloads a supported Python on its own (LiteLLM supports 3.10 to 3.14). If uv is missing, offer to install it the way https://docs.astral.sh/uv/getting-started/installation/ shows for their OS, and run the installer only after the user says yes. The Docker path needs a running Docker daemon. Postgres in step 5 also uses Docker unless the user already has a database +The uv path needs uv, which downloads a supported Python (LiteLLM supports 3.10 to 3.14) when the `--python` range in step 4 rules out the one on the machine. If uv is missing, offer to install it the way https://docs.astral.sh/uv/getting-started/installation/ shows for their OS, and run the installer only after the user says yes. The Docker path needs a running Docker daemon, and the quickstart also needs Docker Compose v2 -## 3. Install +Check the port is free with `lsof -nP -iTCP:4000 -sTCP:LISTEN`, which prints nothing when it is. If it is taken, pick another port and use it everywhere below, because a proxy asked for a busy 4000 silently moves to a random port + +## 3. Postgres with Docker: run the quickstart + +The repo's quickstart starts the gateway, Postgres, and the Admin UI in Docker, listening on this machine only, and installs a `litellm-gateway` command that stops, starts, updates, or removes them. It asks nothing when it has no terminal. It writes the master key, salt key, and database password to its own `.env` and never prints them. Download it, then run it with the folder and port from step 1. It returns once the gateway is ready: + +```bash +curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/quickstart.sh -o quickstart.sh +LITELLM_DIR="$PWD/litellm-gateway" LITELLM_PORT=4000 sh quickstart.sh --yes +cd litellm-gateway +``` + +Models live in the database here, so add each one through the API instead of a `config.yaml`, naming it the way step 5 says. Pipe the request body in, so the provider key never shows up in output or on a command line. If the key is not in your environment, ask the user to add the line `OPENAI_API_KEY=` to this `.env` themselves, and run the loader line first: + +```bash +python3 -c 'import json, os, sys; json.dump({"model_name": "gpt-6.1-sol", "litellm_params": {"model": "openai/gpt-6.1-sol", "api_key": os.environ["OPENAI_API_KEY"]}}, sys.stdout)' \ + | curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:4000/model/new \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d @- +``` + +Run that in one shell command that starts with the loader line, since the master key is in `.env`. `200` means the model was added. Then go to step 8 + +## 4. Install uv: ```bash -uv tool install 'litellm[proxy]' +uv tool install --python '>=3.10,<3.15' 'litellm[proxy]' litellm --version ``` -With Postgres, install `'litellm[proxy,extra_proxy]'` instead, which adds the Prisma client the proxy uses for the database. Running `uv tool install` again with the new extras replaces the old install. If `litellm` is not found, run `uv tool update-shell` for new shells and call `"$(uv tool dir --bin)/litellm"` in this one +With their own Postgres, install `'litellm[proxy,extra_proxy]'` instead, with the same `--python` range, which adds the Prisma client the proxy uses for the database. Running `uv tool install` again with the new extras replaces the old install. If `litellm` is not found, call `"$(uv tool dir --bin)/litellm"` in this shell. `uv tool update-shell` puts it on the PATH of new shells by editing the user's shell profile, so run it only after the user says yes Docker: @@ -45,26 +73,26 @@ Docker: docker pull docker.litellm.ai/berriai/litellm:main-stable ``` -## 4. Write `.env` and `config.yaml` +## 5. Write `.env` and `config.yaml` Run every command from here on in the folder from step 1. Create `.env` readable only by the user, keep it out of git, and add a master key unless it already has one. The proxy refuses to start without one, and `sk-` is the format of every key it issues: ```bash -mkdir -p litellm-gateway && cd litellm-gateway -(umask 077; touch .env); grep -qx '.env' .gitignore 2>/dev/null || echo '.env' >> .gitignore +mkdir -p litellm-proxy && cd litellm-proxy +touch .env && chmod 600 .env; grep -qx '.env' .gitignore 2>/dev/null || echo '.env' >> .gitignore grep -q '^LITELLM_MASTER_KEY=.' .env || printf 'LITELLM_MASTER_KEY=sk-%s\n' "$(openssl rand -hex 32)" >> .env ``` -For each provider key, if it is already in your environment (`[ -n "$OPENAI_API_KEY" ] && echo set`), copy it in without printing it, as in `printf 'OPENAI_API_KEY=%s\n' "$OPENAI_API_KEY" >> .env`. Otherwise ask the user to add the line `OPENAI_API_KEY=` to `.env` themselves and tell you when it is there. If they paste a key into the chat anyway, write it to `.env` and never repeat it. Keep one `NAME=value` per line with no quotes, so the shell and `docker --env-file` read the same value +For each provider key, if it is already in your environment (`[ -n "$OPENAI_API_KEY" ] && echo set`), copy it in without printing it, as in `printf 'OPENAI_API_KEY=%s\n' "$OPENAI_API_KEY" >> .env`. Otherwise ask the user to add the line `OPENAI_API_KEY=` to `.env` themselves and tell you when it is there. If they paste a key into the chat anyway, write it to `.env` and never repeat it. Keep one `NAME=value` per line with no quotes, so `docker --env-file` and the loader line read the same value -Confirm each model's current LiteLLM name in the cost map instead of trusting memory, since model names go stale fast. This lists the chat models for one provider (`openai`, `anthropic`, `gemini`, `azure`, `bedrock`, `ollama`, and so on): +Confirm each model's current LiteLLM name in the cost map instead of trusting memory, since model names go stale fast. This lists the chat models of the provider named last (`openai`, `anthropic`, `gemini`, `azure`, `bedrock`, `vertex_ai`, and so on). It matches by prefix, so `bedrock` also lists the `bedrock_converse` rows, which hold the newer Bedrock models, and `vertex_ai` lists every Vertex family. For Ollama, use a model from `ollama list` instead: ```bash curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json \ - | python3 -c 'import json,sys; [print(k) for k,v in json.load(sys.stdin).items() if isinstance(v,dict) and v.get("litellm_provider")=="openai" and v.get("mode")=="chat"]' + | python3 -c 'import json,sys; [print(v["litellm_provider"], k) for k,v in json.load(sys.stdin).items() if isinstance(v,dict) and str(v.get("litellm_provider","")).startswith(sys.argv[1]) and v.get("mode")=="chat"]' openai ``` -In `config.yaml`, `model` is `/`, except that names which already start with the provider (like `gemini/...`) are used as they are. `model_name` is what clients will send. This example uses the newest GPT model in the cost map when this skill was written, so replace it with what the user picked: +In `config.yaml`, `model` is the name with its provider in front, as in `openai/gpt-6.1-sol` or `bedrock/us.anthropic.`. Names that already start with a provider and a slash (`gemini/...`, `vertex_ai/...`) are used as they are. `model_name` is what clients will send. This example uses the newest GPT model in the cost map when this skill was written, so replace it with what the user picked: ```yaml model_list: @@ -79,32 +107,15 @@ general_settings: Add one entry per model. Providers that need more than a key, like Azure or Bedrock, list their `litellm_params` on their page under https://docs.litellm.ai/docs/providers. Point every secret at `.env` through `os.environ/NAME` the same way -## 5. Postgres, only when the user wants it +## 6. Their own Postgres, on the uv path -Skip this step for plain routing. If the user already has a Postgres, ask them to add `DATABASE_URL=postgresql://...` to `.env` themselves and go to the uv-only part at the end of this step - -Otherwise add a database password, its URL, and a salt key to `.env`. The salt key encrypts the provider credentials the proxy stores in the database and falls back to the master key when unset, so set it now and never change it. Use host `localhost` for the uv path and `litellm-db` for the Docker path: +Ask the user to add `DATABASE_URL=postgresql://...` for their database to `.env` themselves. Then add a salt key. It encrypts the provider credentials the proxy stores in the database and falls back to the master key when unset, so set it once and never change it: ```bash -pw=$(openssl rand -hex 16); host=localhost -printf 'POSTGRES_PASSWORD=%s\nDATABASE_URL=postgresql://litellm:%s@%s:5432/litellm\nLITELLM_SALT_KEY=sk-%s\n' \ - "$pw" "$pw" "$host" "$(openssl rand -hex 32)" >> .env +grep -q '^LITELLM_SALT_KEY=.' .env || printf 'LITELLM_SALT_KEY=sk-%s\n' "$(openssl rand -hex 32)" >> .env ``` -Start Postgres in Docker. On the uv path the proxy reaches it through the published port, so check `lsof -nP -iTCP:5432 -sTCP:LISTEN` prints nothing first, and on the Docker path drop the `-p` flag: - -```bash -docker network create litellm -set -a; . ./.env; set +a -docker run -d --name litellm-db --network litellm -p 127.0.0.1:5432:5432 \ - -e POSTGRES_USER=litellm -e POSTGRES_PASSWORD -e POSTGRES_DB=litellm \ - -v litellm-db:/var/lib/postgresql/data postgres:16 -for i in $(seq 1 60); do docker exec litellm-db pg_isready -h 127.0.0.1 -U litellm >/dev/null 2>&1 && break; sleep 1; done -``` - -Keep `-h 127.0.0.1` in the readiness check. On first boot Postgres answers on its socket during setup and then restarts, so a check without it passes too early - -On the uv path only, generate the Prisma client once inside the tool's environment. A pip or uv install doesn't ship it generated (Docker images do): +Generate the Prisma client once inside the tool's environment. A pip or uv install doesn't ship it generated (Docker images do): ```bash tool="$(uv tool dir)/litellm" @@ -112,40 +123,38 @@ schema=$("$tool/bin/python" -I -c 'import importlib.util, os; print(os.path.join PATH="$tool/bin:$PATH" prisma generate --schema "$schema" ``` -## 6. Start the proxy +## 7. Start the proxy -Check the port is free with `lsof -nP -iTCP:4000 -sTCP:LISTEN`, which prints nothing when it is. If it is taken, pick another port and use it everywhere below, because a proxy asked for a busy 4000 silently moves to a random port. Bind to `127.0.0.1` unless the user wants other machines to reach it +Bind to `127.0.0.1` unless the user wants other machines to reach it -uv, in one shell command, since the installed `litellm` does not read `.env` by itself: +uv, in one shell command that starts with the loader line, since the installed `litellm` does not read `.env` by itself: ```bash -set -a; . ./.env; set +a nohup litellm --config config.yaml --host 127.0.0.1 --port 4000 > litellm.log 2>&1 & echo $! > litellm.pid ``` -Docker, adding `--network litellm` when Postgres runs in Docker: +Docker: ```bash docker run -d --name litellm --env-file .env -v "$PWD/config.yaml:/app/config.yaml:ro" -p 127.0.0.1:4000:4000 \ docker.litellm.ai/berriai/litellm:main-stable --config /app/config.yaml --port 4000 ``` -Wait on the health route. The first start with Postgres applies the database migrations and takes longer: +Wait on the health route, stopping early if the proxy exits. The first start with Postgres applies the database migrations and takes longer. Define `alive` for the path you took, `alive() { kill -0 "$(cat litellm.pid)" 2>/dev/null; }` for uv or `alive() { [ "$(docker inspect -f '{{.State.Running}}' litellm 2>/dev/null)" = true ]; }` for Docker, in the same shell command as the loop: ```bash -for i in $(seq 1 180); do curl -fsS http://127.0.0.1:4000/health/liveliness >/dev/null 2>&1 && break; sleep 1; done +for i in $(seq 1 180); do curl -fsS http://127.0.0.1:4000/health/liveliness >/dev/null 2>&1 && break; alive || break; sleep 1; done curl -sS http://127.0.0.1:4000/health/readiness ``` Readiness reports `"db":"connected"` with Postgres and `"db":"Not connected"` without it. If the proxy never comes up, read the end of `litellm.log` (or `docker logs litellm`) and check the troubleshooting list -## 7. Verify with a chat completion +## 8. Verify with a chat completion -Use a `model_name` from `config.yaml`: +Use a `model_name` from `config.yaml` (or step 3), in one shell command that starts with the loader line: ```bash -set -a; . ./.env; set +a curl -sS http://127.0.0.1:4000/v1/chat/completions \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \ -d '{"model": "gpt-6.1-sol", "messages": [{"role": "user", "content": "Say hello in five words."}]}' @@ -153,23 +162,23 @@ curl -sS http://127.0.0.1:4000/v1/chat/completions \ Show the user the reply in `choices[0].message.content`. An error here means the setup is not done, so fix it before handing off -## 8. Hand off +## 9. Hand off Tell the user, without printing any secret: - The base URL is `http://127.0.0.1:4000`. Any OpenAI-compatible client works with it as the base URL and the master key or a virtual key as the API key - The master key is `LITELLM_MASTER_KEY` in the folder's `.env` -- To stop it, `kill "$(cat litellm.pid)"` (uv) or `docker stop litellm` (Docker). To start it again, rerun the step 6 command or `docker start litellm` -- With Postgres, the Admin UI is at `http://127.0.0.1:4000/ui` (user `admin`, password the master key), and `POST /key/generate` with the master key creates virtual keys with their own budgets and model lists. Without Postgres, both need step 5 first +- To stop it, `kill "$(cat litellm.pid)"` (uv), `docker stop litellm` (Docker), or `litellm-gateway stop` (quickstart). To start it again, rerun the step 7 command, `docker start litellm`, or `litellm-gateway start`. `litellm-gateway update` and `litellm-gateway uninstall` update or remove a quickstart install +- With Postgres, the Admin UI is at `http://127.0.0.1:4000/ui` (user `admin`, password the master key), and `POST /key/generate` with the master key creates virtual keys with their own budgets and model lists. Without Postgres, both need step 3 or 6 first - To manage the running proxy from a coding agent (keys, teams, users, models, MCP servers), install `npx skills add BerriAI/litellm-skills -g` ## Troubleshooting -- `refused to start: no master key is set` means `.env` was not exported into the shell that started `litellm`, or `--env-file .env` is missing from `docker run` -- `a database URL is set but the prisma CLI is neither on PATH nor importable` means the install lacks `extra_proxy`, so rerun step 3 with it -- `The Client hasn't been generated yet` or `Unable to find Prisma binaries` means the Prisma generate command in step 5 has not run -- `Incorrect API key provided` or another provider `AuthenticationError` means a provider key in `.env` is wrong. Fix it and restart, since the proxy reads the environment only at startup -- `Invalid model name passed in model=...` means the request's `model` is not a `model_name` in `config.yaml`. `GET /v1/models` lists the ones the key can use +- `refused to start: no master key is set` means the loader line did not run in the shell command that started `litellm`, or `--env-file .env` is missing from `docker run` +- `a database URL is set but the prisma CLI is neither on PATH nor importable` means the install lacks `extra_proxy`, so rerun step 4 with it +- `The Client hasn't been generated yet` or `Unable to find Prisma binaries` means the Prisma generate command in step 6 has not run +- `Incorrect API key provided` or another provider `AuthenticationError` means a provider key is wrong. Fix it in `.env` and restart, since the proxy reads the environment only at startup. On the quickstart, add the model again with the right key +- `Invalid model name passed in model=...` means the request's `model` is not a `model_name` in `config.yaml` or step 3. `GET /v1/models` lists the ones the key can use - `No connected db.` on a proxy without Postgres means the bearer token is not the master key -- `password authentication failed` from Postgres means an old `litellm-db` volume kept the password it was created with. Put that password back in `.env`, or with the user's OK run `docker rm -f litellm-db && docker volume rm litellm-db` and redo step 5 +- `Conflict. The container name "/litellm" is already in use` means an earlier run left a container. `docker start litellm` reuses it, or with the user's OK `docker rm -f litellm` and rerun step 7 - Inside Docker, `localhost` in `config.yaml` (an Ollama `api_base`, say) is the container itself. Use `host.docker.internal`, and on Linux also pass `--add-host=host.docker.internal:host-gateway` From 5bb66a75a3280c0d9d855e5f7c9ef2d583d6d997 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 20 Oct 2026 19:48:13 -0700 Subject: [PATCH 3/9] fix(skills): guide empty Postgres, docker sudo, and key fixes in litellm-setup Ask for a new empty database on the uv path and DIRECT_URL behind a pooler, add the baseline error to troubleshooting, and drop inherited database settings in the loader line. Check python3 and lsof, explain Docker's permission denied, give quickstart commands by the folder's path, and fix provider keys by recreating the container or deleting the quickstart model. Drop the structure-only test. --- skills/litellm-setup/SKILL.md | 19 ++++++++++--------- tests/unit/test_repo_agent_skills.py | 4 ---- 2 files changed, 10 insertions(+), 13 deletions(-) diff --git a/skills/litellm-setup/SKILL.md b/skills/litellm-setup/SKILL.md index 59b26afdf95..2d4f3e882f3 100644 --- a/skills/litellm-setup/SKILL.md +++ b/skills/litellm-setup/SKILL.md @@ -7,10 +7,10 @@ description: Set up a self-hosted LiteLLM proxy, the OpenAI-compatible LLM gatew Take the user from nothing to a running LiteLLM proxy that has answered one chat completion. Run every command yourself unless a step says the user does it -Never print a secret. Provider keys, the master key, the salt key, the database URL, and virtual keys live only in `.env`, and `config.yaml` reads them through `os.environ/NAME`. Check that a value is there with `grep -c '^NAME=.' .env`, never with `cat .env` or `echo "$NAME"`. Load `.env` into a shell with this line, which exports each line as it is, so a `&` or `$` in a value stays literal. The steps that need it start with it: +Never print a secret. Provider keys, the master key, the salt key, the database URL, and virtual keys live only in `.env`, and `config.yaml` reads them through `os.environ/NAME`. Check that a value is there with `grep -c '^NAME=.' .env`, never with `cat .env` or `echo "$NAME"`. Load `.env` into a shell with this line, which exports each line as it is, so a `&` or `$` in a value stays literal, and first drops any database settings the shell already had, so only `.env` picks the database. The steps that need it start with it: ```bash -while IFS= read -r line || [ -n "$line" ]; do case "$line" in [A-Za-z_]*=*) export "$line" ;; esac; done < .env +unset DATABASE_URL DIRECT_URL DATABASE_HOST; while IFS= read -r line || [ -n "$line" ]; do case "$line" in [A-Za-z_]*=*) export "$line" ;; esac; done < .env ``` ## 1. Ask once @@ -19,7 +19,7 @@ Ask everything in one message, with defaults the user can accept by saying "go": - Providers and models. Default is OpenAI with its newest flagship model - Whether they want virtual keys, teams, spend tracking, or the Admin UI. Those need Postgres, plain routing does not. Default is no -- uv (default) or Docker. With Postgres, Docker runs the repo's quickstart, which brings its own database, and uv needs a Postgres they already have +- uv (default) or Docker. With Postgres, Docker runs the repo's quickstart, which brings its own database, and uv needs a Postgres server they already have, with a new, empty database for LiteLLM - Folder and port. Defaults are `./litellm-proxy` (`./litellm-gateway` for the quickstart) and 4000 Then follow the steps for their answers. Plain routing skips steps 3 and 6. The quickstart is steps 2, 3, 8, and 9. uv with their own Postgres skips step 3 @@ -29,16 +29,16 @@ In the same message, say that someone who would rather answer prompts can run `l ## 2. Check prerequisites ```bash -uv --version; docker info --format '{{.ServerVersion}}'; docker compose version; curl --version | head -1; openssl version +uv --version; docker info --format '{{.ServerVersion}}'; docker compose version; curl --version | head -1; openssl version; command -v python3 lsof ``` -The uv path needs uv, which downloads a supported Python (LiteLLM supports 3.10 to 3.14) when the `--python` range in step 4 rules out the one on the machine. If uv is missing, offer to install it the way https://docs.astral.sh/uv/getting-started/installation/ shows for their OS, and run the installer only after the user says yes. The Docker path needs a running Docker daemon, and the quickstart also needs Docker Compose v2 +Every path needs curl, openssl, python3, and lsof, so ask the user to install any that is missing. The uv path needs uv, which downloads a supported Python (LiteLLM supports 3.10 to 3.14) when the `--python` range in step 4 rules out the one on the machine. If uv is missing, offer to install it the way https://docs.astral.sh/uv/getting-started/installation/ shows for their OS, and run the installer only after the user says yes. The Docker path needs a running Docker daemon, and the quickstart also needs Docker Compose v2. If `docker info` says `permission denied`, the user's account cannot use Docker without sudo. Offer the uv path, or ask them to add themselves to the `docker` group the way https://docs.docker.com/engine/install/linux-postinstall/ shows and start a new login session before you check again Check the port is free with `lsof -nP -iTCP:4000 -sTCP:LISTEN`, which prints nothing when it is. If it is taken, pick another port and use it everywhere below, because a proxy asked for a busy 4000 silently moves to a random port ## 3. Postgres with Docker: run the quickstart -The repo's quickstart starts the gateway, Postgres, and the Admin UI in Docker, listening on this machine only, and installs a `litellm-gateway` command that stops, starts, updates, or removes them. It asks nothing when it has no terminal. It writes the master key, salt key, and database password to its own `.env` and never prints them. Download it, then run it with the folder and port from step 1. It returns once the gateway is ready: +The repo's quickstart starts the gateway, Postgres, and the Admin UI in Docker, listening on this machine only, and puts a `litellm-gateway` command in that folder that stops, starts, updates, or removes them. It asks nothing when it has no terminal. It writes the master key, salt key, and database password to its own `.env` and never prints them. Download it, then run it with the folder and port from step 1. It returns once the gateway is ready: ```bash curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/quickstart.sh -o quickstart.sh @@ -109,7 +109,7 @@ Add one entry per model. Providers that need more than a key, like Azure or Bedr ## 6. Their own Postgres, on the uv path -Ask the user to add `DATABASE_URL=postgresql://...` for their database to `.env` themselves. Then add a salt key. It encrypts the provider credentials the proxy stores in the database and falls back to the master key when unset, so set it once and never change it: +Ask the user to create a new, empty database for LiteLLM on their server (`CREATE DATABASE litellm;`) and add `DATABASE_URL=postgresql://...` for it to `.env` themselves. The proxy creates its tables there on first start and stops on a database that already holds other tables. If that URL goes through a transaction pooler such as PgBouncer, they also add `DIRECT_URL` for the same database without the pooler, which the migrations use. Then add a salt key. It encrypts the provider credentials the proxy stores in the database and falls back to the master key when unset, so set it once and never change it: ```bash grep -q '^LITELLM_SALT_KEY=.' .env || printf 'LITELLM_SALT_KEY=sk-%s\n' "$(openssl rand -hex 32)" >> .env @@ -168,7 +168,7 @@ Tell the user, without printing any secret: - The base URL is `http://127.0.0.1:4000`. Any OpenAI-compatible client works with it as the base URL and the master key or a virtual key as the API key - The master key is `LITELLM_MASTER_KEY` in the folder's `.env` -- To stop it, `kill "$(cat litellm.pid)"` (uv), `docker stop litellm` (Docker), or `litellm-gateway stop` (quickstart). To start it again, rerun the step 7 command, `docker start litellm`, or `litellm-gateway start`. `litellm-gateway update` and `litellm-gateway uninstall` update or remove a quickstart install +- To stop it, `kill "$(cat litellm.pid)"` (uv), `docker stop litellm` (Docker), or `/litellm-gateway stop` (quickstart). To start it again, rerun the step 7 command, `docker start litellm`, or `/litellm-gateway start`. `update` and `uninstall` update or remove a quickstart install. Give the quickstart commands with the folder's full path, since a bare `litellm-gateway` is on PATH only when `~/.local/bin` is, and can belong to an earlier install - With Postgres, the Admin UI is at `http://127.0.0.1:4000/ui` (user `admin`, password the master key), and `POST /key/generate` with the master key creates virtual keys with their own budgets and model lists. Without Postgres, both need step 3 or 6 first - To manage the running proxy from a coding agent (keys, teams, users, models, MCP servers), install `npx skills add BerriAI/litellm-skills -g` @@ -177,7 +177,8 @@ Tell the user, without printing any secret: - `refused to start: no master key is set` means the loader line did not run in the shell command that started `litellm`, or `--env-file .env` is missing from `docker run` - `a database URL is set but the prisma CLI is neither on PATH nor importable` means the install lacks `extra_proxy`, so rerun step 4 with it - `The Client hasn't been generated yet` or `Unable to find Prisma binaries` means the Prisma generate command in step 6 has not run -- `Incorrect API key provided` or another provider `AuthenticationError` means a provider key is wrong. Fix it in `.env` and restart, since the proxy reads the environment only at startup. On the quickstart, add the model again with the right key +- `Incorrect API key provided` or another provider `AuthenticationError` means a provider key is wrong. Fix it in `.env` and restart, since the proxy reads the environment only at startup. On uv, stop it and rerun step 7. On Docker, `docker rm -f litellm` and rerun step 7, since `docker restart` keeps the environment the container was created with. On the quickstart the key is saved with the model, so delete the model and add it again: `GET /model/info` lists each model's `model_info.id`, and `POST /model/delete` with `{"id": ""}` deletes it +- `Cannot automatically baseline this database` means `DATABASE_URL` points at a database that already holds tables LiteLLM did not create. Point it at a new, empty database - `Invalid model name passed in model=...` means the request's `model` is not a `model_name` in `config.yaml` or step 3. `GET /v1/models` lists the ones the key can use - `No connected db.` on a proxy without Postgres means the bearer token is not the master key - `Conflict. The container name "/litellm" is already in use` means an earlier run left a container. `docker start litellm` reuses it, or with the user's OK `docker rm -f litellm` and rerun step 7 diff --git a/tests/unit/test_repo_agent_skills.py b/tests/unit/test_repo_agent_skills.py index afeb498bd7b..0362f60ebe0 100644 --- a/tests/unit/test_repo_agent_skills.py +++ b/tests/unit/test_repo_agent_skills.py @@ -14,10 +14,6 @@ def _frontmatter(skill_directory: Path) -> object: return yaml.safe_load(frontmatter) -def test_litellm_setup_skill_sits_where_skill_installers_look() -> None: - assert _SKILLS_DIR / "litellm-setup" in _SKILL_DIRECTORIES - - @pytest.mark.parametrize("skill_directory", _SKILL_DIRECTORIES, ids=lambda path: path.name) def test_skill_frontmatter_has_string_name_matching_its_directory_and_a_description(skill_directory: Path) -> None: frontmatter: Final = _frontmatter(skill_directory) From fed7afa8c63a7d2939e17903b1a47320cd2bd9b5 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 20 Oct 2026 20:01:05 -0700 Subject: [PATCH 4/9] fix(skills): keep shell profiles untouched when litellm-setup installs uv The uv installer edits shell profiles unless UV_NO_MODIFY_PATH is set, so the skill now sets it and puts ~/.local/bin on PATH for later commands. It also says litellm --setup runs in a separate terminal and how to retire the shared master-key UI login --- skills/litellm-setup/SKILL.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/skills/litellm-setup/SKILL.md b/skills/litellm-setup/SKILL.md index 2d4f3e882f3..f2e51c6b1c5 100644 --- a/skills/litellm-setup/SKILL.md +++ b/skills/litellm-setup/SKILL.md @@ -24,7 +24,7 @@ Ask everything in one message, with defaults the user can accept by saying "go": Then follow the steps for their answers. Plain routing skips steps 3 and 6. The quickstart is steps 2, 3, 8, and 9. uv with their own Postgres skips step 3 -In the same message, say that someone who would rather answer prompts can run `litellm --setup` in their own terminal after step 4. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy. When the user is done, skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it +In the same message, say that someone who would rather answer prompts can run `litellm --setup` in a separate terminal window after step 4, not through Claude Code. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy. When the user is done, skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it ## 2. Check prerequisites @@ -32,7 +32,7 @@ In the same message, say that someone who would rather answer prompts can run `l uv --version; docker info --format '{{.ServerVersion}}'; docker compose version; curl --version | head -1; openssl version; command -v python3 lsof ``` -Every path needs curl, openssl, python3, and lsof, so ask the user to install any that is missing. The uv path needs uv, which downloads a supported Python (LiteLLM supports 3.10 to 3.14) when the `--python` range in step 4 rules out the one on the machine. If uv is missing, offer to install it the way https://docs.astral.sh/uv/getting-started/installation/ shows for their OS, and run the installer only after the user says yes. The Docker path needs a running Docker daemon, and the quickstart also needs Docker Compose v2. If `docker info` says `permission denied`, the user's account cannot use Docker without sudo. Offer the uv path, or ask them to add themselves to the `docker` group the way https://docs.docker.com/engine/install/linux-postinstall/ shows and start a new login session before you check again +Every path needs curl, openssl, python3, and lsof, so ask the user to install any that is missing. The uv path needs uv, which downloads a supported Python (LiteLLM supports 3.10 to 3.14) when the `--python` range in step 4 rules out the one on the machine. If uv is missing, offer to install it, and only after the user says yes run `curl -LsSf https://astral.sh/uv/install.sh | env UV_NO_MODIFY_PATH=1 sh`, which leaves their shell profile alone (https://docs.astral.sh/uv/getting-started/installation/). It puts uv in `~/.local/bin`, so start every later command that runs `uv` or `litellm` with `export PATH="$HOME/.local/bin:$PATH";`. The Docker path needs a running Docker daemon, and the quickstart also needs Docker Compose v2. If `docker info` says `permission denied`, the user's account cannot use Docker without sudo. Offer the uv path, or ask them to add themselves to the `docker` group the way https://docs.docker.com/engine/install/linux-postinstall/ shows and start a new login session before you check again Check the port is free with `lsof -nP -iTCP:4000 -sTCP:LISTEN`, which prints nothing when it is. If it is taken, pick another port and use it everywhere below, because a proxy asked for a busy 4000 silently moves to a random port @@ -170,6 +170,7 @@ Tell the user, without printing any secret: - The master key is `LITELLM_MASTER_KEY` in the folder's `.env` - To stop it, `kill "$(cat litellm.pid)"` (uv), `docker stop litellm` (Docker), or `/litellm-gateway stop` (quickstart). To start it again, rerun the step 7 command, `docker start litellm`, or `/litellm-gateway start`. `update` and `uninstall` update or remove a quickstart install. Give the quickstart commands with the folder's full path, since a bare `litellm-gateway` is on PATH only when `~/.local/bin` is, and can belong to an earlier install - With Postgres, the Admin UI is at `http://127.0.0.1:4000/ui` (user `admin`, password the master key), and `POST /key/generate` with the master key creates virtual keys with their own budgets and model lists. Without Postgres, both need step 3 or 6 first +- The Admin UI warns that the master-key login is a shared secret, which is fine while only this machine reaches the proxy. Before anyone else can reach it, invite an admin with their own password on the Internal Users page, then set `general_settings.disable_env_credential_login: true` in the proxy config - To manage the running proxy from a coding agent (keys, teams, users, models, MCP servers), install `npx skills add BerriAI/litellm-skills -g` ## Troubleshooting From ef3db906f062d65401caf75ed3c8c34f264dcd23 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 20 Oct 2026 20:13:10 -0700 Subject: [PATCH 5/9] fix(skills): offer litellm --setup only on the uv path The Docker paths install no host litellm CLI, and the old wording also passed the skill's own step number on to the user --- skills/litellm-setup/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/skills/litellm-setup/SKILL.md b/skills/litellm-setup/SKILL.md index f2e51c6b1c5..02f18b1ef31 100644 --- a/skills/litellm-setup/SKILL.md +++ b/skills/litellm-setup/SKILL.md @@ -24,7 +24,7 @@ Ask everything in one message, with defaults the user can accept by saying "go": Then follow the steps for their answers. Plain routing skips steps 3 and 6. The quickstart is steps 2, 3, 8, and 9. uv with their own Postgres skips step 3 -In the same message, say that someone who would rather answer prompts can run `litellm --setup` in a separate terminal window after step 4, not through Claude Code. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy. When the user is done, skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it +In the same message, say that someone who picks uv and would rather answer prompts can run `litellm --setup` in a separate terminal window once LiteLLM is installed, not through Claude Code. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy. When the user is done, skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it ## 2. Check prerequisites From f4df9bebadf07f5e1fd1ad98239a40a1e87733a0 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 20 Oct 2026 20:43:09 -0700 Subject: [PATCH 6/9] fix(skills): ask before using an env provider key, print model names config.yaml accepts, probe the port with curl --- skills/litellm-setup/SKILL.md | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/skills/litellm-setup/SKILL.md b/skills/litellm-setup/SKILL.md index 02f18b1ef31..ae0dc379e5a 100644 --- a/skills/litellm-setup/SKILL.md +++ b/skills/litellm-setup/SKILL.md @@ -17,7 +17,7 @@ unset DATABASE_URL DIRECT_URL DATABASE_HOST; while IFS= read -r line || [ -n "$l Ask everything in one message, with defaults the user can accept by saying "go": -- Providers and models. Default is OpenAI with its newest flagship model +- Providers and models. Default is OpenAI with its newest flagship model, using the `OPENAI_API_KEY` in your environment when `[ -n "$OPENAI_API_KEY" ] && echo set` says it is there. Run that check before asking, and name the variable but never its value, since a key in your environment can belong to another account - Whether they want virtual keys, teams, spend tracking, or the Admin UI. Those need Postgres, plain routing does not. Default is no - uv (default) or Docker. With Postgres, Docker runs the repo's quickstart, which brings its own database, and uv needs a Postgres server they already have, with a new, empty database for LiteLLM - Folder and port. Defaults are `./litellm-proxy` (`./litellm-gateway` for the quickstart) and 4000 @@ -29,12 +29,12 @@ In the same message, say that someone who picks uv and would rather answer promp ## 2. Check prerequisites ```bash -uv --version; docker info --format '{{.ServerVersion}}'; docker compose version; curl --version | head -1; openssl version; command -v python3 lsof +uv --version; docker info --format '{{.ServerVersion}}'; docker compose version; curl --version | head -1; openssl version; command -v python3 ``` -Every path needs curl, openssl, python3, and lsof, so ask the user to install any that is missing. The uv path needs uv, which downloads a supported Python (LiteLLM supports 3.10 to 3.14) when the `--python` range in step 4 rules out the one on the machine. If uv is missing, offer to install it, and only after the user says yes run `curl -LsSf https://astral.sh/uv/install.sh | env UV_NO_MODIFY_PATH=1 sh`, which leaves their shell profile alone (https://docs.astral.sh/uv/getting-started/installation/). It puts uv in `~/.local/bin`, so start every later command that runs `uv` or `litellm` with `export PATH="$HOME/.local/bin:$PATH";`. The Docker path needs a running Docker daemon, and the quickstart also needs Docker Compose v2. If `docker info` says `permission denied`, the user's account cannot use Docker without sudo. Offer the uv path, or ask them to add themselves to the `docker` group the way https://docs.docker.com/engine/install/linux-postinstall/ shows and start a new login session before you check again +Every path needs curl, openssl, and python3, so ask the user to install any that is missing. The uv path needs uv, which downloads a supported Python (LiteLLM supports 3.10 to 3.14) when the `--python` range in step 4 rules out the one on the machine. If uv is missing, offer to install it, and only after the user says yes run `curl -LsSf https://astral.sh/uv/install.sh | env UV_NO_MODIFY_PATH=1 sh`, which leaves their shell profile alone (https://docs.astral.sh/uv/getting-started/installation/). It puts uv in `~/.local/bin`, so start every later command that runs `uv` or `litellm` with `export PATH="$HOME/.local/bin:$PATH";`. The Docker path needs a running Docker daemon, and the quickstart also needs Docker Compose v2. If `docker info` says `permission denied`, the user's account cannot use Docker without sudo. Offer the uv path, or ask them to add themselves to the `docker` group the way https://docs.docker.com/engine/install/linux-postinstall/ shows and start a new login session before you check again -Check the port is free with `lsof -nP -iTCP:4000 -sTCP:LISTEN`, which prints nothing when it is. If it is taken, pick another port and use it everywhere below, because a proxy asked for a busy 4000 silently moves to a random port +Check the port is free with `curl -s -o /dev/null --max-time 2 http://127.0.0.1:4000; echo $?`, which prints `7` (could not connect) only when nothing listens there. Don't use `lsof` for this, since it misses ports that root or Docker hold. If it is taken, pick another port and use it everywhere below, because a proxy asked for a busy 4000 silently moves to a random port ## 3. Postgres with Docker: run the quickstart @@ -46,7 +46,7 @@ LITELLM_DIR="$PWD/litellm-gateway" LITELLM_PORT=4000 sh quickstart.sh --yes cd litellm-gateway ``` -Models live in the database here, so add each one through the API instead of a `config.yaml`, naming it the way step 5 says. Pipe the request body in, so the provider key never shows up in output or on a command line. If the key is not in your environment, ask the user to add the line `OPENAI_API_KEY=` to this `.env` themselves, and run the loader line first: +Models live in the database here, so add each one through the API instead of a `config.yaml`, naming it the way step 5 says. Pipe the request body in, so the provider key never shows up in output or on a command line. Unless the user agreed to the key in your environment, as step 5 says, ask them to add the line `OPENAI_API_KEY=` to this `.env` themselves. Run the loader line first: ```bash python3 -c 'import json, os, sys; json.dump({"model_name": "gpt-6.1-sol", "litellm_params": {"model": "openai/gpt-6.1-sol", "api_key": os.environ["OPENAI_API_KEY"]}}, sys.stdout)' \ @@ -83,16 +83,22 @@ touch .env && chmod 600 .env; grep -qx '.env' .gitignore 2>/dev/null || echo '.e grep -q '^LITELLM_MASTER_KEY=.' .env || printf 'LITELLM_MASTER_KEY=sk-%s\n' "$(openssl rand -hex 32)" >> .env ``` -For each provider key, if it is already in your environment (`[ -n "$OPENAI_API_KEY" ] && echo set`), copy it in without printing it, as in `printf 'OPENAI_API_KEY=%s\n' "$OPENAI_API_KEY" >> .env`. Otherwise ask the user to add the line `OPENAI_API_KEY=` to `.env` themselves and tell you when it is there. If they paste a key into the chat anyway, write it to `.env` and never repeat it. Keep one `NAME=value` per line with no quotes, so `docker --env-file` and the loader line read the same value +Copy a provider key that is already in your environment only after the user agreed to that variable, in their step 1 answer or when you ask now, and copy it without printing it, as in `printf 'OPENAI_API_KEY=%s\n' "$OPENAI_API_KEY" >> .env`. Otherwise ask the user to add the line `OPENAI_API_KEY=` to `.env` themselves and tell you when it is there. If they paste a key into the chat anyway, write it to `.env` and never repeat it. Keep one `NAME=value` per line with no quotes, so `docker --env-file` and the loader line read the same value -Confirm each model's current LiteLLM name in the cost map instead of trusting memory, since model names go stale fast. This lists the chat models of the provider named last (`openai`, `anthropic`, `gemini`, `azure`, `bedrock`, `vertex_ai`, and so on). It matches by prefix, so `bedrock` also lists the `bedrock_converse` rows, which hold the newer Bedrock models, and `vertex_ai` lists every Vertex family. For Ollama, use a model from `ollama list` instead: +Confirm each model's current LiteLLM name in the cost map instead of trusting memory, since model names go stale fast. This prints the chat models of the provider named last (`openai`, `anthropic`, `gemini`, `azure`, `bedrock`, `vertex_ai`, and so on) the way `config.yaml` takes them, with the provider in front. It matches by prefix, so `bedrock` also lists the newer Bedrock models and `vertex_ai` lists every Vertex family. For Ollama, use a model from `ollama list` instead: ```bash -curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json \ - | python3 -c 'import json,sys; [print(v["litellm_provider"], k) for k,v in json.load(sys.stdin).items() if isinstance(v,dict) and str(v.get("litellm_provider","")).startswith(sys.argv[1]) and v.get("mode")=="chat"]' openai +curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | python3 -c ' +import json, sys +for name, row in json.load(sys.stdin).items(): + provider = str(row.get("litellm_provider", "")) if isinstance(row, dict) else "" + if provider.startswith(sys.argv[1]) and row.get("mode") == "chat": + prefix = "bedrock" if provider == "bedrock_converse" else "vertex_ai" if provider.startswith("vertex_ai") else provider + print(name if name.startswith(prefix + "/") else prefix + "/" + name) +' openai ``` -In `config.yaml`, `model` is the name with its provider in front, as in `openai/gpt-6.1-sol` or `bedrock/us.anthropic.`. Names that already start with a provider and a slash (`gemini/...`, `vertex_ai/...`) are used as they are. `model_name` is what clients will send. This example uses the newest GPT model in the cost map when this skill was written, so replace it with what the user picked: +In `config.yaml`, `model` is a name from that list, as in `openai/gpt-6.1-sol` or `bedrock/us.anthropic.`, and `model_name` is what clients will send. This example uses the newest GPT model in the cost map when this skill was written, so replace it with what the user picked: ```yaml model_list: @@ -180,6 +186,7 @@ Tell the user, without printing any secret: - `The Client hasn't been generated yet` or `Unable to find Prisma binaries` means the Prisma generate command in step 6 has not run - `Incorrect API key provided` or another provider `AuthenticationError` means a provider key is wrong. Fix it in `.env` and restart, since the proxy reads the environment only at startup. On uv, stop it and rerun step 7. On Docker, `docker rm -f litellm` and rerun step 7, since `docker restart` keeps the environment the container was created with. On the quickstart the key is saved with the model, so delete the model and add it again: `GET /model/info` lists each model's `model_info.id`, and `POST /model/delete` with `{"id": ""}` deletes it - `Cannot automatically baseline this database` means `DATABASE_URL` points at a database that already holds tables LiteLLM did not create. Point it at a new, empty database +- `LLM Provider NOT provided` means a `model` in `config.yaml` (or step 3) lacks its provider in front. Use the name as the step 5 lookup prints it - `Invalid model name passed in model=...` means the request's `model` is not a `model_name` in `config.yaml` or step 3. `GET /v1/models` lists the ones the key can use - `No connected db.` on a proxy without Postgres means the bearer token is not the master key - `Conflict. The container name "/litellm" is already in use` means an earlier run left a container. `docker start litellm` reuses it, or with the user's OK `docker rm -f litellm` and rerun step 7 From e5fba49d461119d8a50db561357f8b14b3817e52 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 20 Oct 2026 20:59:50 -0700 Subject: [PATCH 7/9] fix(skills): state the env key default in the question, chmod the wizard config, keep uv's bin dir on PATH --- skills/litellm-setup/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/skills/litellm-setup/SKILL.md b/skills/litellm-setup/SKILL.md index ae0dc379e5a..3c00eefefe5 100644 --- a/skills/litellm-setup/SKILL.md +++ b/skills/litellm-setup/SKILL.md @@ -17,14 +17,14 @@ unset DATABASE_URL DIRECT_URL DATABASE_HOST; while IFS= read -r line || [ -n "$l Ask everything in one message, with defaults the user can accept by saying "go": -- Providers and models. Default is OpenAI with its newest flagship model, using the `OPENAI_API_KEY` in your environment when `[ -n "$OPENAI_API_KEY" ] && echo set` says it is there. Run that check before asking, and name the variable but never its value, since a key in your environment can belong to another account +- Providers and models. Default is OpenAI with its newest flagship model, using the `OPENAI_API_KEY` in your environment when `[ -n "$OPENAI_API_KEY" ] && echo set` says it is there. Run that check before asking, and say in the question that the default uses that variable, naming it but never its value, since a key in your environment can belong to another account - Whether they want virtual keys, teams, spend tracking, or the Admin UI. Those need Postgres, plain routing does not. Default is no - uv (default) or Docker. With Postgres, Docker runs the repo's quickstart, which brings its own database, and uv needs a Postgres server they already have, with a new, empty database for LiteLLM - Folder and port. Defaults are `./litellm-proxy` (`./litellm-gateway` for the quickstart) and 4000 Then follow the steps for their answers. Plain routing skips steps 3 and 6. The quickstart is steps 2, 3, 8, and 9. uv with their own Postgres skips step 3 -In the same message, say that someone who picks uv and would rather answer prompts can run `litellm --setup` in a separate terminal window once LiteLLM is installed, not through Claude Code. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy. When the user is done, skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it +In the same message, say that someone who picks uv and would rather answer prompts can run `litellm --setup` in a separate terminal window once LiteLLM is installed, not through Claude Code. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy, which then listens on every network interface and stops with Ctrl+C in that window. When the user is done, run `chmod 600 litellm_config.yaml`, since the wizard leaves it readable by every account on the machine, then skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it ## 2. Check prerequisites @@ -65,7 +65,7 @@ uv tool install --python '>=3.10,<3.15' 'litellm[proxy]' litellm --version ``` -With their own Postgres, install `'litellm[proxy,extra_proxy]'` instead, with the same `--python` range, which adds the Prisma client the proxy uses for the database. Running `uv tool install` again with the new extras replaces the old install. If `litellm` is not found, call `"$(uv tool dir --bin)/litellm"` in this shell. `uv tool update-shell` puts it on the PATH of new shells by editing the user's shell profile, so run it only after the user says yes +With their own Postgres, install `'litellm[proxy,extra_proxy]'` instead, with the same `--python` range, which adds the Prisma client the proxy uses for the database. Running `uv tool install` again with the new extras replaces the old install. If `litellm` is not found, start every later command that runs it with `export PATH="$(uv tool dir --bin):$PATH";`. `uv tool update-shell` puts it on the PATH of new shells by editing the user's shell profile, so run it only after the user says yes Docker: @@ -174,7 +174,7 @@ Tell the user, without printing any secret: - The base URL is `http://127.0.0.1:4000`. Any OpenAI-compatible client works with it as the base URL and the master key or a virtual key as the API key - The master key is `LITELLM_MASTER_KEY` in the folder's `.env` -- To stop it, `kill "$(cat litellm.pid)"` (uv), `docker stop litellm` (Docker), or `/litellm-gateway stop` (quickstart). To start it again, rerun the step 7 command, `docker start litellm`, or `/litellm-gateway start`. `update` and `uninstall` update or remove a quickstart install. Give the quickstart commands with the folder's full path, since a bare `litellm-gateway` is on PATH only when `~/.local/bin` is, and can belong to an earlier install +- To stop it, `kill "$(cat litellm.pid)"` (uv), `docker stop litellm` (Docker), `/litellm-gateway stop` (quickstart), or Ctrl+C in its window (`litellm --setup`). To start it again, rerun the step 7 command, `docker start litellm`, or `/litellm-gateway start`. `update` and `uninstall` update or remove a quickstart install. Give the quickstart commands with the folder's full path, since a bare `litellm-gateway` is on PATH only when `~/.local/bin` is, and can belong to an earlier install - With Postgres, the Admin UI is at `http://127.0.0.1:4000/ui` (user `admin`, password the master key), and `POST /key/generate` with the master key creates virtual keys with their own budgets and model lists. Without Postgres, both need step 3 or 6 first - The Admin UI warns that the master-key login is a shared secret, which is fine while only this machine reaches the proxy. Before anyone else can reach it, invite an admin with their own password on the Internal Users page, then set `general_settings.disable_env_credential_login: true` in the proxy config - To manage the running proxy from a coding agent (keys, teams, users, models, MCP servers), install `npx skills add BerriAI/litellm-skills -g` From c380350b00d76b092530f6ec2193adea1a547f6f Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 20 Oct 2026 21:38:53 -0700 Subject: [PATCH 8/9] fix(skills): keep litellm-setup on its own .env and install when the shell has others --- skills/litellm-setup/SKILL.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/skills/litellm-setup/SKILL.md b/skills/litellm-setup/SKILL.md index 3c00eefefe5..0bdc1921db0 100644 --- a/skills/litellm-setup/SKILL.md +++ b/skills/litellm-setup/SKILL.md @@ -7,10 +7,10 @@ description: Set up a self-hosted LiteLLM proxy, the OpenAI-compatible LLM gatew Take the user from nothing to a running LiteLLM proxy that has answered one chat completion. Run every command yourself unless a step says the user does it -Never print a secret. Provider keys, the master key, the salt key, the database URL, and virtual keys live only in `.env`, and `config.yaml` reads them through `os.environ/NAME`. Check that a value is there with `grep -c '^NAME=.' .env`, never with `cat .env` or `echo "$NAME"`. Load `.env` into a shell with this line, which exports each line as it is, so a `&` or `$` in a value stays literal, and first drops any database settings the shell already had, so only `.env` picks the database. The steps that need it start with it: +Never print a secret. Provider keys, the master key, the salt key, the database URL, and virtual keys live only in `.env`, and `config.yaml` reads them through `os.environ/NAME`. Check that a value is there with `grep -c '^NAME=.' .env`, never with `cat .env` or `echo "$NAME"`. Load `.env` into a shell with this line, which exports each line as it is, so a `&` or `$` in a value stays literal, and first drops any database settings the shell already had and sets `LITELLM_MODE=PRODUCTION`, which stops `litellm` from also loading a `.env` it finds above its install folder, such as `~/.env`, so only this `.env` picks the database. The steps that need it start with it: ```bash -unset DATABASE_URL DIRECT_URL DATABASE_HOST; while IFS= read -r line || [ -n "$line" ]; do case "$line" in [A-Za-z_]*=*) export "$line" ;; esac; done < .env +unset DATABASE_URL DIRECT_URL DATABASE_HOST; export LITELLM_MODE=PRODUCTION; while IFS= read -r line || [ -n "$line" ]; do case "$line" in [A-Za-z_]*=*) export "$line" ;; esac; done < .env ``` ## 1. Ask once @@ -19,12 +19,12 @@ Ask everything in one message, with defaults the user can accept by saying "go": - Providers and models. Default is OpenAI with its newest flagship model, using the `OPENAI_API_KEY` in your environment when `[ -n "$OPENAI_API_KEY" ] && echo set` says it is there. Run that check before asking, and say in the question that the default uses that variable, naming it but never its value, since a key in your environment can belong to another account - Whether they want virtual keys, teams, spend tracking, or the Admin UI. Those need Postgres, plain routing does not. Default is no -- uv (default) or Docker. With Postgres, Docker runs the repo's quickstart, which brings its own database, and uv needs a Postgres server they already have, with a new, empty database for LiteLLM +- uv (default) or Docker. With Postgres, Docker runs the repo's quickstart, which brings its own database, and uv needs a Postgres server they already have, with a new, empty database for LiteLLM, so someone with their own Postgres server picks uv - Folder and port. Defaults are `./litellm-proxy` (`./litellm-gateway` for the quickstart) and 4000 Then follow the steps for their answers. Plain routing skips steps 3 and 6. The quickstart is steps 2, 3, 8, and 9. uv with their own Postgres skips step 3 -In the same message, say that someone who picks uv and would rather answer prompts can run `litellm --setup` in a separate terminal window once LiteLLM is installed, not through Claude Code. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. It writes `litellm_config.yaml` in the current folder with the keys inline (keep it out of git) and offers to start the proxy, which then listens on every network interface and stops with Ctrl+C in that window. When the user is done, run `chmod 600 litellm_config.yaml`, since the wizard leaves it readable by every account on the machine, then skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it +In the same message, say that someone who picks uv and would rather answer prompts can run `litellm --setup` in a separate terminal window once LiteLLM is installed, not through Claude Code. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. Give them the command as `LITELLM_MODE=PRODUCTION /litellm --setup`, with `` replaced by the folder `uv tool dir --bin` prints, to run from the folder they want the config in. It writes `litellm_config.yaml` there with the keys inline and offers to start the proxy, which then listens on every network interface and stops with Ctrl+C in that window. When the user is done, run `chmod 600 litellm_config.yaml; grep -qx 'litellm_config.yaml' .gitignore 2>/dev/null || echo 'litellm_config.yaml' >> .gitignore` in that folder, since the wizard leaves the file readable by every account on the machine and nothing keeps it out of git, then skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it ## 2. Check prerequisites @@ -32,7 +32,7 @@ In the same message, say that someone who picks uv and would rather answer promp uv --version; docker info --format '{{.ServerVersion}}'; docker compose version; curl --version | head -1; openssl version; command -v python3 ``` -Every path needs curl, openssl, and python3, so ask the user to install any that is missing. The uv path needs uv, which downloads a supported Python (LiteLLM supports 3.10 to 3.14) when the `--python` range in step 4 rules out the one on the machine. If uv is missing, offer to install it, and only after the user says yes run `curl -LsSf https://astral.sh/uv/install.sh | env UV_NO_MODIFY_PATH=1 sh`, which leaves their shell profile alone (https://docs.astral.sh/uv/getting-started/installation/). It puts uv in `~/.local/bin`, so start every later command that runs `uv` or `litellm` with `export PATH="$HOME/.local/bin:$PATH";`. The Docker path needs a running Docker daemon, and the quickstart also needs Docker Compose v2. If `docker info` says `permission denied`, the user's account cannot use Docker without sudo. Offer the uv path, or ask them to add themselves to the `docker` group the way https://docs.docker.com/engine/install/linux-postinstall/ shows and start a new login session before you check again +On native Windows, ask the user to run Claude Code inside WSL, since every command here is POSIX shell. Every path needs curl, openssl, and python3, so ask the user to install any that is missing. The uv path needs uv, which downloads a supported Python (LiteLLM supports 3.10 to 3.14) when the `--python` range in step 4 rules out the one on the machine. If uv is missing, offer to install it, and only after the user says yes run `curl -LsSf https://astral.sh/uv/install.sh | env UV_NO_MODIFY_PATH=1 sh`, which leaves their shell profile alone (https://docs.astral.sh/uv/getting-started/installation/). It puts uv in `~/.local/bin`, so start every later command that runs `uv` or `litellm` with `export PATH="$HOME/.local/bin:$PATH";`. The Docker path needs a running Docker daemon, and the quickstart also needs Docker Compose v2. If `docker info` says `permission denied`, the user's account cannot use Docker without sudo. Offer the uv path, or ask them to add themselves to the `docker` group the way https://docs.docker.com/engine/install/linux-postinstall/ shows and start a new login session before you check again Check the port is free with `curl -s -o /dev/null --max-time 2 http://127.0.0.1:4000; echo $?`, which prints `7` (could not connect) only when nothing listens there. Don't use `lsof` for this, since it misses ports that root or Docker hold. If it is taken, pick another port and use it everywhere below, because a proxy asked for a busy 4000 silently moves to a random port @@ -62,10 +62,10 @@ uv: ```bash uv tool install --python '>=3.10,<3.15' 'litellm[proxy]' -litellm --version +export PATH="$(uv tool dir --bin):$PATH"; litellm --version ``` -With their own Postgres, install `'litellm[proxy,extra_proxy]'` instead, with the same `--python` range, which adds the Prisma client the proxy uses for the database. Running `uv tool install` again with the new extras replaces the old install. If `litellm` is not found, start every later command that runs it with `export PATH="$(uv tool dir --bin):$PATH";`. `uv tool update-shell` puts it on the PATH of new shells by editing the user's shell profile, so run it only after the user says yes +With their own Postgres, install `'litellm[proxy,extra_proxy]'` instead, with the same `--python` range, which adds the Prisma client the proxy uses for the database. Running `uv tool install` again with the new extras replaces the old install. Start every later command that runs `litellm` with `export PATH="$(uv tool dir --bin):$PATH";`, so this install runs even when the shell has no `litellm` or an older one from a venv or `pip install litellm`. `uv tool update-shell` puts it on the PATH of new shells by editing the user's shell profile, so run it only after the user says yes Docker: @@ -98,7 +98,7 @@ for name, row in json.load(sys.stdin).items(): ' openai ``` -In `config.yaml`, `model` is a name from that list, as in `openai/gpt-6.1-sol` or `bedrock/us.anthropic.`, and `model_name` is what clients will send. This example uses the newest GPT model in the cost map when this skill was written, so replace it with what the user picked: +In `config.yaml`, `model` is a name from that list, as in `openai/gpt-6.1-sol` or `bedrock/us.anthropic.`, and `model_name` is what clients will send. If the folder already has a `config.yaml`, show the user its `model_name` lines and ask before replacing it. This example uses the newest GPT model in the cost map when this skill was written, so replace it with what the user picked: ```yaml model_list: @@ -136,6 +136,7 @@ Bind to `127.0.0.1` unless the user wants other machines to reach it uv, in one shell command that starts with the loader line, since the installed `litellm` does not read `.env` by itself: ```bash +export PATH="$(uv tool dir --bin):$PATH" nohup litellm --config config.yaml --host 127.0.0.1 --port 4000 > litellm.log 2>&1 & echo $! > litellm.pid ``` @@ -182,7 +183,7 @@ Tell the user, without printing any secret: ## Troubleshooting - `refused to start: no master key is set` means the loader line did not run in the shell command that started `litellm`, or `--env-file .env` is missing from `docker run` -- `a database URL is set but the prisma CLI is neither on PATH nor importable` means the install lacks `extra_proxy`, so rerun step 4 with it +- `a database URL is set but the prisma CLI is neither on PATH nor importable` means the install lacks `extra_proxy`, so rerun step 4 with it, or, on plain routing, that the command that started `litellm` did not start with the loader line - `The Client hasn't been generated yet` or `Unable to find Prisma binaries` means the Prisma generate command in step 6 has not run - `Incorrect API key provided` or another provider `AuthenticationError` means a provider key is wrong. Fix it in `.env` and restart, since the proxy reads the environment only at startup. On uv, stop it and rerun step 7. On Docker, `docker rm -f litellm` and rerun step 7, since `docker restart` keeps the environment the container was created with. On the quickstart the key is saved with the model, so delete the model and add it again: `GET /model/info` lists each model's `model_info.id`, and `POST /model/delete` with `{"id": ""}` deletes it - `Cannot automatically baseline this database` means `DATABASE_URL` points at a database that already holds tables LiteLLM did not create. Point it at a new, empty database From af7e6bb7f12400e78c2658e72c07e9040a78f3c5 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 20 Oct 2026 21:59:58 -0700 Subject: [PATCH 9/9] fix(skills): read the setup wizard's config without printing keys and hand off the exact restart command --- skills/litellm-setup/SKILL.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/skills/litellm-setup/SKILL.md b/skills/litellm-setup/SKILL.md index 0bdc1921db0..c6d06e86d4d 100644 --- a/skills/litellm-setup/SKILL.md +++ b/skills/litellm-setup/SKILL.md @@ -24,7 +24,18 @@ Ask everything in one message, with defaults the user can accept by saying "go": Then follow the steps for their answers. Plain routing skips steps 3 and 6. The quickstart is steps 2, 3, 8, and 9. uv with their own Postgres skips step 3 -In the same message, say that someone who picks uv and would rather answer prompts can run `litellm --setup` in a separate terminal window once LiteLLM is installed, not through Claude Code. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. Give them the command as `LITELLM_MODE=PRODUCTION /litellm --setup`, with `` replaced by the folder `uv tool dir --bin` prints, to run from the folder they want the config in. It writes `litellm_config.yaml` there with the keys inline and offers to start the proxy, which then listens on every network interface and stops with Ctrl+C in that window. When the user is done, run `chmod 600 litellm_config.yaml; grep -qx 'litellm_config.yaml' .gitignore 2>/dev/null || echo 'litellm_config.yaml' >> .gitignore` in that folder, since the wizard leaves the file readable by every account on the machine and nothing keeps it out of git, then skip to step 8 and run it against their proxy, reading the master key from that file into a shell variable without printing it +In the same message, say that someone who picks uv and would rather answer prompts can run `litellm --setup` in a separate terminal window once LiteLLM is installed, not through Claude Code. Never run it yourself, since it needs a real terminal and without one prints "Setup cancelled." and writes nothing. Give them the command as `LITELLM_MODE=PRODUCTION /litellm --setup`, with `` replaced by the folder `uv tool dir --bin` prints, to run from the folder they want the config in. It writes `litellm_config.yaml` there with the provider keys and master key in plain text and offers to start the proxy, which then listens on every network interface and stops with Ctrl+C in that window. When the user is done, run this in that folder. It makes the file readable only by them, keeps it out of git (the wizard does neither), and prints only the model names. Never print that file or run `grep`, `sed`, or `cat` on it: + +```bash +chmod 600 litellm_config.yaml; grep -qx 'litellm_config.yaml' .gitignore 2>/dev/null || echo 'litellm_config.yaml' >> .gitignore +"$(uv tool dir)/litellm/bin/python" -I -c 'import yaml; print(*(m["model_name"] for m in yaml.safe_load(open("litellm_config.yaml"))["model_list"]), sep="\n")' +``` + +Then skip to step 8 and run it against their proxy, starting that shell command with this line instead of the loader line, which reads the master key without printing it: + +```bash +export LITELLM_MASTER_KEY="$("$(uv tool dir)/litellm/bin/python" -I -c 'import yaml; print(yaml.safe_load(open("litellm_config.yaml"))["general_settings"]["master_key"])')" +``` ## 2. Check prerequisites @@ -175,7 +186,7 @@ Tell the user, without printing any secret: - The base URL is `http://127.0.0.1:4000`. Any OpenAI-compatible client works with it as the base URL and the master key or a virtual key as the API key - The master key is `LITELLM_MASTER_KEY` in the folder's `.env` -- To stop it, `kill "$(cat litellm.pid)"` (uv), `docker stop litellm` (Docker), `/litellm-gateway stop` (quickstart), or Ctrl+C in its window (`litellm --setup`). To start it again, rerun the step 7 command, `docker start litellm`, or `/litellm-gateway start`. `update` and `uninstall` update or remove a quickstart install. Give the quickstart commands with the folder's full path, since a bare `litellm-gateway` is on PATH only when `~/.local/bin` is, and can belong to an earlier install +- To stop it, `kill "$(cat litellm.pid)"` (uv), `docker stop litellm` (Docker), `/litellm-gateway stop` (quickstart), or Ctrl+C in its window (`litellm --setup`). To start it again, `docker start litellm`, `/litellm-gateway start`, or on uv the whole step 7 command as you ran it, loader line and PATH export included, shown as a code block and never paraphrased, since a new shell has neither. `update` and `uninstall` update or remove a quickstart install. Give the quickstart commands with the folder's full path, since a bare `litellm-gateway` is on PATH only when `~/.local/bin` is, and can belong to an earlier install - With Postgres, the Admin UI is at `http://127.0.0.1:4000/ui` (user `admin`, password the master key), and `POST /key/generate` with the master key creates virtual keys with their own budgets and model lists. Without Postgres, both need step 3 or 6 first - The Admin UI warns that the master-key login is a shared secret, which is fine while only this machine reaches the proxy. Before anyone else can reach it, invite an admin with their own password on the Internal Users page, then set `general_settings.disable_env_credential_login: true` in the proxy config - To manage the running proxy from a coding agent (keys, teams, users, models, MCP servers), install `npx skills add BerriAI/litellm-skills -g`