litellm/tests
Mateo Wang ddb8a14ee9
feat(skills): add a litellm-setup agent skill (#47731)
* feat(skills): add a litellm-setup agent skill

* fix(skills): run the quickstart for Docker with Postgres in litellm-setup

Use scripts/quickstart.sh for the Docker plus Postgres path instead of a
hand-rolled Postgres container, keep uv with the user's own Postgres, load
.env as data instead of sourcing it, pin the uv Python range, chmod an
existing .env, list bedrock_converse and vertex_ai rows in the cost map
lookup, ask before editing the shell profile, and stop the health wait
when the proxy exits

* fix(skills): guide empty Postgres, docker sudo, and key fixes in litellm-setup

Ask for a new empty database on the uv path and DIRECT_URL behind a pooler,
add the baseline error to troubleshooting, and drop inherited database
settings in the loader line. Check python3 and lsof, explain Docker's
permission denied, give quickstart commands by the folder's path, and fix
provider keys by recreating the container or deleting the quickstart model.
Drop the structure-only test.

* fix(skills): keep shell profiles untouched when litellm-setup installs uv

The uv installer edits shell profiles unless UV_NO_MODIFY_PATH is set, so the skill now sets it and puts ~/.local/bin on PATH for later commands. It also says litellm --setup runs in a separate terminal and how to retire the shared master-key UI login

* fix(skills): offer litellm --setup only on the uv path

The Docker paths install no host litellm CLI, and the old wording also passed the skill's own step number on to the user

* fix(skills): ask before using an env provider key, print model names config.yaml accepts, probe the port with curl

* fix(skills): state the env key default in the question, chmod the wizard config, keep uv's bin dir on PATH

* fix(skills): keep litellm-setup on its own .env and install when the shell has others

* fix(skills): read the setup wizard's config without printing keys and hand off the exact restart command

* fix(skills): hide the database-access dev skill from end-user skill installs

* fix(skills): give the litellm --setup path a restart command that keeps its config

* docs(skills): point litellm --setup users to the master key in litellm_config.yaml

* fix(skills): hide every non-end-user skill, use Azure deployment names, keep the restart block out of the user's shell

* test(skills): find non-end-user skills in-process instead of through git

* docs(skills): price Azure deployments with base_model and put uv on PATH in the start block
2026-10-28 15:44:43 -07:00
..
_support fix(proxy)!: end a chat completions stream with an error event when a guardrail blocks mid-stream (#46953) 2026-10-17 18:00:12 -07:00
audio_tests feat(vertex-ai)!: support gemini 3.1 flash tts (#31915) 2026-10-21 04:18:42 +00:00
base_sdk_tests refactor(rust-bridge): require the native extension (#48852) 2026-10-28 13:16:56 -07:00
benchmarks refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) 2026-10-27 17:47:50 +00:00
code_coverage_tests fix(audio): return 400 for unsupported transcription providers (#47971) 2026-10-28 19:20:17 +00:00
documentation_tests fix(proxy): let configured pass-through endpoints outrank built-in provider prefix routes (#44133) 2026-10-10 12:26:17 -07:00
e2e fix(responses): forward top-level moderation param to OpenAI (#48708) 2026-10-28 10:33:11 -07:00
e2e_harness test(perf): ramp the load cases until the pod stops keeping up and floor sustained_rps (#47758) 2026-10-20 22:31:08 -07:00
fixtures test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
harness_e2e fix(harness): set PWD to the working directory of local sandbox commands (#47530) 2026-10-20 20:06:11 +00:00
integration test(routing): audit #46601 (#48910) 2026-10-28 15:42:51 -07:00
rust-python-harness feat(rust): encode LiteLLM response ids on native responses (#46572) 2026-10-18 20:52:34 -07:00
test_litellm_rust refactor(rust-bridge): require the native extension (#48852) 2026-10-28 13:16:56 -07:00
typing fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927) 2026-10-09 19:45:50 -07:00
unit feat(skills): add a litellm-setup agent skill (#47731) 2026-10-28 15:44:43 -07:00
windows_tests refactor(rust-bridge): require the native extension (#48852) 2026-10-28 13:16:56 -07:00
__init__.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_logging.conf
test_rust_python_harness.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
VCR_TESTING.md test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.