litellm/tests
Mateo Wang 6cdb9959df
feat(router): opt in to non-chat deployment filtering (#46257)
* feat(proxy): reject chat routes on models declared non-chat via model_info.supported_endpoints

A model whose every deployment sets a non-chat model_info.mode and a supported_endpoints list with no chat endpoint now answers /v1/chat/completions, /v1/messages and /v1/responses with a 400 before any provider call. Models without both fields keep today's behavior.

* fix(proxy): let fallbacks answer and cover more chat routes in the non-chat model check

The check now steps aside when a request, key, team or router fallback would answer, skips such models in the rate-limit fallback loop, ignores mode and supported_endpoints copied from the cost map by old Admin UI saves, treats a blank mode as unset, and also covers /v1/completions, the Responses WebSocket, Gemini generateContent and /v1beta/interactions

* fix(proxy): only step aside for fallbacks that can answer chat, gate the Responses WebSocket after alias resolution, and compare declarations to the loaded catalog

* fix(proxy): gate the Responses WebSocket with the router after aliases resolve, and ignore client-set internal fallback flags

* fix(proxy): walk fallback groups breadth first and release reservations on a rejected WebSocket

Each fallback group is looked up once however often the request repeats it, and a WebSocket closed by the endpoint gate runs the failure hook so rate-limit reservations are released

* fix(proxy): walk fallback layers without recursion and gate the compact Responses route

The fallback walk now folds one layer at a time, so the recursion check passes, and /v1/responses/compact on a model declared non-chat gets the same 400 as the other chat-shaped routes

* fix(proxy): keep chat on rows whose cost map entry serves chat

A row that carries the cost map key of an entry that serves chat counts as declaring nothing, so a catalog edit that adds a chat path never turns old Admin UI copies into 400s

* fix(proxy): check chat sent to a deployment id or backend model name

* fix(proxy): check a model the read-through loads from the database

* fix(proxy): step aside for fallbacks the check cannot follow and wildcards that serve a backend name

A fallback target with no deployments of its own, a client routing stamp, and a wildcard that does not match the requested backend name no longer turn a working request into a 400

* fix(proxy): ignore a client model_group that is not a string when building rate-limit headers

A client metadata.model_group sent as an int or a list crashed the error handler with a 500 once the embedding-only check rejected the call before routing. The header helper now treats a non-string stamp as no model group. Drops the pattern clause in the backend-name lookup, since the group lookup already returns wildcard matches. Adds the third audit round's routing-stamp cells.

* fix(proxy): keep a non-string client model_group out of spend rows

* fix(proxy): validate standard fallbacks before the typed fallback-group lookup

* feat(router): opt in to non-chat deployment filtering

* fix(router): skip unresolved local rate-limit fallbacks

* fix(router): ignore caller-provided endpoint selector values

* fix(router): normalize declared chat endpoint prefixes

* test(router): cover preflight error-value behavior

* test(router): derive replay metadata defaults from model info

* fix(router): preserve fallback aliases and check synchronous text calls

* test(routing): rewrite the WebSocket model with a pre-call callback, since main no longer runs guardrails at socket admission

* test(routing): keep the endpoint gate's owned proxies on the worker's database

* fix(router): keep the endpoint gate on model_info patches that omit it and on same-group stream retries

A partial PATCH of model_info reset a stored enforce_supported_endpoints true back to false, because the
field defaulted to False and the update merges model_dump(exclude_none=True). The field now defaults to None
so an omitted flag keeps the stored value

The same-group retry of a stream dropped before content called the provider without the endpoint selection
context, so it could land on an enforced embedding-only deployment of the same group. It now runs inside
endpoint_selection like every other router attempt

* fix(router): probe rate-limit fallbacks through get_routable_deployments so the router surface does not grow

Main's router fence lets code outside the router use only listed Router members and
router_utils modules. The proxy's fallback probe now calls the listed
get_routable_deployments with the request's call type, and the gate's 400 moves to
litellm.exceptions as EndpointNotSupportedByModelError. It stays separate from main's
UnsupportedEndpointError, which the router retries on the group's other deployments,
while the gate refuses the whole group

* test(responses): cover the WebSocket error frame for a model that refuses the endpoint

* fix(proxy): let the rate-limit fallback probe try a fallback this worker has not loaded yet

The probe now skips a fallback only when the endpoint check removed every deployment it has, so a model added on another worker still reaches route_request and its database lookup

* fix(proxy): keep key router settings off the Responses WebSocket, read unrooted or single-string endpoint lists, and add status to the refusal frame

* fix(router): warn when enforce_supported_endpoints has no usable list and match chat endpoints regardless of case, trailing slash, or Gemini path

* test(router): type the load-warning test's caplog and router

* test(routing): match the endpoint gate's integration tests to base fallback behavior on unloaded fallbacks and the Responses WebSocket

---------

Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain>
2026-10-29 08:33:36 -07:00
..
_support fix(proxy)!: end a chat completions stream with an error event when a guardrail blocks mid-stream (#46953) 2026-10-17 18:00:12 -07:00
audio_tests feat(vertex-ai)!: support gemini 3.1 flash tts (#31915) 2026-10-21 04:18:42 +00:00
base_sdk_tests build(model-catalog): ship the cost map inside the native extension (#48907) 2026-10-29 07:55:02 -07:00
benchmarks test(json_fragment_accumulator): assert peak allocation instead of a drain timing ratio (#43370) 2026-10-28 22:04:47 -07:00
code_coverage_tests feat(responses): support Codex client tool_search on Anthropic and other bridged providers (#48972) 2026-10-29 00:29:40 -07:00
documentation_tests fix(proxy): let configured pass-through endpoints outrank built-in provider prefix routes (#44133) 2026-10-10 12:26:17 -07:00
e2e feat(router): opt in to non-chat deployment filtering (#46257) 2026-10-29 08:33:36 -07:00
e2e_harness test(perf): ramp the load cases until the pod stops keeping up and floor sustained_rps (#47758) 2026-10-20 22:31:08 -07:00
fixtures test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
harness_e2e fix(harness): set PWD to the working directory of local sandbox commands (#47530) 2026-10-20 20:06:11 +00:00
integration feat(router): opt in to non-chat deployment filtering (#46257) 2026-10-29 08:33:36 -07:00
rust-python-harness feat(rust): encode LiteLLM response ids on native responses (#46572) 2026-10-18 20:52:34 -07:00
test_litellm_rust fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893) 2026-10-29 03:36:20 -07:00
typing fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927) 2026-10-09 19:45:50 -07:00
unit feat(router): opt in to non-chat deployment filtering (#46257) 2026-10-29 08:33:36 -07:00
windows_tests refactor(rust-bridge): require the native extension (#48852) 2026-10-28 13:16:56 -07:00
__init__.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py
_process_helpers.py test: count a zombie grandchild as gone in the migrate deploy timeout test (#42570) 2026-09-22 14:59:26 -07:00
_vcr_conftest_common.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
capturing_transport.py test(vcr): guard leaked cassette patches and make injected-transport embedding tests immune (#42542) 2026-09-22 14:20:18 -07:00
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD test: finish the non-proxy half of tests/test_litellm (#43281) 2026-09-25 22:43:41 -07:00
test_logging.conf
test_rust_python_harness.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
VCR_TESTING.md test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
white_100x100.png test: stop CI tests from downloading tokenizer files and images (#43257) 2026-09-25 19:27:48 -07:00

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.