mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-12 08:42:18 +00:00
6cdb9959df
* feat(proxy): reject chat routes on models declared non-chat via model_info.supported_endpoints A model whose every deployment sets a non-chat model_info.mode and a supported_endpoints list with no chat endpoint now answers /v1/chat/completions, /v1/messages and /v1/responses with a 400 before any provider call. Models without both fields keep today's behavior. * fix(proxy): let fallbacks answer and cover more chat routes in the non-chat model check The check now steps aside when a request, key, team or router fallback would answer, skips such models in the rate-limit fallback loop, ignores mode and supported_endpoints copied from the cost map by old Admin UI saves, treats a blank mode as unset, and also covers /v1/completions, the Responses WebSocket, Gemini generateContent and /v1beta/interactions * fix(proxy): only step aside for fallbacks that can answer chat, gate the Responses WebSocket after alias resolution, and compare declarations to the loaded catalog * fix(proxy): gate the Responses WebSocket with the router after aliases resolve, and ignore client-set internal fallback flags * fix(proxy): walk fallback groups breadth first and release reservations on a rejected WebSocket Each fallback group is looked up once however often the request repeats it, and a WebSocket closed by the endpoint gate runs the failure hook so rate-limit reservations are released * fix(proxy): walk fallback layers without recursion and gate the compact Responses route The fallback walk now folds one layer at a time, so the recursion check passes, and /v1/responses/compact on a model declared non-chat gets the same 400 as the other chat-shaped routes * fix(proxy): keep chat on rows whose cost map entry serves chat A row that carries the cost map key of an entry that serves chat counts as declaring nothing, so a catalog edit that adds a chat path never turns old Admin UI copies into 400s * fix(proxy): check chat sent to a deployment id or backend model name * fix(proxy): check a model the read-through loads from the database * fix(proxy): step aside for fallbacks the check cannot follow and wildcards that serve a backend name A fallback target with no deployments of its own, a client routing stamp, and a wildcard that does not match the requested backend name no longer turn a working request into a 400 * fix(proxy): ignore a client model_group that is not a string when building rate-limit headers A client metadata.model_group sent as an int or a list crashed the error handler with a 500 once the embedding-only check rejected the call before routing. The header helper now treats a non-string stamp as no model group. Drops the pattern clause in the backend-name lookup, since the group lookup already returns wildcard matches. Adds the third audit round's routing-stamp cells. * fix(proxy): keep a non-string client model_group out of spend rows * fix(proxy): validate standard fallbacks before the typed fallback-group lookup * feat(router): opt in to non-chat deployment filtering * fix(router): skip unresolved local rate-limit fallbacks * fix(router): ignore caller-provided endpoint selector values * fix(router): normalize declared chat endpoint prefixes * test(router): cover preflight error-value behavior * test(router): derive replay metadata defaults from model info * fix(router): preserve fallback aliases and check synchronous text calls * test(routing): rewrite the WebSocket model with a pre-call callback, since main no longer runs guardrails at socket admission * test(routing): keep the endpoint gate's owned proxies on the worker's database * fix(router): keep the endpoint gate on model_info patches that omit it and on same-group stream retries A partial PATCH of model_info reset a stored enforce_supported_endpoints true back to false, because the field defaulted to False and the update merges model_dump(exclude_none=True). The field now defaults to None so an omitted flag keeps the stored value The same-group retry of a stream dropped before content called the provider without the endpoint selection context, so it could land on an enforced embedding-only deployment of the same group. It now runs inside endpoint_selection like every other router attempt * fix(router): probe rate-limit fallbacks through get_routable_deployments so the router surface does not grow Main's router fence lets code outside the router use only listed Router members and router_utils modules. The proxy's fallback probe now calls the listed get_routable_deployments with the request's call type, and the gate's 400 moves to litellm.exceptions as EndpointNotSupportedByModelError. It stays separate from main's UnsupportedEndpointError, which the router retries on the group's other deployments, while the gate refuses the whole group * test(responses): cover the WebSocket error frame for a model that refuses the endpoint * fix(proxy): let the rate-limit fallback probe try a fallback this worker has not loaded yet The probe now skips a fallback only when the endpoint check removed every deployment it has, so a model added on another worker still reaches route_request and its database lookup * fix(proxy): keep key router settings off the Responses WebSocket, read unrooted or single-string endpoint lists, and add status to the refusal frame * fix(router): warn when enforce_supported_endpoints has no usable list and match chat endpoints regardless of case, trailing slash, or Gemini path * test(router): type the load-warning test's caplog and router * test(routing): match the endpoint gate's integration tests to base fallback behavior on unloaded fallbacks and the Responses WebSocket --------- Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain> |
||
|---|---|---|
| .. | ||
| fixtures | ||
| realtime | ||
| conftest.py | ||
| conversational_matrix.py | ||
| LLM_TRANSLATION_COVERAGE_MATRIX.md | ||
| passthrough_client.py | ||
| responses_helpers.py | ||
| sdk_clients.py | ||
| structured_output.py | ||
| test_anthropic_completion_e2e.py | ||
| test_audio_speech_e2e.py | ||
| test_audio_transcriptions_e2e.py | ||
| test_bedrock_converse_redacted_thinking_e2e.py | ||
| test_bedrock_guardrail_config_e2e.py | ||
| test_bedrock_kimi_k3_messages_e2e.py | ||
| test_bedrock_mantle_output_limit_e2e.py | ||
| test_bedrock_native_e2e.py | ||
| test_bedrock_provider_matrix_e2e.py | ||
| test_bedrock_web_search_server_tool_e2e.py | ||
| test_cache_control.py | ||
| test_cache_control_injection_tool_calls_e2e.py | ||
| test_chat_completion_prediction_e2e.py | ||
| test_chat_completions_contract_e2e.py | ||
| test_chat_completions_regression_e2e.py | ||
| test_chat_mid_conversation_system_e2e.py | ||
| test_chat_stream_contract_e2e.py | ||
| test_chat_tool_round_trip_e2e.py | ||
| test_completions_endpoint_e2e.py | ||
| test_containers_e2e.py | ||
| test_conversational_matrix_e2e.py | ||
| test_count_tokens_source_e2e.py | ||
| test_credential_messages_e2e.py | ||
| test_custom_pricing_e2e.py | ||
| test_deepseek_reasoning_e2e.py | ||
| test_embedding_only_supported_endpoints_e2e.py | ||
| test_embeddings_endpoint_e2e.py | ||
| test_files_batches_contract_e2e.py | ||
| test_gemini_url_context_e2e.py | ||
| test_google_interactions_e2e.py | ||
| test_google_native_e2e.py | ||
| test_image_edits_e2e.py | ||
| test_image_generation_e2e.py | ||
| test_messages_azure_foundry_e2e.py | ||
| test_messages_bedrock_e2e.py | ||
| test_messages_beta_headers_e2e.py | ||
| test_messages_count_tokens_bedrock_e2e.py | ||
| test_messages_e2e.py | ||
| test_messages_mid_conversation_system_e2e.py | ||
| test_messages_mid_conversation_system_native_providers_e2e.py | ||
| test_messages_prompt_caching_and_tool_search_e2e.py | ||
| test_moderations_e2e.py | ||
| test_ocr_rust_e2e.py | ||
| test_ollama_e2e.py | ||
| test_passthrough_e2e.py | ||
| test_passthrough_headers_e2e.py | ||
| test_provider_features_e2e.py | ||
| test_realtime_http_e2e.py | ||
| test_reasoning_effort_e2e.py | ||
| test_rerank_e2e.py | ||
| test_responses_bridge_streaming_e2e.py | ||
| test_responses_codex_tool_search_e2e.py | ||
| test_responses_e2e.py | ||
| test_responses_retrieve_e2e.py | ||
| test_sagemaker_nova_e2e.py | ||
| test_sail_e2e.py | ||
| test_together_ai_e2e.py | ||
| test_token_counter_gemini_contents_e2e.py | ||
| test_vector_stores_e2e.py | ||
| test_vertex_capabilities_e2e.py | ||
| test_vertex_passthrough_e2e.py | ||