mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-15 23:31:29 +00:00
39e0efa11d
* fix(anthropic-messages): send bare-string tool_choice to Responses API, propagate router-alias litellm_params
The Anthropic /v1/messages -> Responses API adapter always wrapped
tool_choice in an object ({"type": "auto"}, {"type": "required"}), but
the Responses API's tool_choice schema for these cases is a bare
string ("auto"/"required"/"none"). Sending the object shape to an
OpenAI-compatible backend (e.g. vLLM) fails Pydantic validation with a
400. The "none" case also fell through to "auto" instead of mapping to
"none".
Separately, litellm_params configured directly on a router-alias
deployment (auto_router/complexity_router, adaptive_router,
quality_router, or semantic auto_router) - e.g.
cache_control_injection_points, drop_params - were silently dropped
for every request through that alias. async_pre_routing_hook swaps
`model` from the alias name to the selected tier/route's model before
the deployment lookup runs, so the outbound call only ever merged in
the tier deployment's own litellm_params, never the alias's. Register
non-routing-config litellm_params from the alias deployment and apply
them to the request whenever a pre-routing hook substitutes the model.
* fix: satisfy ruff-strict-budget UP006 and router coverage checker
Use builtin dict[...] generics instead of typing.Dict for the two new
annotations introduced in the previous commit, since they pushed
UP006 over the codebase ceiling in ruff-strict-budget.json. Add a
direct unit test for _register_pre_routing_alias_overrides so the
text-based router_code_coverage.py checker sees it exercised by name.
* fix(router): replace alias-param denylist with a tight allowlist
_PRE_ROUTING_ALIAS_RESERVED_PARAMS excluded router-init-only keys from
the alias's litellm_params before forwarding the rest as request
kwargs, but GenericLiteLLMParams also holds deployment-management
fields (tpm, rpm, weight, tags, max_budget, budget_duration,
use_in_pass_through, litellm_credential_name, ...) on the same object.
Any of those left off the denylist would get silently forwarded as if
they were request kwargs.
Replace the denylist with a tight allowlist of exactly the two
request-shaping params this feature exists for - drop_params and
cache_control_injection_points - so unrelated management fields never
reach the outbound call regardless of what else GenericLiteLLMParams
grows to hold.
* fix(router): re-register adaptive-alias overrides on set_model_list reload
set_model_list() unconditionally clears pre_routing_alias_overrides on
every call (e.g. /config/reload), but _finalize_adaptive_router_if_configured()
skips rebuilding an AdaptiveRouter whose model_name already exists in
self.adaptive_routers - so _register_pre_routing_alias_overrides() never
ran again for an auto_router/adaptive_router alias after a reload,
silently dropping its drop_params/cache_control_injection_points.
Build the Deployment unconditionally and re-register its overrides even
on the skip-existing-router path; only the (expensive) AdaptiveRouter
construction itself stays skipped.
* style: ruff format after merging litellm_internal_staging
* fix(router): drop the alias-param allowlist, exclude only model
Per review discussion: instead of a router.py-local allowlist of exactly
which litellm_params an alias (auto_router/complexity_router,
adaptive_router, quality_router, semantic auto_router) can forward to
the request it routes, _register_pre_routing_alias_overrides now
forwards everything except `model` (the alias marker itself, e.g.
auto_router/complexity_router, never a real provider model).
Router-init-only fields (complexity_router_config,
complexity_router_default_model, auto_router_config,
auto_router_config_path, auto_router_default_model,
auto_router_embedding_model, adaptive_router_config,
adaptive_router_default_model, quality_router_config,
quality_router_default_model) now flow into request_kwargs unfiltered
too. That's safe because litellm.completion()/acompletion() already
strips anything in litellm.types.utils.all_litellm_params before
building the provider request - added these 10 keys there, alongside
the deployment-management fields (tpm, rpm, weight, ...) already listed.
Verified live: without that addition, complexity_router_config lands in
extra_body and ships raw to the provider; with it, it's stripped.
This moves the "which fields aren't real LLM params" list from a
router.py-local allowlist to the single existing global list every
completion() call already depends on, instead of maintaining two.
* refactor(router): look up alias litellm_params on demand instead of caching them
_register_pre_routing_alias_overrides cached each alias's litellm_params
into self.pre_routing_alias_overrides at deployment-init time, which
required keeping that cache in sync with set_model_list() reloads - the
exact bug the previous adaptive-router-reload fix was patching around
(AdaptiveRouter survives a reload, but the cache didn't always get
refreshed to match).
Delete the cache and the registration method entirely. async_pre_routing_hook
now looks up the alias's own litellm_params directly from self.model_list
via self.model_name_to_deployment_indices at request time, the same
model_list that's already correctly rebuilt on every set_model_list()
call. No second piece of state to invalidate, so the reload staleness
bug class isn't possible anymore, and it's less code than before.
|
||
|---|---|---|
| .. | ||
| a2a/chat | ||
| aiml/image_generation | ||
| amazon_nova/chat | ||
| anthropic | ||
| apiserpent | ||
| azure | ||
| azure_ai | ||
| base_llm | ||
| baseten/chat | ||
| bedrock | ||
| bedrock_mantle | ||
| black_forest_labs | ||
| bytez/chat | ||
| chat | ||
| chatgpt | ||
| cloudflare | ||
| cohere | ||
| cometapi/chat | ||
| compactifai | ||
| crusoe | ||
| custom_httpx | ||
| dashscope | ||
| databricks | ||
| datarobot | ||
| deepgram | ||
| deepinfra | ||
| deepseek | ||
| docker_model_runner | ||
| elevenlabs | ||
| fal_ai/image_generation | ||
| fastcrw/search | ||
| featherless_ai/chat | ||
| fireworks_ai | ||
| gdc/chat | ||
| gemini | ||
| github_copilot | ||
| heroku | ||
| hosted_vllm | ||
| huggingface | ||
| inception | ||
| jina_ai/embedding | ||
| langflow | ||
| lemonade | ||
| litellm_proxy | ||
| llamafile/chat | ||
| lm_studio | ||
| manus | ||
| meta_llama | ||
| minimax | ||
| mistral | ||
| modelscope | ||
| moonshot | ||
| nebius | ||
| neosantara | ||
| novita/chat | ||
| nscale/chat | ||
| nvidia_riva | ||
| oci | ||
| ocr | ||
| ollama | ||
| openai | ||
| openai_like | ||
| openrouter | ||
| ovhcloud | ||
| parallel_ai | ||
| parasail | ||
| pass_through | ||
| perplexity | ||
| pg_vector/vector_stores | ||
| publicai | ||
| ragflow/chat | ||
| recraft | ||
| reducto | ||
| runwayml | ||
| s3_vectors | ||
| sagemaker | ||
| sambanova | ||
| sap | ||
| scaleway | ||
| snowflake | ||
| soniox | ||
| stability | ||
| tencent | ||
| tinyfish | ||
| vercel_ai_gateway | ||
| vertex_ai | ||
| volcengine | ||
| voyage | ||
| wandb | ||
| watsonx | ||
| xai | ||
| you_com | ||
| zai | ||
| test_cache_control_and_reasoning.py | ||
| test_file_content_block.py | ||
| test_file_search_responses.py | ||
| test_lifecycle_fix.py | ||
| test_oom_fixes.py | ||
| test_polling_url_origin_match.py | ||
| test_predibase_transformation.py | ||