* fix(main.py): fix async retryer
Fixes https://github.com/BerriAI/litellm/issues/12830
* fix(forward_clientside_headers_by_model_group.py): filter out 'content-type' from forwardable headers
clientside content-type != proxy content type, can cause requests to hang
* test(tests/): update tests
* feat: add Morph provider support
- Add MorphChatConfig implementation for OpenAI-compatible API
- Support morph-v3-fast and morph-v3-large models
- Add pricing: morph-v3-fast (/bin/zsh.8/.2 per 1M tokens), morph-v3-large (/bin/zsh.9/.9 per 1M tokens)
- Both models support 16k context window and system messages
- Add comprehensive documentation and unit tests
- Update all necessary integration points (constants, init, provider logic)
* feat: Add Morph provider support in ProviderConfigManager
- Extend ProviderConfigManager to include MorphChatConfig for the Morph LLM provider.
- Update MorphChatConfig by removing unused parameters from the configuration.
- Add Hyperbolic as a new OpenAI-compatible provider
- Implement HyperbolicChatConfig inheriting from OpenAILikeChatConfig
- Register Hyperbolic in provider lists and constants
- Add comprehensive model configurations with pricing for:
- DeepSeek models (V3, R1, etc.)
- Qwen models (2.5, 3, QwQ, etc.)
- Meta Llama models (3.1, 3.2, 3.3)
- Other models like Kimi K2, Hermes 3, etc.
- Configure default API base URL: https://api.hyperbolic.xyz/v1
- Add provider documentation with usage examples
- Create unit tests for provider functionality
- Support all standard OpenAI parameters
Hyperbolic provides low-cost inference with OpenAI-compatible APIs,
supporting latest models without infrastructure overhead.
* feat: add Lambda AI provider support
Add support for Lambda AI (lambda.ai) as a new LLM provider in LiteLLM. Lambda AI provides access to a wide range of open-source models through their cloud GPU infrastructure.
Changes:
- Add Lambda AI provider implementation (OpenAI-compatible)
- Register 20 Lambda AI models with accurate pricing and 131k context windows
- Add comprehensive tests for Lambda AI integration
- Add detailed documentation with usage examples
- Use "lambda_ai" as provider name to avoid Python keyword conflict
Models include Llama 3.x, DeepSeek, Hermes, Qwen, and specialized models for coding and vision tasks.
* fix(tests): ensure lambda_ai_models list is repopulated after model cost reload
Updated test cases to clear and repopulate the lambda_ai_models list after reloading the model cost map. This ensures that the tests accurately reflect the current state of available models.
* feat: add Lambda AI chat configuration support
Added support for Lambda AI chat configuration in the ProviderConfigManager. This enhancement allows the integration of Lambda AI as a provider, expanding the capabilities of LiteLLM.
* Feature/track bedrock gov cloud models (#12771)
* feat: add AWS Bedrock GovCloud model support (LIT-257)
- Added 18 GovCloud-specific model entries (9 per region) to model_prices_and_context_window.json
- Updated is_bedrock_pricing_only_model() to allow GovCloud models (us-gov-east-1, us-gov-west-1)
- Added comprehensive test suite for GovCloud model support
- Ensures GovCloud models use appropriate APIs (Converse for Claude/Llama, Invoke for Titan)
Models added:
- Claude 3.5 Sonnet and Claude 3 Haiku (FedRAMP/IL4/5 approved)
- Llama 3 8B and 70B (FedRAMP/IL4/5 approved)
- Amazon Titan Text and Embedding models
* fix: add bedrock_converse GovCloud model mappings for Claude models
Added missing bedrock_converse model entries for AWS GovCloud regions:
- bedrock_converse/us-gov-east-1/anthropic.claude-3-5-sonnet-20240620-v1:0
- bedrock_converse/us-gov-east-1/anthropic.claude-3-haiku-20240307-v1:0
- bedrock_converse/us-gov-west-1/anthropic.claude-3-5-sonnet-20240620-v1:0
- bedrock_converse/us-gov-west-1/anthropic.claude-3-haiku-20240307-v1:0
This fixes test failures where supports_tool_choice() returned True but
the models weren't properly mapped in the configuration files.
* fix: correct AWS GovCloud Bedrock model pricing and configurations
- Fix Claude 3.5 Sonnet pricing (3.6e-06 input, 1.8e-05 output)
- Fix Claude 3 Haiku pricing (3e-07 input, 1.5e-06 output)
- Update Claude 3.5 Sonnet max_tokens from 4096 to 8192
- Add bedrock_converse entries for Llama models with correct token limits
- Add Amazon Nova Pro model for both GovCloud regions
- Add supports_pdf_input flag to Claude models
* fix: handle bedrock_converse prefix in get_non_litellm_routing_model_name
Fixes test failure where bedrock_converse/region/model paths were not properly
stripped to get the base model name, causing supports_function_calling to
return false for regional bedrock_converse models.
* revert: reset bedrock/common_utils.py to match main branch
Remove bedrock_converse prefix handling from get_non_litellm_routing_model_name
to align with main branch implementation.
* revert: reset litellm/__init__.py to match main branch
- Remove public_model_groups variables
- Remove GovCloud exception handling in is_bedrock_pricing_only_model
- Fix comment formatting
* revert: reset litellm/__init__.py to exact main branch content
Copy exact content from origin/main with no modifications
* fix: remove bedrock_converse prefixed models from pricing files
- Remove 10 bedrock_converse entries from model_prices_and_context_window.json
- Remove 4 bedrock_converse entries from litellm/model_prices_and_context_window_backup.json
- These were GovCloud-specific entries that are no longer needed
* fix: correct AWS GovCloud Bedrock model pricing and configurations
- Fix Anthropic Claude 3.5 Sonnet pricing: $3.60/$18.00 per million tokens (was $3.00/$15.00)
- Fix Anthropic Claude 3 Haiku pricing: $0.30/$1.50 per million tokens (was $0.25/$1.25)
- Fix Claude 3.5 Sonnet max_tokens: 8192 (was 4096)
- Fix Llama model max_tokens: 2048 (was 8192) and max_input_tokens: 8000 (was 8192)
- Fix Llama3-8b output pricing: $2.65 per million tokens (was $0.60)
- Add missing Amazon Nova Pro models for both GovCloud regions
- Add supports_pdf_input flag to Llama models
Based on official AWS Bedrock pricing documentation for GovCloud regions
* test: fix GovCloud bedrock models test to match implementation
Update test_govcloud_model_in_bedrock_models_list to correctly verify that
GovCloud models are excluded from bedrock_models list as they are
pricing-only models following the bedrock/<region>/<model> pattern.
---------
Co-authored-by: Cole McIntosh <colemcintosh6@gmail.com>
Co-authored-by: Cole McIntosh <82463175+colesmcintosh@users.noreply.github.com>
* add tests
* add tests
* Added test costs
* Added test costs
---------
Co-authored-by: Cole McIntosh <colemcintosh6@gmail.com>
Co-authored-by: Cole McIntosh <82463175+colesmcintosh@users.noreply.github.com>
* feat: add v0 provider support to LiteLLM
- Add v0 as a new OpenAI-compatible provider
- Support all three v0 models: v0-1.0-md, v0-1.5-md, v0-1.5-lg
- Configure correct token limits and pricing for each model
- Enable vision support for all v0 models (multimodal)
- Add provider detection for v0/ prefix and api.v0.dev endpoint
- Include comprehensive unit tests for the provider
The v0 provider uses the standard OpenAI-compatible implementation
and supports all standard features including streaming, function
calling, and system messages.
* fix: add v0 provider to ProviderConfigManager
Add V0ChatConfig to the get_provider_chat_config method to fix
test_supports_tool_choice test failure. The v0 provider needs to
be included in the provider config manager to return the correct
configuration for tool choice support detection.
* docs: add documentation for v0 provider
- Add comprehensive v0 provider documentation
- Cover all supported models and their capabilities
- Include examples for SDK usage, proxy configuration, and all features
- Document supported OpenAI parameters based on v0 API docs
- Add v0 to the providers sidebar navigation
* fix: correct v0 supported OpenAI parameters
Based on review feedback and v0 API documentation:
- v0 only supports: messages, model, stream, tools, tool_choice
- Remove unsupported parameters like temperature, max_tokens, etc.
- Update tests to verify correct parameter set
- Update documentation to reflect actual API capabilities
- Remove JSON mode example as response_format is not supported
Reference: https://v0.dev/docs/v0-model-api#request-body
* fix: remove supports_response_schema from v0 models
Remove the supports_response_schema property from all v0 models in the model configuration files as v0 does not support this feature.
Models updated:
- v0/v0-1.0-md
- v0/v0-1.5-md
- v0/v0-1.5-lg
* fix(prompt_templates/factory.py): handle anthropic cache control on individual tool results
Fixes issue where cache control on individual tool result was being ignored
* test(test_vertex_And_google_ai_studio_gemini.py): initial unit test covering translation for grounding metadata on streaming chunk
* fix(google_genai/adapters/transformation.py): enable calling non-googlegenai models via streaming
Fixes https://github.com/BerriAI/litellm/issues/12562
* test(test_openai.py): add unit test asserting streaming works as expected
* fix(proxy_server.py): fixes for handling team only models via `/v2/model/info`
ensures team only models show up on the correct team on `Models + Endpoints`
* test: update tests
- Add ttl parameter to cache_control for Gemini models
- Support Google's TTL format (e.g., '3600s', '7200s')
- Implement robust TTL extraction and validation
- Extract TTL before system message transformation to handle all cases
- Add comprehensive test suite with 17 test cases in tests/test_litellm/
- Update documentation with TTL usage examples
- Maintain backward compatibility with existing cache_control usage
Fixes#9810
* fix(batches_endpoints/endpoints.py): support passing target model names for batch list as a query param
Fixes issue where cloud run fails calls because GET can't contain request body
* test(test_openai_batches_endpoints.py): add unit test
* docs(managed_batches.md): update docs
* feat(spend_tracking_utils.py): support STORE_PROMPTS_IN_SPEND_LOGS env var
ensures prompt is stored in spend logs
* fix(streaming_iterator.py): fix anthropic - completion streaming iterator to yield content block stop
ensures claude code renders messages
* test: skip local test
* fix(proxy_server.py): handle empty config yaml
Fixes https://github.com/BerriAI/litellm/issues/12163
* fix(gemini/common_utils.py): replace models/ as expected, instead of using 'strip'
Fixes https://github.com/BerriAI/litellm/issues/12160
* fix(anthropic/experimental_pass_through/messages/transformation.py): check for env var when selecting api key
* fix(anthropic/transformation.py): return tool_use content block start on anthropic bridge
Closes https://github.com/BerriAI/litellm/issues/12158
* fix(anthropic/streaming_iterator.py): fix setting index in block
ensure index is set just once and increments correctly when a new block is created
* fix(anthropic/adapters/handler.py): update logging obj with stream options value if set
* feat(anthropic/streaming_iterator.py): return usage from chat completion to messages bridge
enables usage tracking for non-anthropic models
Closes https://github.com/BerriAI/litellm/issues/12132
* fix(streaming_iterator.py): safely access usage chunk
* fix: suppress linting error
* test: update tests
* fix: fix streaming errors
* fix(proxy_server.py): handle empty config yaml
Fixes https://github.com/BerriAI/litellm/issues/12163
* fix(gemini/common_utils.py): replace models/ as expected, instead of using 'strip'
Fixes https://github.com/BerriAI/litellm/issues/12160
* fix(anthropic/experimental_pass_through/messages/transformation.py): check for env var when selecting api key
* docs(config_settings.md): add api key to docs
* refactor(passthrough_endpoints-success-handler): refactor llm passthrough logging logic
isolate the llm translation work to enable cost tracking on sdk
* feat: initial implementation of passthrough SDK cost calculation
enables bedrock passthrough cost tracking to work
* feat(cost_calculator.py): working cost calculation for bedrock passthrough
* feat(litellm_logging.py): consider allm_passthrough in cost tracking
allows async calls (e.g. via proxy) to work
* feat(bedrock/passthrough): working event stream decoding for bedrock passthrough calls + logging instrumentation for passthrough sdk calls (log on stream completion)
Enables bedrock streaming cost calculation
* feat(litellm_logging.py): support streaming passthrough cost tracking
* feat(passthrough/main.py): working async streaming cost calculation
Closes https://github.com/BerriAI/litellm/issues/11359
* feat(proxy_server.py): fix passthrough routing when llm router enabled
* feat: further fixes
* feat(bedrock/): working bedrock passthrough cost tracking (non-streaming)
* feat(litellm_logging.py): working usage tracking for bedrock passthrough calls
ensures tokens are logged
* feat(bedrock/passthrough): add converse passthrough cost tracking support
* feat(base_llm/passthrough): remove redundant function
* refactor(litellm_logging.py): refactor function to be below 50 LOC
* test: update test
* test: remove redundant test
* feat: initial commit adding bedrock support via the new sdk passthrough logic
ensures correct sequencing of tasks (pre call checks etc. can run before signing request)
* fix(route_llm_requests.py): passthrough to allm_passthrough_route if no model found
* feat(bedrock/passthrough): working bedrock passthrough via sdk support
* fix(passthrough/main.py): re-add data and json
* feat(passthrough/main): support async passthrough calls to bedrock
* feat(passthrough/main.py): async streaming + completion support
* feat(llm_passthrough_endpoints.py): migrate bedrock passthrough calls to to new bedrock passthrough sdk
Enables calls to work correctly
* fix: fix linting errors
* test: update test
* fix(handler.py): support routing custom llm's to chat completion handler
Adds custom llm support for anthropic
* test(test_anthropic_experimental_pass_through_messages_handler.py): add unit test confirming custom llm respected
* docs(custom_llm_server.md): document anthropic custom llm translation
* test(volcengine.py): map thinking in extra body
Fixes https://github.com/BerriAI/litellm/issues/11879
* feat(main.py): support `azure/responses/<deployment-name>` model string
this allows us to route the model correctly
Closes https://github.com/BerriAI/litellm/issues/11879
* docs(azure_responses.md): document calling azure responses api models via chat completions bridge
Closes https://github.com/BerriAI/litellm/issues/11917
* fix: fix custom provider check
* test: update tests
* Enhance Mistral API: Add support for parallel tool calls and refine name handling in tool messages. Plus, introduce a new test for parallel tool calls in the Mistral model.
* tests
* make mypy happy
* Refine name handling in Mistral chat transformation: clarify conditions for removing the 'name' field based on message role and content.
* fix(spend_tracking_utils.py): add user agent tags from standard logging payload, in spend logs payload
* feat(litellm_logging.py): identify user agent tags as `User-Agent: ..` and allow admin to disable storing user agent as tag
* fix(azure_ai/): pass content type header in azure ai request
Fixes https://github.com/BerriAI/litellm/issues/11227
* test: add unit test
* fix(router.py): fix passing dynamic credentials to retrieve batch
Fixes batch retrieval when using router
* test: add more unit tests
* fix(anthropic/chat/transformation.py): append prefix to start of assistant response, if set
ensures assistant response contains complete response
* fix(anthropic/chat/transformation.py): add flag to allow user to opt out of enabling prefix in prompt
* fix(anthropic/chat/transformation.py): working e2e support for prefix prompt in assistant response
* feat(networking.tsx): always include model access groups on UI
show admin created access groups when giving key/user/team model permissions
* feat(add_model_tab.tsx): initial ui component for adding to an existing model access group
allows user to add model to an access group (simplify giving users/keys/teams model access)
* feat(proxy_server.py): add 'only_model_access_groups' flag support to `/v1/models`
simplifies listing available access groups on UI
* test: add e2e test for new only_model_access_groups param
* feat(add_model_tab.tsx): allow adding+viewing model access groups on models tab
make feature functional on UI
* feat(view_users.tsx): route edit user to user info page
more detailed user edit
* feat(columns.tsx): route edit user to user info page
more detailed user edit
* fix(columns.tsx): fix linting error
* build(ui/): fix linting errors
* fix(utils.py): convert stringified numbers to numbers
Closes https://github.com/BerriAI/litellm/issues/11266
* fix(convert_dict_to_model_response_object/): bubble up azure content_filter_results
* fix: fix linting error
* fix: fix linting errors
* fix(types/utils.py): ensure choices is correctly set
* fix: delete field if not set
* fix: expand scope of choicelogprobs value
* refactor(responses/): refactor to move responses_to_completion in separate folder
future work to support completion_to_responses bridge
allow calling codex mini via chat completions (and other endpoints)
* Revert "refactor(responses/): refactor to move responses_to_completion in separate folder"
This reverts commit ff87cb8958.
* feat: initial responses api bridge
write it like a custom llm - requires lesser 'new' components
* style: add __init__'s and bubble up the responses api bridge
* feat(responses/transformation): working sync completion -> responses and back bridge (non-streaming)
* feat(responses/): working async (non-streaming) completion <-> responses bridge
Allows calling codex mini via proxy
* feat(responses/): working sync + async streaming for base model response iterator
* fix: reduce function size
maintain <50 LOC
* fix(main.py): safely handle responses api model check
* fix: fix linting errors
