* feat: Add Pillar Security guardrail integration
Implements comprehensive LLM security guardrails using Pillar Security API with support for prompt injection detection, PII/secret detection, content moderation, and multi-mode execution (pre_call, during_call, post_call). Includes complete documentation, testing, and configurable actions on flagged content.
* fix: Resolve MyPy type error in Pillar guardrail config
Restructure PillarGuardrailConfigModel to properly inherit from GuardrailConfigModel[T]
and resolve return type compatibility issue in get_config_model method.
* fix: Resolve MyPy type error in Pillar guardrail config
Restructure PillarGuardrailConfigModel to properly inherit from GuardrailConfigModel[T]
and resolve return type compatibility issue in get_config_model method.
* fix docs
* fix docs
* improved docs
* fix examples, READY
* feat(litellm_pre_call_utils.py): add num_retries to litellm data for backend call
allow user to pass in num retries via request headers
* test(test_litellm_pre_call_utils.py): add unit test
* docs(request_headers.md): document new request header
* fix(common_daily_activity.py): show spend breakdown by model group
Partial fix for https://github.com/BerriAI/litellm/issues/12887
* feat(new_usage.tsx): new tab switcher for viewing usage by model group vs. received model
Closes https://github.com/BerriAI/litellm/issues/12887
* fix(main.py): fix async retryer
Fixes https://github.com/BerriAI/litellm/issues/12830
* fix(forward_clientside_headers_by_model_group.py): filter out 'content-type' from forwardable headers
clientside content-type != proxy content type, can cause requests to hang
This aligns the proxy experience with other models that think
automatically (e.g. Deepseek R1 and grok3). It does so by setting
the necessary request input to return thinking, but not specifying
a budget or effort (thus defaulting to the internal automatic level).
* fix(gpt_transformation.py): remove 'cache_control' flag for openai/openai-compatible calls
Fixes https://github.com/BerriAI/litellm/issues/12787
* fix(openrouter/chat/transformation.py): allow passing openrouter cache control flag for claude models
* fix(gpt_transformation.py): fix import
* fix: fix adding tools
* fix(main.py): fix async retryer
Fixes https://github.com/BerriAI/litellm/issues/12830
* fix(forward_clientside_headers_by_model_group.py): filter out 'content-type' from forwardable headers
clientside content-type != proxy content type, can cause requests to hang
* test(tests/): update tests
* fix(team_endpoints.py): always remove team member budget from updated_kv
this is not a field for the litellm team table
Prevents startup issue
* test(test_team_endpoints.py): add unit test to ensure 'team_member_budget' is never in update to table - separate logic
* refactor: cleanup
* feat: add Morph provider support
- Add MorphChatConfig implementation for OpenAI-compatible API
- Support morph-v3-fast and morph-v3-large models
- Add pricing: morph-v3-fast (/bin/zsh.8/.2 per 1M tokens), morph-v3-large (/bin/zsh.9/.9 per 1M tokens)
- Both models support 16k context window and system messages
- Add comprehensive documentation and unit tests
- Update all necessary integration points (constants, init, provider logic)
* feat: Add Morph provider support in ProviderConfigManager
- Extend ProviderConfigManager to include MorphChatConfig for the Morph LLM provider.
- Update MorphChatConfig by removing unused parameters from the configuration.
- Add Hyperbolic as a new OpenAI-compatible provider
- Implement HyperbolicChatConfig inheriting from OpenAILikeChatConfig
- Register Hyperbolic in provider lists and constants
- Add comprehensive model configurations with pricing for:
- DeepSeek models (V3, R1, etc.)
- Qwen models (2.5, 3, QwQ, etc.)
- Meta Llama models (3.1, 3.2, 3.3)
- Other models like Kimi K2, Hermes 3, etc.
- Configure default API base URL: https://api.hyperbolic.xyz/v1
- Add provider documentation with usage examples
- Create unit tests for provider functionality
- Support all standard OpenAI parameters
Hyperbolic provides low-cost inference with OpenAI-compatible APIs,
supporting latest models without infrastructure overhead.
* feat: add Lambda AI provider support
Add support for Lambda AI (lambda.ai) as a new LLM provider in LiteLLM. Lambda AI provides access to a wide range of open-source models through their cloud GPU infrastructure.
Changes:
- Add Lambda AI provider implementation (OpenAI-compatible)
- Register 20 Lambda AI models with accurate pricing and 131k context windows
- Add comprehensive tests for Lambda AI integration
- Add detailed documentation with usage examples
- Use "lambda_ai" as provider name to avoid Python keyword conflict
Models include Llama 3.x, DeepSeek, Hermes, Qwen, and specialized models for coding and vision tasks.
* fix(tests): ensure lambda_ai_models list is repopulated after model cost reload
Updated test cases to clear and repopulate the lambda_ai_models list after reloading the model cost map. This ensures that the tests accurately reflect the current state of available models.
* feat: add Lambda AI chat configuration support
Added support for Lambda AI chat configuration in the ProviderConfigManager. This enhancement allows the integration of Lambda AI as a provider, expanding the capabilities of LiteLLM.
* Feature/track bedrock gov cloud models (#12771)
* feat: add AWS Bedrock GovCloud model support (LIT-257)
- Added 18 GovCloud-specific model entries (9 per region) to model_prices_and_context_window.json
- Updated is_bedrock_pricing_only_model() to allow GovCloud models (us-gov-east-1, us-gov-west-1)
- Added comprehensive test suite for GovCloud model support
- Ensures GovCloud models use appropriate APIs (Converse for Claude/Llama, Invoke for Titan)
Models added:
- Claude 3.5 Sonnet and Claude 3 Haiku (FedRAMP/IL4/5 approved)
- Llama 3 8B and 70B (FedRAMP/IL4/5 approved)
- Amazon Titan Text and Embedding models
* fix: add bedrock_converse GovCloud model mappings for Claude models
Added missing bedrock_converse model entries for AWS GovCloud regions:
- bedrock_converse/us-gov-east-1/anthropic.claude-3-5-sonnet-20240620-v1:0
- bedrock_converse/us-gov-east-1/anthropic.claude-3-haiku-20240307-v1:0
- bedrock_converse/us-gov-west-1/anthropic.claude-3-5-sonnet-20240620-v1:0
- bedrock_converse/us-gov-west-1/anthropic.claude-3-haiku-20240307-v1:0
This fixes test failures where supports_tool_choice() returned True but
the models weren't properly mapped in the configuration files.
* fix: correct AWS GovCloud Bedrock model pricing and configurations
- Fix Claude 3.5 Sonnet pricing (3.6e-06 input, 1.8e-05 output)
- Fix Claude 3 Haiku pricing (3e-07 input, 1.5e-06 output)
- Update Claude 3.5 Sonnet max_tokens from 4096 to 8192
- Add bedrock_converse entries for Llama models with correct token limits
- Add Amazon Nova Pro model for both GovCloud regions
- Add supports_pdf_input flag to Claude models
* fix: handle bedrock_converse prefix in get_non_litellm_routing_model_name
Fixes test failure where bedrock_converse/region/model paths were not properly
stripped to get the base model name, causing supports_function_calling to
return false for regional bedrock_converse models.
* revert: reset bedrock/common_utils.py to match main branch
Remove bedrock_converse prefix handling from get_non_litellm_routing_model_name
to align with main branch implementation.
* revert: reset litellm/__init__.py to match main branch
- Remove public_model_groups variables
- Remove GovCloud exception handling in is_bedrock_pricing_only_model
- Fix comment formatting
* revert: reset litellm/__init__.py to exact main branch content
Copy exact content from origin/main with no modifications
* fix: remove bedrock_converse prefixed models from pricing files
- Remove 10 bedrock_converse entries from model_prices_and_context_window.json
- Remove 4 bedrock_converse entries from litellm/model_prices_and_context_window_backup.json
- These were GovCloud-specific entries that are no longer needed
* fix: correct AWS GovCloud Bedrock model pricing and configurations
- Fix Anthropic Claude 3.5 Sonnet pricing: $3.60/$18.00 per million tokens (was $3.00/$15.00)
- Fix Anthropic Claude 3 Haiku pricing: $0.30/$1.50 per million tokens (was $0.25/$1.25)
- Fix Claude 3.5 Sonnet max_tokens: 8192 (was 4096)
- Fix Llama model max_tokens: 2048 (was 8192) and max_input_tokens: 8000 (was 8192)
- Fix Llama3-8b output pricing: $2.65 per million tokens (was $0.60)
- Add missing Amazon Nova Pro models for both GovCloud regions
- Add supports_pdf_input flag to Llama models
Based on official AWS Bedrock pricing documentation for GovCloud regions
* test: fix GovCloud bedrock models test to match implementation
Update test_govcloud_model_in_bedrock_models_list to correctly verify that
GovCloud models are excluded from bedrock_models list as they are
pricing-only models following the bedrock/<region>/<model> pattern.
---------
Co-authored-by: Cole McIntosh <colemcintosh6@gmail.com>
Co-authored-by: Cole McIntosh <82463175+colesmcintosh@users.noreply.github.com>
* add tests
* add tests
* Added test costs
* Added test costs
---------
Co-authored-by: Cole McIntosh <colemcintosh6@gmail.com>
Co-authored-by: Cole McIntosh <82463175+colesmcintosh@users.noreply.github.com>
When using Model Armor guardrail with explicit project_id in config,
the project_id was being overwritten to None due to incorrect
initialization order between ModelArmorGuardrail and VertexBase parent class.
This fix ensures that user-provided project_id is preserved by initializing
parent classes before setting instance attributes.
Fixes#12757
* feat: initial commit for forwarding client headers by model group
* fix(router.py): support new forwarclientsideheadersbymodelgroup class
enables headers to be forwarded to backend model, by model group
* fix(proxy_server.py): load in model group settings from config correctly
* refactor(litellm_pre_call_utils.py): litellm_pre_call_utils.py
introduce new 'secret_fields' field
includes raw request headers (not the sanitized ones used for logging) - needed to support forwarding clientside headers to llm api
* feat(router.py): log the deployment model name as well
allows wildcard models to support forward_client_headers_to_llm_api
* test(test_router.py): add more unit testing
* feat(router.py): specify the model group alias in metadata kwargs
allows usage for internal routing logic
* fix: fix ruff check errors
* fix(router.py): refactor to cleanup optional pre-call checks
* fix: fix ruff check
* test: add missing unit test
* fix(prompt_templates/factory.py): handle anthropic cache control on individual tool results
Fixes issue where cache control on individual tool result was being ignored
* test(test_vertex_And_google_ai_studio_gemini.py): initial unit test covering translation for grounding metadata on streaming chunk
* fix(vertex_and_google_ai_studio.py): ensure grounding metadata is preserved on streaming
Closes https://github.com/BerriAI/litellm/issues/10237
* fix(core_helpers.py): include usage in expected openai keys
* feat: add v0 provider support to LiteLLM
- Add v0 as a new OpenAI-compatible provider
- Support all three v0 models: v0-1.0-md, v0-1.5-md, v0-1.5-lg
- Configure correct token limits and pricing for each model
- Enable vision support for all v0 models (multimodal)
- Add provider detection for v0/ prefix and api.v0.dev endpoint
- Include comprehensive unit tests for the provider
The v0 provider uses the standard OpenAI-compatible implementation
and supports all standard features including streaming, function
calling, and system messages.
* fix: add v0 provider to ProviderConfigManager
Add V0ChatConfig to the get_provider_chat_config method to fix
test_supports_tool_choice test failure. The v0 provider needs to
be included in the provider config manager to return the correct
configuration for tool choice support detection.
* docs: add documentation for v0 provider
- Add comprehensive v0 provider documentation
- Cover all supported models and their capabilities
- Include examples for SDK usage, proxy configuration, and all features
- Document supported OpenAI parameters based on v0 API docs
- Add v0 to the providers sidebar navigation
* fix: correct v0 supported OpenAI parameters
Based on review feedback and v0 API documentation:
- v0 only supports: messages, model, stream, tools, tool_choice
- Remove unsupported parameters like temperature, max_tokens, etc.
- Update tests to verify correct parameter set
- Update documentation to reflect actual API capabilities
- Remove JSON mode example as response_format is not supported
Reference: https://v0.dev/docs/v0-model-api#request-body
* fix: remove supports_response_schema from v0 models
Remove the supports_response_schema property from all v0 models in the model configuration files as v0 does not support this feature.
Models updated:
- v0/v0-1.0-md
- v0/v0-1.5-md
- v0/v0-1.5-lg
