* fix(custom_guardrail.py): initial logic for model level guardrails
* feat(custom_guardrail.py): working pre call guardrails
* fix(custom_guardrails.py): check if custom guardrails set before running event hook
* test(test_custom_guardrail.py): add unit tests for async pre call deployment hook on custom guardrail
* feat(custom_guardrail.py): add post call processing support for guardrails
allows model based guardrails to run on the post call event for that model only
* fix(utils.py): only run if call type is in enum
* test: update unit tests to work
* feat(key_management_endpoints.py): Support new 'key_type' field
allow user to specify if key should be 'management' or 'llm api' key
Security fix
* test(test_route_checks.py): add unit tests
* fix(create_key_button.tsx): add ui component to select key type
allows specifying if key can call llm api vs. management routes
* feat(create_key_button.tsx): add specifying key type to ui
* fix(route_checks.py): add sensitive data masker for user id on not allowed error message
prevent leaking sensitive information
* feat(litellm_pre_call_utils.py): add num_retries to litellm data for backend call
allow user to pass in num retries via request headers
* test(test_litellm_pre_call_utils.py): add unit test
* docs(request_headers.md): document new request header
* fix(common_daily_activity.py): show spend breakdown by model group
Partial fix for https://github.com/BerriAI/litellm/issues/12887
* feat(new_usage.tsx): new tab switcher for viewing usage by model group vs. received model
Closes https://github.com/BerriAI/litellm/issues/12887
* fix(main.py): fix async retryer
Fixes https://github.com/BerriAI/litellm/issues/12830
* fix(forward_clientside_headers_by_model_group.py): filter out 'content-type' from forwardable headers
clientside content-type != proxy content type, can cause requests to hang
This aligns the proxy experience with other models that think
automatically (e.g. Deepseek R1 and grok3). It does so by setting
the necessary request input to return thinking, but not specifying
a budget or effort (thus defaulting to the internal automatic level).
* fix(gpt_transformation.py): remove 'cache_control' flag for openai/openai-compatible calls
Fixes https://github.com/BerriAI/litellm/issues/12787
* fix(openrouter/chat/transformation.py): allow passing openrouter cache control flag for claude models
* fix(gpt_transformation.py): fix import
* fix: fix adding tools
* fix(main.py): fix async retryer
Fixes https://github.com/BerriAI/litellm/issues/12830
* fix(forward_clientside_headers_by_model_group.py): filter out 'content-type' from forwardable headers
clientside content-type != proxy content type, can cause requests to hang
* test(tests/): update tests
* fix(team_endpoints.py): always remove team member budget from updated_kv
this is not a field for the litellm team table
Prevents startup issue
* test(test_team_endpoints.py): add unit test to ensure 'team_member_budget' is never in update to table - separate logic
* refactor: cleanup
When using Model Armor guardrail with explicit project_id in config,
the project_id was being overwritten to None due to incorrect
initialization order between ModelArmorGuardrail and VertexBase parent class.
This fix ensures that user-provided project_id is preserved by initializing
parent classes before setting instance attributes.
Fixes#12757
* feat: initial commit for forwarding client headers by model group
* fix(router.py): support new forwarclientsideheadersbymodelgroup class
enables headers to be forwarded to backend model, by model group
* fix(proxy_server.py): load in model group settings from config correctly
* refactor(litellm_pre_call_utils.py): litellm_pre_call_utils.py
introduce new 'secret_fields' field
includes raw request headers (not the sanitized ones used for logging) - needed to support forwarding clientside headers to llm api
* feat(router.py): log the deployment model name as well
allows wildcard models to support forward_client_headers_to_llm_api
* test(test_router.py): add more unit testing
* feat(router.py): specify the model group alias in metadata kwargs
allows usage for internal routing logic
* fix: fix ruff check errors
* fix(router.py): refactor to cleanup optional pre-call checks
* fix: fix ruff check
* test: add missing unit test
* fix(prompt_templates/factory.py): handle anthropic cache control on individual tool results
Fixes issue where cache control on individual tool result was being ignored
* test(test_vertex_And_google_ai_studio_gemini.py): initial unit test covering translation for grounding metadata on streaming chunk
* fix(vertex_and_google_ai_studio.py): ensure grounding metadata is preserved on streaming
Closes https://github.com/BerriAI/litellm/issues/10237
* fix(core_helpers.py): include usage in expected openai keys
* fix(lowest_latency.py): Handle ZeroDivisionError with zero completion tokens (#12641)
Fixes ZeroDivisionError when LLM responses have zero completion tokens, which can
occur with Gemini models on very long contexts that only use tool calls.
Changes:
- Add check for completion_tokens > 0 before division in log_success_event
- Handle both timedelta and float types for response times (supporting both time.time() and datetime)
- Apply fix to both occurrences of the division operation in the file
- Add comprehensive tests for zero completion token scenarios
This ensures the lowest latency routing continues to work properly even when
models return responses with no completion tokens.
* test: Move lowest_latency zero tokens test to test_litellm for CI execution
* refactor: Use safe_divide helper to eliminate code duplication
- Added safe_divide utility function to litellm_core_utils.core_helpers
- Handles both timedelta and float types for numerator
- Prevents ZeroDivisionError and negative denominator issues
- Replaced duplicated division logic in lowest_latency.py
- Added comprehensive unit tests for safe_divide function
This improves code quality by reducing duplication and centralizing the
division safety logic in a reusable helper function.
* refactor: Rename to safe_divide_seconds for clarity
- Renamed safe_divide to safe_divide_seconds to better indicate it handles time durations
- Simplified implementation with single-line ternary for seconds conversion
- Updated all references and tests accordingly
The function name now clearly indicates it's specifically for dividing
time durations (in seconds) by a denominator.
* refactor: Simplify safe_divide_seconds to only accept float arguments
- Changed safe_divide_seconds to accept only float parameters for consistency
- Callers now handle timedelta to seconds conversion explicitly
- Removed timedelta-specific test cases
* fix: Remove unused timedelta import
* fix: Remove Union[timedelta, float] type annotations
Since safe_divide_seconds now only accepts floats, we handle the
timedelta conversion explicitly in the code. The type annotations
are no longer needed and can be simplified.
* Revert "fix: Remove Union[timedelta, float] type annotations"
This reverts commit 19c6c62154.
* fix: Clean up implementation
- Remove unnecessary type annotations
- Use safe_divide_seconds utility for zero-division protection
- Handle both timedelta and float types explicitly
- Maintain compatibility with both datetime and time.time() usage
* feat: integrate Google Cloud Model Armor guardrails (LIT-298)
- Add ModelArmorGuardrail class that extends CustomGuardrail and VertexBase
- Support for both pre-call (sanitizeUserPrompt) and post-call (sanitizeModelResponse) sanitization
- Integrate with existing Vertex AI authentication using VertexBase
- Add configuration model for Model Armor in guardrail types
- Register Model Armor in guardrail initializers and registry
- Include comprehensive test suite for Model Armor functionality
- Support content masking for both requests and responses
- Handle streaming responses with content sanitization
This integration allows LiteLLM to use Google Cloud Model Armor API for
content moderation and sanitization, providing similar functionality to
Bedrock Guards but using Google Cloud's security infrastructure.
* fix: remove unused imports flagged by ruff linter
- Remove unused asyncio import at top level (moved to local import where needed)
- Remove unused TextCompletionResponse import
* fix: remove additional unused imports
- Remove unused TextChoices import from line 34
- Remove duplicate asyncio import from line 384
- Replace asyncio.iscoroutine() with hasattr check for __await__
* fix: remove final unused imports
- Remove unused 'import sys' from line 9
- Remove unused 'StreamingChoices' from imports
* fix: remove unused import from model_armor.py
- Remove unused 'import os' from the top of the file
* fix: remove commented-out header from model_armor.py
- Eliminate unnecessary comments at the top of the file to improve code clarity.
* feat(guardrails): Add Model Armor UI support
- Convert model_armor.py to a directory structure with __init__.py for dynamic discovery
- Add get_config_model() method to ModelArmorGuardrail class for UI integration
- Add ui_friendly_name() to ModelArmorConfigModel returning "Google Cloud Model Armor"
- Remove manual registration from guardrail_registry.py to use dynamic discovery
- Model Armor now appears in the UI guardrails dropdown with proper configuration fields
This enables users to configure Model Armor guardrails through the LiteLLM UI interface.
* fix(guardrails): Fix undefined name 'GuardrailConfigModel' in Model Armor
- Import TYPE_CHECKING and GuardrailConfigModel from base module
- Fixes F821 linting error for undefined name in type annotation
- Follows same pattern as other guardrail implementations
* fix(guardrails): Fix Model Armor type errors and config model inheritance
- Create ModelArmorGuardrailConfigModel that properly inherits from GuardrailConfigModel base class
- Move config model to litellm/types/proxy/guardrails/guardrail_hooks/model_armor.py following convention
- Update get_config_model() to return the properly typed config model
- Remove ModelArmorConfigModel from LitellmParams inheritance chain
- Add template_id field to BaseLitellmParams instead
This fixes the mypy type errors and follows the same pattern as other guardrail implementations.
* fix(guardrails): Add missing Model Armor fields to BaseLitellmParams
- Add location, credentials, api_endpoint, and fail_on_error fields
- Fixes mypy errors about missing attributes in LitellmParams
- All Model Armor configuration parameters are now properly defined
* fix: Apply PR review feedback for Model Armor guardrail
- Move test file to tests/test_litellm/ for GitHub Actions
- Extract only last consecutive user messages to avoid context limits
- Use get_content_from_model_response helper for response extraction
- Handle non-ModelResponse types (e.g., TTS) gracefully
- Maintain newline separation for multi-part content
* refactor: Simplify message content extraction in ModelArmorGuardrail
- Removed the custom _extract_content_from_messages method.
- Integrated get_last_user_message helper for improved content extraction.
- Updated test to reflect changes in content formatting.
* refactor: Remove unused import in model_armor.py
- Deleted the unused import of AllMessageValues to clean up the codebase.
* Add unit tests for Model Armor guardrail functionality
- Implement tests for pre-call and post-call hooks, including content sanitization and blocking behavior.
- Validate error handling for API responses and credential management.
- Test streaming responses and handling of list content in user messages.
- Ensure proper assertions for API interactions and response sanitization.
* Add comprehensive test coverage for Model Armor guardrail
- Add test for requests with no messages field
- Add test for empty message content handling
- Add test for system/assistant-only messages
- Add test for fail_on_error=False behavior
- Add test for custom API endpoint configuration
- Add test for dictionary credentials (non-file path)
- Add test for action=NONE response handling
- Add test for missing sanitized_text field fallback
- Add test for non-text response types (TTS/image)
- Add test for auth token refresh behavior
All tests ensure robust edge case handling and proper error management.
* feat: improve Model Armor handling of non-ModelResponse types
- Add debug logging when skipping non-text responses (TTS, images, etc.)
- Improve docstring to clarify behavior for non-text responses
- Add test coverage for non-ModelResponse handling
- Ensure guardrail gracefully skips processing for response types it cannot handle
* build: move build_and_test to use prisma migrate
* fix(guardrails_ai.py): default to guardrail accepting 'llmOutput' as the input param
enables same guardrail to work for pre call and post call
* fix(__init__.py): set default value
* fix(guardrails_ai.py): updates
* fix: fix linting error
