* init guardrails
* init guardrails
* some fixes
* some fixes
* ruff
* some fixes
* some fixes
* some fixes
* some fixes
* some fixes
* some fixes
* docs
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens
Fixes#18520
## Problem
Langfuse integration was not capturing cached tokens from Gemini models.
Gemini returns cached tokens in `usage.prompt_tokens_details.cached_tokens`,
but Langfuse only read from top-level `usage.cache_read_input_tokens`
(which only Anthropic populates).
## Solution
Updated langfuse.py to check both locations:
1. First check top-level cache_read_input_tokens (for Anthropic)
2. Then check prompt_tokens_details.cached_tokens (for Gemini, OpenAI, others)
This ensures all providers' cached tokens are properly reported to Langfuse.
## Changes
- Modified litellm/integrations/langfuse/langfuse.py (lines 742-761)
- Added 3 unit tests in tests/test_litellm/integrations/langfuse/test_gemini_cached_tokens.py
- All existing Langfuse tests still pass (11/11)
## Testing
- test_cached_tokens_extraction: Verifies Gemini cached_tokens extraction
- test_cached_tokens_not_present: Backward compatibility (no cached_tokens)
- test_cached_tokens_is_zero: Edge case when cached_tokens = 0
* Refactor: Extract cache token logic into helper function
Address review feedback from @officer47p
- Created _extract_cache_read_input_tokens() helper function
- Reduces code bloat in _log_langfuse_v2 method
- Improves testability and reusability
- All tests still passing (11/11)
This extends the previous fix to handle capitalized fields across ALL guardrail types,
including Presidio, Azure, Lakera, Bedrock, etc.
Discovery:
- Database investigation revealed the issue affects multiple guardrail types
- Found 4 affected guardrails in staging: 3 Presidio + 1 Azure
- All had default_action: 'Deny' causing the same validation failures
- The initial fix only covered ToolPermissionGuardrailConfigModel
Root Cause (Deeper):
- LitellmParams inherits from 13+ different guardrail config models
- Models use ConfigDict(extra="allow") allowing any field to be set
- Users can set default_action/on_disallowed_action on ANY guardrail type
- Only ToolPermissionGuardrailConfigModel was validating these fields
Solution:
- Added field validators to LitellmParams class (parent of all guardrails)
- Validators run for ALL guardrail types: Presidio, Azure, Bedrock, Lakera, etc.
- Added comprehensive tests covering multiple guardrail types
Changes:
- litellm/types/guardrails.py:
* Added @field_validator for default_action in LitellmParams
* Added @field_validator for on_disallowed_action in LitellmParams
* Added normalization in LitellmParams.__init__ as backup
* Imported field_validator from pydantic
- tests/test_litellm/types/test_guardrails_case_normalization.py:
* New test file with 7 tests covering multiple guardrail types
* Tests verify Presidio, Azure, Tool Permission, Lakera, Bedrock
* All tests passing
Impact:
- Previous fix: Only tool_permission guardrails protected
- This fix: ALL guardrail types now protected (13+ types)
- Handles both new writes and existing database records
- Tested against actual database with Presidio/Azure guardrails
Testing:
- 7 new cross-guardrail tests (all passing)
- 27 existing tool_permission tests (all passing)
- Verified fix works for real database records
Invalid routing_strategy values (e.g., "simple" instead of "simple-shuffle") previously failed silently, causing confusing "No deployments available" errors downstream. This change adds upfront validation in routing_strategy_init() to:
- Check if the provided strategy matches valid string values or RoutingStrategy enum
- Raise a clear ValueError listing valid options if invalid
- Fail fast at startup instead of at request time
Fixes behavior reported in #11330 where users had to debug cryptic errors.
Valid strategies: simple-shuffle, least-busy, usage-based-routing, latency-based-routing, cost-based-routing, usage-based-routing-v2
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
This fixes a critical issue where capitalized values in tool_permission guardrail
configurations (e.g., "Deny" instead of "deny") caused Pydantic validation errors
during proxy startup, leading to repeated initialization failures and latency issues.
Problem:
- Users could save guardrails with capitalized values through UI/API
- Data was written to database without validation (e.g., default_action: "Deny")
- On proxy startup, loading from DB triggered strict Pydantic validation
- ValidationError caused guardrail initialization to fail in a retry loop
- This resulted in startup delays and repeated error logging
Root Cause:
- Write path had no case normalization
- Read path enforced strict lowercase Literal validation
- Asymmetry between write and read caused latent data corruption
Solution:
Added field validators to normalize case before Pydantic validation:
1. ToolPermissionRule.decision ("allow"/"deny")
- Normalizes decision field in rules array
2. ToolPermissionGuardrailConfigModel.default_action ("allow"/"deny")
- Normalizes default fallback action
3. ToolPermissionGuardrailConfigModel.on_disallowed_action ("block"/"rewrite")
- Normalizes disallowed tool behavior
4. ToolPermissionGuardrail.__init__ normalization
- Defensive normalization for direct instantiation
- Ensures normalization regardless of code path
Impact:
- Prevents validation errors during guardrail initialization
- Eliminates startup retry loops and latency issues
- Handles existing database records with capitalized values
- Accepts case-insensitive input from all sources (UI, API, direct calls)
- Fully backward compatible with existing lowercase configurations
Testing:
- Added 3 comprehensive tests for case-insensitive handling
- All 27 existing tests still pass
- Tests verify normalization across all affected fields
Files Changed:
- litellm/types/proxy/guardrails/guardrail_hooks/tool_permission.py
Added @field_validator decorators for case normalization
- litellm/proxy/guardrails/guardrail_hooks/tool_permission.py
Added runtime normalization in __init__ method
- tests/test_litellm/proxy/guardrails/guardrail_hooks/test_tool_permission.py
Added case-insensitive validation tests
* refactor: migrate utils.py lazy imports to registry pattern
- Refactored utils.py __getattr__ to use cached registry pattern (similar to __init__.py)
- Added UTILS_MODULE_NAMES tuple and _UTILS_MODULE_IMPORT_MAP to _lazy_imports_registry.py
- Added _get_utils_globals() helper function to _lazy_imports.py
- Added _lazy_import_utils_module() handler function for utils module lazy imports
- Updated _get_lazy_import_registry() to include utils module lazy imports
- Removed redundant _get_utils_globals() from utils.py (now in _lazy_imports.py)
- Added comprehensive tests for utils module lazy imports in test_lazy_imports.py
This refactoring:
- Reduces code duplication (from 670+ lines to ~10 lines in __getattr__)
- Improves maintainability (new lazy imports just need registry entry)
- Maintains consistency with __init__.py lazy import pattern
- All existing functionality preserved and tested
* Fix NameError: get_coroutine_checker not defined in check_coroutine function
* Fix lazy loading for get_coroutine_checker in function_setup and check_coroutine
- Add lazy loading for get_coroutine_checker at start of function_setup to ensure all calls use lazy-loaded version
- Fix check_coroutine function to use lazy loading pattern via getattr
- All direct calls to get_coroutine_checker() now properly use lazy import mechanism
- Keep **kwargs approach (no exec()) for security
- Integrate path traversal validation from main branch
- Add URL encoding for path parameters
- Merge both test suites (edge cases + security tests)
- All 14 tests passing
Fix metric name inconsistency for litellm_remaining_requests_metric
and litellm_remaining_tokens_metric. The factory received names
without the _metric suffix, causing _is_metric_enabled to fail when
users configured these metrics in prometheus_metrics_config.
Fixes#18221
Signed-off-by: majiayu000 <1835304752@qq.com>
Fix Ollama_chatException "illegal base64 data at input byte 4" error
when using images with ollama_chat provider. Ollama expects pure base64
data, not the full data URL format (data:image/png;base64,...).
Fixes#18338
Signed-off-by: majiayu000 <1835304752@qq.com>
Add support for Z.AI GLM-4.7, latest flagship model with enhanced reasoning capabilities.
Changes:
- Add zai/glm-4.7 to model pricing with /bin/bash.60/M input, .20/M output
- Add cached input pricing (/bin/bash.11/M) for GLM-4.7
- Add supports_reasoning flag to enable thinking parameter
- Update ZAIChatConfig to support thinking parameter for models with reasoning
- Update documentation with GLM-4.7 as latest flagship model
- Add cached input column to pricing table (GLM-4.7 only)
- Add tests for GLM-4.7 reasoning support and cost calculation
- Update all examples to use GLM-4.7
Model specifications:
- Context: 200K input, 128K output
- Supports: reasoning, function calling, tool choice, prompt caching
- Pricing: Same as GLM-4.6 with cache support
See: https://docs.z.ai/guides/llm/glm-4.7
Add the output_text convenience property to ResponsesAPIResponse that
aggregates all output_text items from the output list, matching the
OpenAI SDK's Response.output_text behavior.
The property iterates through output items, collects text content from
message-type outputs, and returns them concatenated into a single
string. Returns empty string if no output_text content exists.
Handles both dict and Pydantic model access patterns for compatibility
with different output formats.
Fixes#18470🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: yurekami <yurekami@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Some OpenAI-compatible providers (e.g., Apertis) return empty error
objects even on successful responses. The previous check only verified
that error was not None, causing spurious APIErrors.
Now the code checks if the error object contains meaningful data:
- For dict errors: non-empty message OR non-null code
- For string errors: non-empty string
- Other truthy values are still treated as errors
Fixes#18407🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: yurekami <yurekami@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Fixes#18599
When OpenAI models (gpt-5-nano, o1-*, o3-*) and other providers return
reasoning_tokens in completion_tokens_details but don't provide text_tokens,
LiteLLM was incorrectly calculating costs using only reasoning_tokens,
ignoring the remaining completion tokens.
Changes:
- Modified generic_cost_per_token() in llm_cost_calc/utils.py to calculate
text_tokens as: completion_tokens - reasoning_tokens - audio_tokens - image_tokens
when text_tokens is not explicitly provided
- Added comprehensive test case test_reasoning_tokens_without_text_tokens_gpt5_nano()
to verify all completion_tokens are billed correctly
Example:
- completion_tokens: 977
- reasoning_tokens: 768
- Before: only 768 tokens billed (21% less)
- After: all 977 tokens billed correctly
Affected models:
- OpenAI: gpt-5-nano, o1-*, o3-*
- Perplexity: sonar-reasoning*
- Any model returning reasoning_tokens without text_tokens
- /v2/model/info now returns {"data": []} when llm_router is None or model_list is empty
- /model_group/info now returns {"data": []} when llm_model_list is None or empty
- Fixes UI crash on fresh installs with STORE_MODEL_IN_DB=True
- Added 4 unit tests for empty model list scenarios
Adds log_format parameter supporting json_array (default), ndjson, and single formats. NDJSON format enables webhook integrations like Sumo Logic to parse individual log records at ingest time. Defaults to json_array for backward compatibility.
