* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens
Fixes#18520
## Problem
Langfuse integration was not capturing cached tokens from Gemini models.
Gemini returns cached tokens in `usage.prompt_tokens_details.cached_tokens`,
but Langfuse only read from top-level `usage.cache_read_input_tokens`
(which only Anthropic populates).
## Solution
Updated langfuse.py to check both locations:
1. First check top-level cache_read_input_tokens (for Anthropic)
2. Then check prompt_tokens_details.cached_tokens (for Gemini, OpenAI, others)
This ensures all providers' cached tokens are properly reported to Langfuse.
## Changes
- Modified litellm/integrations/langfuse/langfuse.py (lines 742-761)
- Added 3 unit tests in tests/test_litellm/integrations/langfuse/test_gemini_cached_tokens.py
- All existing Langfuse tests still pass (11/11)
## Testing
- test_cached_tokens_extraction: Verifies Gemini cached_tokens extraction
- test_cached_tokens_not_present: Backward compatibility (no cached_tokens)
- test_cached_tokens_is_zero: Edge case when cached_tokens = 0
* Refactor: Extract cache token logic into helper function
Address review feedback from @officer47p
- Created _extract_cache_read_input_tokens() helper function
- Reduces code bloat in _log_langfuse_v2 method
- Improves testability and reusability
- All tests still passing (11/11)
The litellm-database Docker image was missing the libsndfile system
library, which is required by the soundfile Python package for audio
file processing. This caused failures when using audio transcription
endpoints that attempt to calculate audio duration.
This adds libsndfile to the runtime dependencies in Dockerfile.database,
consistent with Dockerfile.alpine which already includes this library.
Invalid routing_strategy values (e.g., "simple" instead of "simple-shuffle") previously failed silently, causing confusing "No deployments available" errors downstream. This change adds upfront validation in routing_strategy_init() to:
- Check if the provided strategy matches valid string values or RoutingStrategy enum
- Raise a clear ValueError listing valid options if invalid
- Fail fast at startup instead of at request time
Fixes behavior reported in #11330 where users had to debug cryptic errors.
Valid strategies: simple-shuffle, least-busy, usage-based-routing, latency-based-routing, cost-based-routing, usage-based-routing-v2
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
- Move BaseBatchesConfig, BaseContainerConfig, BaseEmbeddingConfig, BaseImageEditConfig, BaseImageGenerationConfig, BaseImageVariationConfig, BasePassthroughConfig, BaseRealtimeConfig, BaseRerankConfig, BaseVectorStoreConfig, BaseVectorStoreFilesConfig, BaseVideoConfig to lazy loading
- Move ANTHROPIC_API_ONLY_HEADERS, AnthropicThinkingParam, RerankResponse to lazy loading
- Move ChatCompletionDeltaToolCallChunk, ChatCompletionToolCallChunk, ChatCompletionToolCallFunctionChunk, LiteLLM_Params to lazy loading
- Add type stubs to TYPE_CHECKING block for mypy support
- Add lazy loading handlers in __getattr__ method
- These imports are not used in utils.py runtime code, only in type annotations (safe with from __future__ import annotations)
* refactor(utils): lazy load 15 additional imports to improve import time
- Move Rules, AsyncHTTPHandler, HTTPHandler to lazy loading via __getattr__
- Move get_num_retries_from_retry_policy, reset_retry_policy to lazy loading
- Move get_secret to lazy loading
- Move cached_imports functions (get_coroutine_checker, get_litellm_logging_class, get_set_callbacks) to lazy loading
- Move core_helpers functions (get_litellm_metadata_from_kwargs, map_finish_reason, process_response_headers) to lazy loading
- Move dot_notation_indexing functions (delete_nested_value, is_nested_path) to lazy loading
- Move get_litellm_params functions to lazy loading
- Move _ensure_extra_body_is_safe, get_formatted_prompt, get_response_headers, update_response_metadata to lazy loading
- Move executor to lazy loading
- Move BaseAnthropicMessagesConfig, BaseAudioTranscriptionConfig to lazy loading
- Add type stubs in TYPE_CHECKING block for mypy type checking
- These functions/classes are exported for other modules but not used internally in utils.py, so lazy loading is safe and improves startup performance
* fix(utils): use getattr for Rules and get_coroutine_checker in client decorator
- Update Rules() instantiation in client decorator to use getattr for lazy loading
- Update Rules.has_pre_call_rules() usage in function_setup to use getattr
- Update get_coroutine_checker() usage in client decorator to use getattr
- Fixes NameError: name 'Rules' is not defined error that occurs when Rules is lazy-loaded
* fix(utils): use getattr for get_litellm_logging_class in function_setup
- Update get_litellm_logging_class() usage in function_setup to use getattr for lazy loading
- Fixes NameError: name 'get_litellm_logging_class' is not defined error that occurs when get_litellm_logging_class is lazy-loaded
* fix(utils): use getattr for get_set_callbacks in function_setup
- Update get_set_callbacks() usage in function_setup to use getattr for lazy loading
- Fixes NameError: name 'get_set_callbacks' is not defined error that occurs when get_set_callbacks is lazy-loaded
* fix(utils): use getattr for all lazy-loaded imports in utils.py
- Update update_response_metadata (4 occurrences) to use getattr
- Update executor.submit (1 occurrence) to use getattr
- Update get_num_retries_from_retry_policy (2 occurrences) to use getattr
- Update reset_retry_policy (2 occurrences) to use getattr
- Update is_nested_path and delete_nested_value (1 occurrence each) to use getattr
- Update _ensure_extra_body_is_safe (1 occurrence) to use getattr
Fixes NameError errors that occur when these functions/classes are lazy-loaded but used directly in utils.py
* fix(utils): use getattr for _get_base_model_from_litellm_call_metadata in _get_base_model_from_metadata
- Update _get_base_model_from_litellm_call_metadata usage to use getattr for lazy loading
- Fixes NameError: name '_get_base_model_from_litellm_call_metadata' is not defined
* fix(utils): use getattr for second _get_base_model_from_litellm_call_metadata usage
- Fix the second occurrence of _get_base_model_from_litellm_call_metadata on line 7052
- Both occurrences in _get_base_model_from_metadata now use getattr for lazy loading
* fix(utils): fix indentation in _get_base_model_from_metadata function
* fix(utils): use getattr for get_litellm_metadata_from_kwargs in _get_litellm_params
- Update get_litellm_metadata_from_kwargs usage to use getattr for lazy loading
- Fixes NameError: name 'get_litellm_metadata_from_kwargs' is not defined
* fix(utils): fix syntax error in get_litellm_metadata_from_kwargs fix
- Move getattr call before cast statement to fix syntax error
* refactor(utils): lazy load redact_messages imports to improve import time
- Move LiteLLMLoggingObject and redact_message_input_output_from_logging to lazy loading via __getattr__
- Add type stubs in TYPE_CHECKING block for mypy type checking
- These are only used in type annotations (with from __future__ import annotations), so lazy loading works correctly
This reduces import time by deferring the redact_messages module import until these are actually accessed.
* refactor(utils): lazy load CustomStreamWrapper to improve import time
- Move CustomStreamWrapper from streaming_handler to lazy loading via __getattr__
- Add type stub in TYPE_CHECKING block for mypy type checking
- CustomStreamWrapper is not used internally in utils.py, only exported for other modules
This reduces import time by deferring the streaming_handler module import until CustomStreamWrapper is actually accessed.
* refactor(utils): lazy load BaseGoogleGenAIGenerateContentConfig to improve import time
- Move BaseGoogleGenAIGenerateContentConfig from google_genai.transformation to lazy loading via __getattr__
- Add type stub in TYPE_CHECKING block for mypy type checking
- BaseGoogleGenAIGenerateContentConfig is only used in type annotations (with from __future__ import annotations), so lazy loading works correctly
This reduces import time by deferring the google_genai.transformation module import until BaseGoogleGenAIGenerateContentConfig is actually accessed.
* refactor(utils): lazy load BaseOCRConfig, BaseSearchConfig, and BaseTextToSpeechConfig
- Move BaseOCRConfig, BaseSearchConfig, and BaseTextToSpeechConfig to lazy loading via __getattr__
- Add type stubs in TYPE_CHECKING block for mypy type checking
- These config classes are only used in quoted type annotations (forward references), so lazy loading works correctly
This reduces import time by deferring the transformation module imports until these config classes are actually accessed.
* refactor(utils): lazy load BedrockModelInfo, CohereModelInfo, and MistralOCRConfig
- Move BedrockModelInfo, CohereModelInfo, and MistralOCRConfig to lazy loading via __getattr__
- Add type stubs in TYPE_CHECKING block for mypy type checking
- Update internal usages to use getattr pattern for accessing lazy-loaded classes
- These provider-specific model info classes are only used in specific code paths, so lazy loading reduces initial import time
This reduces import time by deferring the bedrock, cohere, and mistral module imports until these classes are actually accessed.
* fix(utils): remove duplicate MistralOCRConfig import in TYPE_CHECKING block
* refactor(utils): lazy load heavy imports to improve import time
- Move BaseVectorStore, CredentialAccessor, and exception_mapping_utils imports to lazy loading via __getattr__
- Add _get_utils_globals() helper function following pattern from _lazy_imports.py
- Refactor __getattr__ to use consistent caching pattern matching __init__.py
- Update load_credentials_from_list to use lazy-loaded CredentialAccessor
This reduces import time and memory usage by only loading these modules when they're actually accessed, not during module import.
* refactor(utils): lazy load additional heavy imports to improve import time
- Move get_llm_provider, _is_non_openai_azure_model to lazy loading
- Move get_supported_openai_params to lazy loading
- Move convert_dict_to_response functions (LiteLLMResponseObjectHandler, convert_to_model_response_object, etc.) to lazy loading
- Move get_api_base and ResponseMetadata to lazy loading
- Move _parse_content_for_reasoning to lazy loading
- Update all internal usages to access via getattr(sys.modules[__name__], ...)
This reduces import time and memory usage by only loading these modules when they're actually accessed, not during module import.
* fix(utils): suppress PLR0915 linter warning for __getattr__ function
The __getattr__ function intentionally has many statements to handle
multiple lazy-loaded imports. Add noqa comment to suppress the warning.
* fix(utils): add type stubs for lazy-loaded functions in TYPE_CHECKING block
Add type imports and declarations in TYPE_CHECKING block to help mypy
understand the types of lazy-loaded functions accessed via __getattr__.
This follows the same pattern used in __init__.py for lazy-loaded items.
Fix metric name inconsistency for litellm_remaining_requests_metric
and litellm_remaining_tokens_metric. The factory received names
without the _metric suffix, causing _is_metric_enabled to fail when
users configured these metrics in prometheus_metrics_config.
Fixes#18221
Signed-off-by: majiayu000 <1835304752@qq.com>
Fix Ollama_chatException "illegal base64 data at input byte 4" error
when using images with ollama_chat provider. Ollama expects pure base64
data, not the full data URL format (data:image/png;base64,...).
Fixes#18338
Signed-off-by: majiayu000 <1835304752@qq.com>
Instead of limiting grpcio < 1.68.0, specifically exclude the versions
affected by the reconnect bug, and allow installation with either
older or newer versions.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
Add support for Z.AI GLM-4.7, latest flagship model with enhanced reasoning capabilities.
Changes:
- Add zai/glm-4.7 to model pricing with /bin/bash.60/M input, .20/M output
- Add cached input pricing (/bin/bash.11/M) for GLM-4.7
- Add supports_reasoning flag to enable thinking parameter
- Update ZAIChatConfig to support thinking parameter for models with reasoning
- Update documentation with GLM-4.7 as latest flagship model
- Add cached input column to pricing table (GLM-4.7 only)
- Add tests for GLM-4.7 reasoning support and cost calculation
- Update all examples to use GLM-4.7
Model specifications:
- Context: 200K input, 128K output
- Supports: reasoning, function calling, tool choice, prompt caching
- Pricing: Same as GLM-4.6 with cache support
See: https://docs.z.ai/guides/llm/glm-4.7
Add the output_text convenience property to ResponsesAPIResponse that
aggregates all output_text items from the output list, matching the
OpenAI SDK's Response.output_text behavior.
The property iterates through output items, collects text content from
message-type outputs, and returns them concatenated into a single
string. Returns empty string if no output_text content exists.
Handles both dict and Pydantic model access patterns for compatibility
with different output formats.
Fixes#18470🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: yurekami <yurekami@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Some OpenAI-compatible providers (e.g., Apertis) return empty error
objects even on successful responses. The previous check only verified
that error was not None, causing spurious APIErrors.
Now the code checks if the error object contains meaningful data:
- For dict errors: non-empty message OR non-null code
- For string errors: non-empty string
- Other truthy values are still treated as errors
Fixes#18407🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: yurekami <yurekami@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Fixes#18599
When OpenAI models (gpt-5-nano, o1-*, o3-*) and other providers return
reasoning_tokens in completion_tokens_details but don't provide text_tokens,
LiteLLM was incorrectly calculating costs using only reasoning_tokens,
ignoring the remaining completion tokens.
Changes:
- Modified generic_cost_per_token() in llm_cost_calc/utils.py to calculate
text_tokens as: completion_tokens - reasoning_tokens - audio_tokens - image_tokens
when text_tokens is not explicitly provided
- Added comprehensive test case test_reasoning_tokens_without_text_tokens_gpt5_nano()
to verify all completion_tokens are billed correctly
Example:
- completion_tokens: 977
- reasoning_tokens: 768
- Before: only 768 tokens billed (21% less)
- After: all 977 tokens billed correctly
Affected models:
- OpenAI: gpt-5-nano, o1-*, o3-*
- Perplexity: sonar-reasoning*
- Any model returning reasoning_tokens without text_tokens
