Resolves issue #17910 where Amazon Nova models (like amazon.nova-pro-v1:0)
were incorrectly identified as Amazon Titan models, causing requests to
use textGenerationConfig instead of inferenceConfig.
The fix moves the "nova" check before the initial provider check on the
split model name. This ensures that models containing "nova" (like
amazon.nova-pro-v1:0 or amazon.nova-2-lite-v1:0) are correctly identified
as Nova models, rather than matching "amazon" first.
* feat: Add Prometheus metrics for request queue time and guardrails
- Add litellm_request_queue_time_seconds metric to track time from request arrival to processing start
- Add guardrail metrics: latency, errors_total, and requests_total counters
- Track arrival time in litellm_pre_call_utils.py
- Calculate queue time in common_request_processing.py
- Record guardrail metrics in pre_call_hook and during_call_hook
- Add comprehensive unit tests for all new metrics
Fixes#17863
* perf: optimize timing calls for queue time and guardrail metrics
* fix: resolve conflicts in utils.py - integrate Prometheus metrics with guardrail load balancing
* feat: prisma migrate deploy with lock
Author: Mini Jeong <mini.jeong@navercorp.com>
* fix: use redis cache from proxy server
Author: Mini Jeong <mini.jeong@navercorp.com>
* fix: add type checks and fix unit tests for migration lock
- Add DATABASE_URL validation in _create_baseline_migration() and _resolve_all_migrations()
- Fix MyPy type errors by adding None checks before using database_url in subprocess calls
- Add _resolve_all_migrations mock to failing unit tests to prevent filesystem errors
- Apply Black formatting to modified files
Fixes:
- MyPy type errors: database_url could be None when passed to subprocess
- Unit test failures: _resolve_all_migrations tried to create directories in read-only /test path
* fix: resolve MyPy type error in vertex_ai vertex_llm_base
Fix MyPy type checking error where vertex_api_version parameter type
was incompatible with function signature expectation.
* fix: Return 403 exception when calling GET responses api
* fix: added new step into rotate master key function for processing credentials table
* Add redisvl in requirements.txt
* fix: fixed the issue of handling root paths when processing Discovery protected resource metadata and authorization server metadata URLs.
* fix: added additional grant type into oauth_authorization_server response for fixing mcp auth register bad request issue
* fix: added RFC RECOMMENDED property(scopes_supported) to protected resource and authorization server metadata
* fix: removed initialize the tool name to MCP server name mapping(oauth2) on startup for avoiding 401 error
* fix: upgraded mcp sdk depency version for fixing ClosedResourceError
* Use already configured opentelemetry providers
Users that instrument using opentelemetry-instrument can now setup exporters as per their environment.
* Handle all protocols for all telemetry
* Add more tests
* feat(mcp): parallelize tool fetching from multiple MCP servers (#18627)
* feat(mcp): parallelize tool fetching from multiple MCP servers
Replace sequential tool fetching with asyncio.gather() to reduce
client timeouts when using multiple MCP servers.
Changes:
- mcp_server_manager.py: list_tools() now fetches tools in parallel
- server.py: _get_tools_from_mcp_servers() now fetches tools in parallel
Real-world impact (7 MCP servers example):
- Sequential: ~4.5+ seconds (exceeds typical 5-second client timeouts)
- Parallel: ~1.2 seconds (max of all servers)
Fixes#18626
* fix: copy oauth2_headers to avoid shared dict mutation in parallel tasks
* feat: add display_name, model_vendor, and model_version metadata
* added the option of adding langsmith tenant id in the env (#18623)
* fix(router): Validate routing_strategy at startup to fail fast with helpful error. (#18624)
Invalid routing_strategy values (e.g., "simple" instead of "simple-shuffle") previously failed silently, causing confusing "No deployments available" errors downstream. This change adds upfront validation in routing_strategy_init() to:
- Check if the provided strategy matches valid string values or RoutingStrategy enum
- Raise a clear ValueError listing valid options if invalid
- Fail fast at startup instead of at request time
Fixes behavior reported in #11330 where users had to debug cryptic errors.
Valid strategies: simple-shuffle, least-busy, usage-based-routing, latency-based-routing, cost-based-routing, usage-based-routing-v2
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
* Add libsndfile to database Docker image for audio processing (#18612)
The litellm-database Docker image was missing the libsndfile system
library, which is required by the soundfile Python package for audio
file processing. This caused failures when using audio transcription
endpoints that attempt to calculate audio duration.
This adds libsndfile to the runtime dependencies in Dockerfile.database,
consistent with Dockerfile.alpine which already includes this library.
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens (#18614)
* Fix: Map Gemini cached_tokens to Langfuse cache_read_input_tokens
Fixes#18520
## Problem
Langfuse integration was not capturing cached tokens from Gemini models.
Gemini returns cached tokens in `usage.prompt_tokens_details.cached_tokens`,
but Langfuse only read from top-level `usage.cache_read_input_tokens`
(which only Anthropic populates).
## Solution
Updated langfuse.py to check both locations:
1. First check top-level cache_read_input_tokens (for Anthropic)
2. Then check prompt_tokens_details.cached_tokens (for Gemini, OpenAI, others)
This ensures all providers' cached tokens are properly reported to Langfuse.
## Changes
- Modified litellm/integrations/langfuse/langfuse.py (lines 742-761)
- Added 3 unit tests in tests/test_litellm/integrations/langfuse/test_gemini_cached_tokens.py
- All existing Langfuse tests still pass (11/11)
## Testing
- test_cached_tokens_extraction: Verifies Gemini cached_tokens extraction
- test_cached_tokens_not_present: Backward compatibility (no cached_tokens)
- test_cached_tokens_is_zero: Edge case when cached_tokens = 0
* Refactor: Extract cache token logic into helper function
Address review feedback from @officer47p
- Created _extract_cache_read_input_tokens() helper function
- Reduces code bloat in _log_langfuse_v2 method
- Improves testability and reusability
- All tests still passing (11/11)
* Adding Role Mappings
* Fixing Edit SSO Settings Modal
* feat: add user_mcp_management_mode for view_all visibility
* Fixing tests
* fix: missing mcp_allow_all_ui.png
* docs: add user_mcp_management_mode
* Align responses API streaming hooks with chat pipeline
* Clarify responses API streaming context
* Address review comments
* feat: Add GigaChat provider support (#18564)
* feat: Add GigaChat provider support
Add native support for GigaChat API (Sber AI, Russia's leading LLM).
Supported features:
- Chat completions (sync/async)
- Streaming (sync/async)
- Function calling / Tools
- Structured output via JSON schema (emulated through function calls)
- Image input (base64 and URL)
- Embeddings
Closes#18515
* fix: resolve mypy type errors in GigaChat handler
- Fix _prepare_file_data return type (use 3-tuple for cleaner type flow)
- Add type annotations for lists in _process_content_parts methods
- Add type annotations in _collapse_user_messages
- Use ChatCompletionToolCallChunk for proper tool_use typing
- Add type: ignore[override] for astreaming async generator
* refactor(gigachat): migrate to BaseConfig pattern
* fix: remove unused imports
* fix: resolve mypy type errors
* fix: mypy type errors
* refactor: address review feedback for GigaChat provider
- Remove singleton pattern, reuse litellm HTTPHandler
- Move constants/errors to transformation files, delete common_utils.py
- Add models to model_prices_and_context_window.json
- Fix ssl_verify not passed to HTTP client for embeddings
* docs: update GigaChat documentation with ssl_verify requirement
* Revert "Add redisvl in requirements.txt"
* Put reasoning summary behind feat flag
* fix: model eol
* fix: anthropic claude-3-opus-20240229 EOL
* Revert "fix: model eol"
This reverts commit 5aa1665d79.
* Fix: ImportError: qualifire package is required for QualifireGuardrail. Install it with: pip install qualifire
* fix: test_secret_manager_failure_does_not_block_email
* fix: test_update_ui_settings_allowlisted_value
* fix: test_aaamodel_prices_and_context_window_json_is_valid
* fix: test_all_models_have_display_name
* fix: async def test_bedrock_apply_guardrail_blocked()
* fix: test_databricks_embeddings[True]
* fix:test_anthropic_beta_header
* fix:test_api_error_handling
* fix:mypy mcp management
* Revert "feat(model_cost): add display_name, model_vendor, and model_version metadata to model entries"
* [Feat] New API Endpoint - Responses API (v1/responses/compact) (#18697)
* init transform_compact_response_api_request
* init acompact_responses
* init async_compact_response_api_handler in llm http handler
* init transform_compact_response_api_request for openai
* init acompact_responses
* fix acompact_responses
* add OAI Compact API
* docs responses API Compact
* code qa checks
* test_openai_compact_responses_api
* fix mypy linting
* fix: remove display name
* Add the LITELLM_REASONING_AUTO_SUMMARY in doc
* fix model map
* [UI] - Feat add request provider form on UI (#18704)
* add request provider form
* fix link to github
* add button
* fix link
* fix(streaming): normalize status code extraction to prevent 4xx errors from triggering mid-stream fallback (#18698)
在流式处理错误时，添加状态码标准化逻辑，确保 4xx 客户端错误直接抛出而不是被包装成 MidStreamFallbackError。
- 新增 _normalize_status_code 函数用于从异常对象提取状态码
- 优先从异常的 status_code 属性获取，其次从 response.status_code 获取
- 当映射异常或原始异常的状态码在 400-499 范围内时，直接抛出映射异常
- 添加单元测试验证 Vertex AI 400 错误正确抛出为 BadRequestError
- 确保流式处理中的客户端错误能够正确传播，而不会触发回退机制
---------
Co-authored-by: Eric84626 <lixiannan@gmail.com>
Co-authored-by: Eric84626 <97266539+Eric84626@users.noreply.github.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: mangabits <1457532+mangabits@users.noreply.github.com>
Co-authored-by: Costa Tsaousis <costa@tsaousis.gr>
Co-authored-by: Nik <nikolas.garza5@gmail.com>
Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com>
Co-authored-by: FlibbertyGibbitz <seth@evenkeelconsultingllc.com>
Co-authored-by: Flibbert E. Gibbitz <flibbertygibbitz@runelabs.ai>
Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com>
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Co-authored-by: Yuta Saito <uc4w6c@bma.biglobe.ne.jp>
Co-authored-by: LingXuanYin <3546599908@qq.com>
Co-authored-by: YutaSaito <36355491+uc4w6c@users.noreply.github.com>
Co-authored-by: 0717376 <103773680+0717376@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Kris Xia <xiajiayi0506@gmail.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Calculate `total_tokens` in usage data in Response manually if:
- `total_tokens` is missing
- `total_tokens` can be calculated from input and output tokens
Run the test for this feature with:
`poetry run pytest tests/test_litellm/responses/test_responses_utils.py -k "test_transform_response_api_usage_calculates_total_from_input_and_output_tokens_if_available" -v`
