basedpyright counts a named capture after an exhaustive match under
reportUnnecessaryComparison, which pushed the merged tree one over the
budget; the wildcard case with assert_never on the bound subject is the
shape the rest of the codebase uses
litellm_settings keys are set on the litellm module with no allowlist, so a typo
in vector_store_search_failure_mode reached assert_never and turned every
vector-store request into a 500. Validate the configured value and fall back to
the permissive default with a warning naming the supported modes.
A vector store search that fails is swallowed by the pre-call hook, so the
request goes to the model with an un-augmented prompt and the caller gets a
200 answering from the model's own knowledge with no way to tell the
knowledge base was skipped.
Failed searches now ride the same channel their successes already use: a
vector_store_search_failures entry on provider_specific_fields naming the
store id, provider, and error. That is additive and always on. For callers
who would rather fail than answer ungrounded, litellm_settings
vector_store_search_failure_mode: error raises VectorStoreSearchError (400)
instead; the default stays annotate, today's permissive behavior.
The hook's outer catch-all also now names the requested vector store ids in
its log line, and only wraps the augmentation itself, so the fail-closed
raise is not swallowed by it.
One unreachable vector store used to wipe out every store's context on a
chat completion carrying vector_store_ids: the search raised, the blanket
handler returned the original messages, and the request answered with no
retrieved context at all. Each store's search now has its own handler that
warns with the vector store id and moves on to the next store.
The same loop appended every store's results to the original messages
instead of the running copy, so with two healthy stores only the last one
reached the model. It now chains through modified_messages.
The Router is injected through a ProxyRuntime protocol instead of an
in-function litellm.proxy.proxy_server import, so the hook's routing can
be driven in tests without touching proxy globals.
Narrows reportAny / reportExplicitAny hot spots in provider transformations,
proxy endpoints, integrations and secret managers by introducing TypedDicts,
Protocols and object-typed boundaries instead of Any, then ratchets the
budget ceilings down to match.
reportAny 14765 -> 14076, reportExplicitAny 4493 -> 4128, ANN401 387 -> 307
About 35,000 fixes ruff marks safe across 32 rules (UP006/UP045/UP007
modern annotations, UP032 f-strings, SIM114/SIM118, RET501, and
friends), removal of the 1,296 typing imports the rewrite orphaned, and
hand fixes for what the fixers could not see: five star-import
freeloaders of typing names, two F823 late-import annotations, the
/get/config/list introspection crash on types.UnionType, redundant
function-local RoleMappings imports in ui_sso.py that shadowed the
module-level name once the annotation lost its quotes, and one FURB168
tautology.
B009/B010/PIE804/RUF019 are excluded on purpose: their safe fixes
rewrite getattr/setattr/**-splat/key-in-dict escape hatches into forms
basedpyright then rejects (283 new errors measured), so their budgets
stay at base values.
ruff-strict-budget.json drops by 39,579 this commit (39,968 across the
branch) with 28 rules at an actual 0 and 9 more sharply down.
type-discipline-budget.json ratchets LIT002/LIT006/LIT009 down; LIT001
moves to the now-honest total: the checker matches the spelling `set`
but not the alias `Set`, so the 160 typing.Set annotations rewritten to
set[...] were always mutable-set annotations and only now count.
The repo linted at 120 (E501, isort) but ran ruff format at 88 via a
--line-length 88 override in the Makefile and CI, leaving the formatter
and the linter disagreeing on wrap width. Drop the override so ruff.toml's
line-length = 120 is the single source of truth and reformat the tree to
match.
* Fix duplicate imports in SAP embedding transformation
* fix: add missing prompt_spec parameter to HumanloopLogger.get_chat_completion_prompt
- Add prompt_spec: Optional[PromptSpec] = None parameter to match base class signature
- Import PromptSpec from litellm.types.prompts.init_prompts
- Pass prompt_spec to super().get_chat_completion_prompt() call
- Fixes mypy type error: Signature incompatible with supertype CustomLogger
* fix: add missing parameters to AnthropicCacheControlHook.async_get_chat_completion_prompt
- Add ignore_prompt_manager_model and ignore_prompt_manager_optional_params parameters
- Change litellm_logging_obj type from Any to LiteLLMLoggingObj using TYPE_CHECKING pattern
- Pass all parameters including prompt_spec to get_chat_completion_prompt call
- Fixes mypy type errors: Signature incompatible with supertype CustomLogger and PromptManagementBase
* fix: add missing parameters to DotpromptManager.async_get_chat_completion_prompt
- Add ignore_prompt_manager_model and ignore_prompt_manager_optional_params parameters
- Change litellm_logging_obj type from Any to LiteLLMLoggingObj using TYPE_CHECKING pattern
- Pass all parameters including ignore flags to PromptManagementBase.async_get_chat_completion_prompt
- Fixes mypy type errors: Signature incompatible with supertype CustomLogger and PromptManagementBase
* fix: document envs
* fix: add missing parameters to LangfusePromptManagement.async_get_chat_completion_prompt
- Add ignore_prompt_manager_model and ignore_prompt_manager_optional_params parameters
- Pass all parameters including prompt_spec and ignore flags to get_chat_completion_prompt
- Fixes mypy type errors: Signature incompatible with supertype CustomLogger and PromptManagementBase
* fix: add missing parameters to prompt management async methods (Category 1)
- vector_store_pre_call_hook: add ignore_prompt_manager_model, ignore_prompt_manager_optional_params, prompt_spec
- gitlab_prompt_manager: add ignore parameters, fix litellm_logging_obj type
- bitbucket_prompt_manager: add ignore parameters, fix litellm_logging_obj type
- proxy/custom_prompt_management: add prompt_spec parameter
- Fixes mypy type errors: Signature incompatible with supertype
* fix: fix arize_phoenix_prompt_manager and custom_prompt_management (Category 2)
- arize_phoenix_prompt_manager: add prompt_spec to all methods, fix prompt_id types, implement async_compile_prompt_helper
- custom_prompt_management: implement async_compile_prompt_helper abstract method
- Fixes mypy type errors: Signature incompatible with supertype and abstract method errors
* fix: fix obvious type errors (Category 3 - Quick Wins)
- langfuse: change 'callable' to 'Callable' type annotation
- presidio: add type narrowing check for Choices vs StreamingChoices
- StreamingChoices doesn't have .message attribute, only Choices does
- Add hasattr check before accessing choice.message
- Fixes mypy type errors: callable? not callable and union-attr errors
* fix: handle expires_after None in Azure files handler (Todo 14)
- Extract logic to _prepare_create_file_data helper method
- Remove expires_after from dict if None to match SDK's Omit pattern
- Add type ignore for FileExpiresAfter -> file_create_params.ExpiresAfter mismatch
- Fixes mypy error: Argument expires_after has incompatible type
* fix: change purpose parameter type to OpenAIFilesPurpose (Todo 18)
- Import OpenAIFilesPurpose in storage_backend_service.py
- Change upload_file_to_storage_backend purpose parameter from str to OpenAIFilesPurpose
- Change _create_file_object_with_storage_metadata purpose parameter from str to OpenAIFilesPurpose
- Fixes mypy error: Argument purpose has incompatible type str; expected Literal type
- Purpose is already validated in files_endpoints.py before reaching these functions
* fix: handle UploadFile | str type for expires_after form fields (Todo 19)
- Validate expires_after[anchor] and expires_after[seconds] are strings, not UploadFiles
- Validate anchor equals 'created_at' before using literal in TypedDict
- Use literal 'created_at' (not variable) in FileExpiresAfter to satisfy Literal type
- Add proper error handling for invalid anchor values and int conversion
- Fixes mypy errors: Incompatible types for anchor and seconds in FileExpiresAfter
* fix: add type narrowing for expires_after_seconds_str to fix mypy error
- Add assert statement after UploadFile validation to help mypy narrow type
- Use validated variable with explicit str type annotation
- Fixes: Argument of type 'UploadFile | str' cannot be assigned to int()
* fix: trigger async_success_handler for MCP tool calls to enable cost tracking and logging
- Set call_type to CallTypes.call_mcp_tool.value before calling async_success_handler
- Update mcp_tool_call_metadata with cost info when server is found
- Call async_success_handler to build standard_logging_object and trigger callbacks
- Fixes test_mcp_cost_tracking by ensuring standard_logging_payload is populated
* refactor: use positive isinstance check for safer type narrowing
- Replace assert with positive isinstance(..., str) check
- Matches codebase pattern (see pass_through_endpoints.py)
- Safer than assert: assertions can be disabled with -O flag
- Mypy properly narrows type after positive isinstance check
- More explicit and readable than assert statement
* fix: add missing REDIS_DAILY_AGENT_SPEND_UPDATE_QUEUE to ServiceTypes enum (Todo 17)
- Add REDIS_DAILY_AGENT_SPEND_UPDATE_QUEUE enum value following the pattern of other daily spend queues
- Add corresponding entry to DEFAULT_SERVICE_CONFIGS with GAUGE metrics
- Fixes mypy error: 'type[ServiceTypes]' has no attribute 'REDIS_DAILY_AGENT_SPEND_UPDATE_QUEUE'
- This enum value is already used in redis_update_buffer.py for agent spend tracking
* Implement fix for thinking_blocks and converse API calls
This fixes Claude's models via the Converse API, which should also fix
Claude Code.
* Add thinking literal
* Fix mypy issues
* Type fix for redacted thinking
* Add voyage model integration in sagemaker
* Add config file logic
* Use already exiting voyage transformation
* refactor code as per comments
* fix merge error
* refactor code as per comments
* refactor code as per comments
* UI new build
* [Fix] router - regression when adding/removing models (#15451)
* fix(router): update model_name_to_deployment_indices on deployment removal
When a deployment is deleted, the model_name_to_deployment_indices map
was not being updated, causing stale index references. This could lead
to incorrect routing behavior when deployments with the same model_name
were dynamically removed.
Changes:
- Update _update_deployment_indices_after_removal to maintain
model_name_to_deployment_indices mapping
- Remove deleted indices and decrement indices greater than removed index
- Clean up empty entries when no deployments remain for a model name
- Update test to verify proper index shifting and cleanup behavior
* fix(router): remove redundant index building during initialization
Remove duplicate index building operations that were causing unnecessary
work during router initialization:
1. Removed redundant `_build_model_id_to_deployment_index_map` call in
__init__ - `set_model_list` already builds all indices from scratch
2. Removed redundant `_build_model_name_index` call at end of
`set_model_list` - the index is already built incrementally via
`_create_deployment` -> `_add_model_to_list_and_index_map`
Both indices (model_id_to_deployment_index_map and
model_name_to_deployment_indices) are properly maintained as lookup
indexes through existing helper methods. This change eliminates O(N)
duplicate work during initialization without any behavioral changes.
The indices continue to be correctly synchronized with model_list on
all operations (add/remove/upsert).
* fix(prometheus): Fix Prometheus metric collection in a multi-workers environment (#14929)
Co-authored-by: sotazhang <sotazhang@tencent.com>
* Add tiered pricing and cost calculation for xai
* Use generic cost calculator
* Resolve conflicts in generated HTML files
* Remove penalty params as supported params for gemini preview model (#15503)
* fix conversion of thinking block
* add application level encryption in SQS (#15512)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* add application level encryption in SQS
* add application level encryption in SQS
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
* [Feat] Bedrock Knowledgebase - return search_response when using /chat/completions API with LiteLLM (#15509)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* add AnthropicCitation
* fix async_post_call_success_deployment_hook
* fix add vector_store_custom_logger to global callbacks
* test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call
* async_post_call_success_deployment_hook
* add async_post_call_streaming_deployment_hook
* async def test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call_streaming(setup_vector_store_registry):
* fix _call_post_streaming_deployment_hook
* fix async_post_call_streaming_deployment_hook
* test update
* docs: Accessing Search Results
* docs KB
* fix chatUI
* fix searchResults
* fix onSearchResults
* fix kb
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* [Feat] Add dynamic rate limits on LiteLLM Gateway (#15518)
* docs: fix doc
* docs(index.md): bump rc
* [Fix] GEMINI - CLI - add google_routes to llm_api_routes (#15500)
* fix: add google_routes to llm_api_routes
* test: test_virtual_key_llm_api_routes_allows_google_routes
* build: bump version
* bump: version 1.78.0 → 1.78.1
* fix: KeyRequestBase
* fix rpm_limit_type
* fix dynamic rate limits
* fix use dynamic limits here
* fix _should_enforce_rate_limit
* fix _should_enforce_rate_limit
* fix counter
* test_dynamic_rate_limiting_v3
* use _create_rate_limit_descriptors
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
* Add google rerank endpoint
* Add docs
* fix mypy error
* fix mypy and lint errors
* Add haiku 4.5 integration
* Add haiku 4.5 integration for other regions as well
* Handle citation field correctly
* Fix filtering headers for signature calcs
* Add haiku 4.5 integration (#15650)
---------
Co-authored-by: Leslie Cheng <leslie.cheng5@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com>
Co-authored-by: Lucas <10226902+LoadingZhang@users.noreply.github.com>
Co-authored-by: sotazhang <sotazhang@tencent.com>
Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: deepanshu <deepanshu.lulla@hq.bill.com>
