- Promote _fetch_managed_vector_stores_by_uuids from @staticmethod to a module-level
async helper get_managed_vector_store_rows_by_uuids, following the same standalone
helper pattern as get_team_object / get_key_object so the hot-path DB read is a
named importable function rather than an inline prisma_client.db.* call
- Pass no-log=True to both inner _call_aresponses sub-calls so they do not fire
independent billing/monitoring callbacks; cost is accumulated in the synthesized
response's _hidden_params for the outer responses() call
- Add test_H11b covering the primary queries (plural array) function-tool schema,
complementing H11 which exercises only the backward-compat singular query path
Made-with: Cursor
- Re-add should_use_emulated_file_search() to emulated_handler.py so H5/H6/H7/H13 tests don't fail with ImportError
- Remove per-file-id deduplication from _build_search_results_for_include so all chunks are returned (matching OpenAI native file_search behaviour); update test_H14 to assert 2 results
- Extract raw prisma DB query in check_vector_store_ids_access into a static _fetch_managed_vector_stores_by_uuids helper so the hot request path uses a named, testable function instead of an inline prisma_client.db.* call
- Remove developer-local path from test module docstring
Made-with: Cursor
Strip internal logging ids from emulated sub-calls, dedupe included search_results by file_id, clean unused imports, and add unit coverage for dedupe behavior.
Made-with: Cursor
Include all function_call items when building emulated follow-up input and update tests to assert real emulated routing + Responses-format function tool structure.
Made-with: Cursor
* fix(test): add missing mocks for test_streamable_http_mcp_handler_mock
The test was missing mocks for extract_mcp_auth_context and set_auth_context,
causing the handler to fail silently in the except block instead of reaching
session_manager.handle_request. This mirrors the fix already applied to the
sibling test_sse_mcp_handler_mock.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): route OpenAI models through chat completions in pass-through tests
The test_anthropic_messages_openai_model_streaming_cost_injection test fails
because the OpenAI Responses API returns 400 for requests routed through the
Anthropic Messages endpoint. Setting LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true
routes OpenAI models through the stable chat completions path instead.
Cost injection still works since it happens at the proxy level.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): fix assemblyai custom auth and router wildcard test flakiness
1. custom_auth_basic.py: Add user_role='proxy_admin' so the custom auth
user can access management endpoints like /key/generate. The test
test_assemblyai_transcribe_with_non_admin_key was hidden behind an
earlier -x failure and was never reached before.
2. test_router_utils.py: Add flaky(retries=3) and increase sleep from 1s
to 2s for test_router_get_model_group_usage_wildcard_routes. The async
callback needs time to write usage to cache, and 1s is insufficient on
slower CI hardware.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* ci: retrigger CI pipeline
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): use LitellmUserRoles enum instead of raw string in custom_auth_basic
Fixes mypy error: Argument 'user_role' has incompatible type 'str'; expected 'LitellmUserRoles | None'
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: don't close HTTP/SDK clients on LLMClientCache eviction (#22926)
* fix: don't close HTTP/SDK clients on LLMClientCache eviction
Removing the _remove_key override that eagerly called aclose()/close()
on evicted clients. Evicted clients may still be held by in-flight
streaming requests; closing them causes:
RuntimeError: Cannot send a request, as the client has been closed.
This is a regression from commit fb72979432. Clients that are no longer
referenced will be garbage-collected naturally. Explicit shutdown cleanup
happens via close_litellm_async_clients().
Fixes production crashes after the 1-hour cache TTL expires.
* test: update LLMClientCache unit tests for no-close-on-eviction behavior
Flip the assertions: evicted clients must NOT be closed. Replace
test_remove_key_closes_async_client → test_remove_key_does_not_close_async_client
and equivalents for sync/eviction paths.
Add test_remove_key_removes_plain_values for non-client cache entries.
Remove test_background_tasks_cleaned_up_after_completion (no more _background_tasks).
Remove test_remove_key_no_event_loop variant that depended on old behavior.
* test: add e2e tests for OpenAI SDK client surviving cache eviction
Add two new e2e tests using real AsyncOpenAI clients:
- test_evicted_openai_sdk_client_stays_usable: verifies size-based eviction
doesn't close the client
- test_ttl_expired_openai_sdk_client_stays_usable: verifies TTL expiry
eviction doesn't close the client
Both tests sleep after eviction so any create_task()-based close would
have time to run, making the regression detectable.
Also expand the module docstring to explain why the sleep is required.
* docs(AGENTS.md): add rule — never close HTTP/SDK clients on cache eviction
* docs(CLAUDE.md): add HTTP client cache safety guideline
* [Fix] Install bsdmainutils for column command in security scans
The security_scans.sh script uses `column` to format vulnerability
output, but the package wasn't installed in the CI environment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle string callback values in prometheus multiproc setup
When callbacks are configured as a plain string (e.g., `callbacks: "my_callback"`)
instead of a list, the proxy crashes on startup with:
TypeError: can only concatenate str (not "list") to str
Normalize each callback setting to a list before concatenating.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* bump: version 1.82.2 → 1.82.3
* fix(test): update test_startup_fails_when_db_setup_fails for opt-in enforcement
The --enforce_prisma_migration_check flag is now required to trigger
sys.exit(1) on DB migration failure, after #23675 flipped the default
behavior to warn-and-continue.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(cost_calculator): use model name for per-request custom pricing when router_model_id has no pricing
When custom pricing is passed as per-request kwargs (input_cost_per_token/output_cost_per_token),
completion() registers pricing under the model name, but _select_model_name_for_cost_calc was
selecting the router deployment hash (which has no pricing data), causing response_cost to be 0.0.
Now checks whether the router_model_id entry actually has pricing before preferring it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add avideo_create_character, avideo_get_character, avideo_edit, and avideo_extension
to the skip condition since Azure video calls don't use initialize_azure_sdk_client.
Tests now properly skip with expected behavior instead of failing:
- test_ensure_initialize_azure_sdk_client_always_used[avideo_create_character] ✓
- test_ensure_initialize_azure_sdk_client_always_used[avideo_get_character] ✓
- test_ensure_initialize_azure_sdk_client_always_used[avideo_edit] ✓
- test_ensure_initialize_azure_sdk_client_always_used[avideo_extension] ✓
Made-with: Cursor
* fix: Fixes https://github.com/BerriAI/litellm/issues/23185
* fix(responses/main.py): ensure litellm metadata custom cost works
* refactor: move all logging updates to a common function, to have just 1 place to update logging kwarg updates
The eager tiktoken tests were clearing all litellm modules from
sys.modules and re-importing, creating new module objects with different
class identities. This broke unittest.mock.patch for all subsequent
tests on the same xdist worker. Running these tests in subprocesses
provides perfect isolation.
Fixes: test_metadata_passed_to_custom_callback_codex_models,
test_oidc_github_success, test_oidc_google_cached,
test_oidc_google_failure,
test_encrypted_content_affinity_bypasses_rpm_limits, and 5 others.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The isolate_litellm_state conftest fixture saved/restored litellm.callbacks
but never cleared it before each test, unlike the other callback lists. It
also didn't handle litellm.model_fallbacks. Leaked callbacks and fallback
config caused mocked tests to route through Router/fallback paths, hitting
real APIs with mock keys.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4 integration tests were flaky in CI because the AsyncHTTPHandler.post
mock was bypassed when aiohttp transport is used. Mock at the higher
BaseLLMHTTPHandler.async_response_api_handler level instead, which
bypasses the HTTP layer entirely while still exercising router deployment
selection, pre-call checks, and response post-processing (item ID rewriting).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests failed intermittently in CI (-n 8 workers) because cached
AsyncHTTPHandler instances from other tests bypassed the class-level
mock on AsyncHTTPHandler.post, causing real requests to OpenAI with
mock API keys. Router retries (default 2) masked the root cause.
- Add autouse fixture to flush litellm.in_memory_llm_clients_cache
before/after each test so mocks always apply to fresh clients
- Set num_retries=0 on all Router instances to surface mock failures
immediately instead of silently retrying
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, stream_chunk_builder only took annotations from the first
chunk that contained them, losing any annotations from later chunks.
This is a problem because providers like Gemini/Vertex AI send grounding
metadata (converted to annotations) in the final streaming chunk, while
other providers may spread annotations across multiple chunks.
Changes:
- Collect and merge annotations from ALL annotation-bearing chunks
instead of only using the first one
- Fix __main__ block: test returns None now, so always exited 1
- Close litellm_async_client in test_ssl_verification_with_aiohttp_transport
- Save/restore litellm.force_ipv4 and litellm.disable_aiohttp_transport
in test_force_ipv4_transport and test_aiohttp_disabled_transport
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
count_aiohttp_sessions() iterates every object in the Python GC,
which hangs in CI when xdist workers have millions of loaded objects.
Replace with direct session.closed checks — same coverage, no hang.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove real HTTP call to example.com in test_force_ipv4_transport
(hangs in CI when network is slow/unavailable)
- Close leaked aiohttp.ClientSession in test_ssl_verification_with_aiohttp_transport
- Add cleanup for transports in test_aiohttp_transport_trust_env_setting
- Add cleanup for handler in test_ssl_security_level
- Add cleanup for transport in test_ssl_context_transport
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When scope_user_search_to_org flag is ON, team admins (non-org-admins) were
getting 403 because the code only checked for ORG_ADMIN role in org memberships.
Now checks all org memberships (any role) and falls back to the API key's team_id
to resolve the org.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add SecretRedactionFilter to scrub API keys, tokens, and credentials
from all log records (messages, args, tracebacks, extra fields).
- Enable redaction by default; opt out with LITELLM_DISABLE_REDACT_SECRETS=true
- Redact patterns: sk-*, Bearer tokens, x-api-key values, base64 creds
- Handle JSON formatter exception hooks and percent-style format args
- Snapshot dict iteration to avoid RuntimeError during concurrent logging
* feat: add sagemaker_nova provider for Nova models on SageMaker
Add support for custom/fine-tuned Amazon Nova models (Nova Micro, Nova Lite,
Nova 2 Lite) deployed on SageMaker Inference real-time endpoints.
Nova uses OpenAI-compatible request/response format with additional
Nova-specific parameters (top_k, reasoning_effort, allowed_token_ids,
truncate_prompt_tokens) and requires stream:true in the request body.
Nova endpoints also reject 'model' in the request body.
Changes:
- New provider: sagemaker_nova/<endpoint-name>
- SagemakerNovaConfig inherits from SagemakerChatConfig
- Override transform_request to strip 'model' from request body
- Override supports_stream_param_in_request_body (True for Nova)
- Extend get_supported_openai_params with Nova-specific params
- Refactored SagemakerChatConfig to use custom_llm_provider param
instead of hardcoded strings (backwards-compatible)
- Consolidated main.py routing for sagemaker_chat and sagemaker_nova
- 22 unit tests + 9 integration tests (skip-gated)
- Documentation with SDK, streaming, multimodal, and proxy examples
- All tests verified against live SageMaker Nova endpoint
* fix: move integration tests to tests/local_testing/ per test directory policy
* fix: remove unused module-level SagemakerNovaConfig instance
The sagemaker_nova_config singleton was never imported or used — the
ProviderConfigManager creates its own instance via the lambda registered
in utils.py. Removing this leftover boilerplate.
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
- Fix mypy arg-type error in background_streaming.py by adding proper
type annotation and cast for terminal_status
- Fix ruff F401 false positive for httpx import in vantage_destination.py
caused by from __future__ import annotations
- Fix flaky test_arouter_responses_api_bridge by providing a properly
structured mock response to prevent exception mapping errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-request `httpx.AsyncClient` with `get_async_httpx_client`
to avoid the +500ms latency penalty from creating new clients per
request. Updates tests to mock the cached client factory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix genuine regression in responses_api_bridge_check where the second
call assigned to `model_info` instead of `responses_api_model_info`,
preventing gpt-5.4 + tools + reasoning_effort from routing to the
Responses API bridge.
Also update outdated tests:
- Vantage tests: match "csv" file key and use supported column names
- Anthropic caching test: add "type": "custom" to expected tool payload
- Claude Agent SDK test: remove non-deterministic LLM content assertion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map provider finish_reason "content_filtered" to the OpenAI-compatible "content_filter" and extend core_helpers tests to cover this case.
Made-with: Cursor
* fix(bedrock): respect s3_region_name for batch file uploads (#23569)
* fix(bedrock): respect s3_region_name for batch file uploads (GovCloud fix)
* fix: s3_region_name always wins over aws_region_name for S3 signing (Greptile feedback)
* fix: _filter_headers_for_aws_signature - Bedrock KB (#23571)
* fix: _filter_headers_for_aws_signature
* fix: filter None header values in all post-signing re-merge paths
Addresses Greptile feedback: None-valued headers were being filtered
during SigV4 signing but re-merged back into the final headers dict
afterward, which would cause downstream HTTP client failures.
Made-with: Cursor
* feat(router): tag_regex routing — route by User-Agent regex without per-developer tag config (#23594)
* feat(router): add tag_regex support for header-based routing
Adds a new `tag_regex` field to litellm_params that lets operators route
requests based on regex patterns matched against request headers — primarily
User-Agent — without requiring per-developer tag configuration.
Use case: route all Claude Code traffic (User-Agent: claude-code/x.y.z) to
a dedicated deployment by setting:
tag_regex:
- "^User-Agent: claude-code\\/"
in the deployment's litellm_params. Works alongside existing `tags` routing;
exact tag match takes precedence over regex match. Unmatched requests fall
through to deployments tagged `default`.
The matched deployment, pattern, and user_agent are recorded in
`metadata["tag_routing"]` so they flow through to SpendLogs automatically.
* fix(tag_regex): address backwards-compat, metadata overwrite, and warning noise
Three issues from code review:
1. Backwards-compat: `has_tag_filter` was widened to activate on any non-empty
User-Agent, which would raise ValueError for existing deployments using plain
tags without a `default` fallback. Fix: only activate header-based regex
filtering when at least one candidate deployment has `tag_regex` configured.
2. Metadata overwrite: `metadata["tag_routing"]` was overwritten for every
matching deployment in the loop, leaving inaccurate provenance when multiple
deployments match. Fix: write only for the first match.
3. Warning noise: an invalid regex pattern logged one warning per header string
rather than once per pattern. Fix: compile first (catching re.error once),
then iterate over header strings.
Also adds two new tests covering these cases, and adds docs page for
tag_regex routing with a Claude Code walk-through.
* refactor(tag_regex): remove unnecessary _healthy_list copy
* docs: merge tag_regex section into tag_routing.md, remove standalone page
- Add ## Regex-based tag routing (tag_regex) section to existing
tag_routing.md instead of a separate page
- Remove tag_regex_routing.md standalone doc (odd UX to have a separate
page for a sub-feature)
- Remove proxy/tag_regex_routing from sidebars.js
- Add match_any=False debug warning in tag_based_routing.py when regex
routing fires under strict mode (regex always uses OR semantics)
* fix(tag_regex): address greptile review - security docs, strict-mode enforcement, validation order
- Strengthen security note in tag_routing.md: explicitly state User-Agent
is client-supplied and can be set to any value; frame tag_regex as a
traffic classification hint, not an access-control mechanism
- Move tag_regex startup validation before _add_deployment() so an invalid
pattern never leaves partial router state
- Enforce match_any=False strict-tag policy: when a deployment has both
tags and tag_regex and the strict tag check fails, skip the regex fallback
rather than silently bypassing the operator's intent
- Extract per-deployment match logic into _match_deployment() helper to
keep get_deployments_for_tag() readable
- Add two new tests: strict-mode blocks regex fallback, regex-only
deployment still matches under match_any=False
* fix(ci): apply Black formatting to 14 files and stabilize flaky caplog tests
- Run Black formatter on 14 files that were failing the lint check
- Replace caplog-based assertions in TestAliasConflicts with
unittest.mock.patch on verbose_logger.warning for xdist compatibility
- The caplog fixture can produce empty text in pytest-xdist workers
in certain CI environments, causing flaky test failures
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
