test_parallel_function_call_anthropic_error_msg was flaky because other
tests set litellm.modify_params=True without resetting it. When True,
the validation adds a dummy tool instead of raising UnsupportedParamsError.
Fix: save/restore modify_params around the test.
test_vertex_ai_llama_tool_calling failed on intermittent 404 from the
Llama model endpoint in us-east5. Fix: skip on NotFoundError like
the existing RateLimitError handling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test used a fixed 40s sleep before checking spend logs, but async
spend logging in CI sometimes takes longer to flush. Replace with a
polling loop (10s interval, 120s max) that exits early on success.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The eager tiktoken tests were clearing all litellm modules from
sys.modules and re-importing, creating new module objects with different
class identities. This broke unittest.mock.patch for all subsequent
tests on the same xdist worker. Running these tests in subprocesses
provides perfect isolation.
Fixes: test_metadata_passed_to_custom_callback_codex_models,
test_oidc_github_success, test_oidc_google_cached,
test_oidc_google_failure,
test_encrypted_content_affinity_bypasses_rpm_limits, and 5 others.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test asserts that a ttl=0 cached entry expires immediately, so the
second call should not return cached content. Both calls used the same
mock_response text, making the content != assertion always fail. Use
different mock_response values so a cache hit is distinguishable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The isolate_litellm_state conftest fixture saved/restored litellm.callbacks
but never cleared it before each test, unlike the other callback lists. It
also didn't handle litellm.model_fallbacks. Leaked callbacks and fallback
config caused mocked tests to route through Router/fallback paths, hitting
real APIs with mock keys.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The check_batch_cost_job runs on a 10-40s interval and sets
batch_processed=True after sending the S3 callback. 30s (6×5s) wasn't
enough margin; 50s (10×5s) covers the worst-case poll interval plus
processing time, while still being 3.6x faster than the original 180s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4 integration tests were flaky in CI because the AsyncHTTPHandler.post
mock was bypassed when aiohttp transport is used. Mock at the higher
BaseLLMHTTPHandler.async_response_api_handler level instead, which
bypasses the HTTP layer entirely while still exercising router deployment
selection, pre-call checks, and response post-processing (item ID rewriting).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace real OpenAI/Anthropic/Bedrock API calls with mock_response in
~20 cache tests to eliminate network-dependent flakiness
- Remove -x (fail-fast) from caching_unit_tests so all failures are reported
- Add parallelism: 2 with circleci tests run --split-by=timings
- Improve pip dependency cache key (v2-caching-deps) with fallback key
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests failed intermittently in CI (-n 8 workers) because cached
AsyncHTTPHandler instances from other tests bypassed the class-level
mock on AsyncHTTPHandler.post, causing real requests to OpenAI with
mock API keys. Router retries (default 2) masked the root cause.
- Add autouse fixture to flush litellm.in_memory_llm_clients_cache
before/after each test so mocks always apply to fresh clients
- Set num_retries=0 on all Router instances to surface mock failures
immediately instead of silently retrying
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a completed batch is served from the DB cache, _hidden_params was empty,
causing the managed files hook to skip output_file_id translation from raw
provider IDs to unified IDs. This fix populates unified_batch_id and model_id
on the early-return path, with a guard against double-encoding when the DB
already stores unified IDs.
Also reduces file deletion retry delay (20s→5s), reruns (5→2), and CI timeout
(30m→15m) to cut worst-case runtime from ~16min to ~4min.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Llama-3.2-3B-Instruct-Turbo is no longer available as a serverless model
on Together AI. Switch to Llama-3.3-70B-Instruct-Turbo which is still
available and has cost data in the model prices map.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add num_retries=0 to the async acompletion call to prevent retries
when the mock returns invalid response data. The test only validates
request payload format, not retry behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_router_context_window_check_pre_call_check_out_group: replace deprecated
gpt-3.5-turbo-1106 (removed from model_cost, returns max_input_tokens=0) with
gpt-4.1-mini + mock_response
- test_async_fallbacks: filter "Task was destroyed but it is pending" messages
that leak from parallel test execution in CI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_post_call_rule_streaming in test_rules.py sets
litellm.post_call_rules but never cleans up. Since
pytest_collection_modifyitems sorts tests by name across modules,
the leaked rule causes failures in test_streaming.py,
test_register_model.py, and test_sagemaker.py.
Add pre_call_rules and post_call_rules to the isolate_litellm_state
fixture's save/restore and clear lists.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, stream_chunk_builder only took annotations from the first
chunk that contained them, losing any annotations from later chunks.
This is a problem because providers like Gemini/Vertex AI send grounding
metadata (converted to annotations) in the final streaming chunk, while
other providers may spread annotations across multiple chunks.
Changes:
- Collect and merge annotations from ALL annotation-bearing chunks
instead of only using the first one
The test relied on a global side effect (customLogger initialization in
litellm_logging.py) from prior tests' success callbacks to dispatch
failure callbacks. When tests run in parallel by file, no prior test
initializes customLogger, so the Router's deployment_callback_on_failure
was never invoked and cooldowns were never set.
Rewrite to directly call deployment_callback_on_failure with a proper
RateLimitError containing retry-after headers, testing the cooldown
logic without depending on the logging callback chain.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_hanging_request_azure: mock httpx.AsyncClient.send to simulate slow
response instead of racing real network latency against a 10ms timeout.
The old non-existent deployment (gpt-4o-new-test) returned 404 faster
than the timeout, causing NotFoundError instead of APITimeoutError.
- test_completion_together_ai_llama: update model from deprecated
Meta-Llama-3.1-8B-Instruct-Turbo to Llama-3.2-3B-Instruct-Turbo
(Together AI removed the old model from serverless).
- conftest.py: clear litellm.callbacks list before each test to prevent
proxy hooks (SkillsInjectionHook, VirtualKeyModelMaxBudgetLimiter)
from leaking across tests via Router initialization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test was failing because it depended on real API calls to deprecated
models. Now uses mock_response to validate streaming through the router
without external dependencies.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_async_fallbacks_streaming: replace deprecated gpt-3.5-turbo fallback
with gpt-4o-mini, fix use of module-level kwargs variable
- test_ausage_based_routing_fallbacks: remove Redis dependency to prevent
shared state across parallel CI containers (test already uses mock_response)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Router testing: add CircleCI parallelism=4 with timing-based test splitting
- Guardrails testing: add pytest-xdist -n 4, suppress DEBUG logs with LITELLM_LOG=WARNING
- Rewrite conftest.py in both test dirs for xdist compatibility (save/restore pattern)
- Fix module-level Router instances in test_router_fallback_handlers, test_router_custom_routing, test_acooldowns_router
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix __main__ block: test returns None now, so always exited 1
- Close litellm_async_client in test_ssl_verification_with_aiohttp_transport
- Save/restore litellm.force_ipv4 and litellm.disable_aiohttp_transport
in test_force_ipv4_transport and test_aiohttp_disabled_transport
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
count_aiohttp_sessions() iterates every object in the Python GC,
which hangs in CI when xdist workers have millions of loaded objects.
Replace with direct session.closed checks — same coverage, no hang.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove real HTTP call to example.com in test_force_ipv4_transport
(hangs in CI when network is slow/unavailable)
- Close leaked aiohttp.ClientSession in test_ssl_verification_with_aiohttp_transport
- Add cleanup for transports in test_aiohttp_transport_trust_env_setting
- Add cleanup for handler in test_ssl_security_level
- Add cleanup for transport in test_ssl_context_transport
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: prisma migrate deploy failures on pre-existing instances
Fixes failed migrations due to idempotent schema changes on pre-existing litellm instances.
Problems:
1. P3018 recovery handler never returned True on successful resolution, causing "Database setup failed after multiple retries" even when the final recovery succeeded
2. _roll_back_migration exceptions escaped the P3018 handler, preventing _resolve_specific_migration from running
3. Migration SQL used ADD COLUMN/DROP COLUMN without IF [NOT] EXISTS, failing if schema was already modified
Changes:
- Add return True after successful P3018 idempotent error recovery
- Wrap _roll_back_migration in try/except to allow recovery continuation even if rollback fails
- Make migration.sql idempotent with IF NOT EXISTS / IF EXISTS clauses
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* test: add migration SQL idempotency safety tests
Adds TestMigrationSQLIdempotency test class that statically validates all
migration SQL files created after 2026-03-11 use idempotent DDL:
- ADD COLUMN must use IF NOT EXISTS
- DROP COLUMN must use IF EXISTS
- DROP INDEX must use IF EXISTS
- CREATE INDEX must use IF NOT EXISTS
This prevents the class of errors where prisma migrate deploy fails on
pre-existing instances because the schema was already modified.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: also catch TimeoutExpired in P3018 rollback handler
_roll_back_migration uses subprocess.run with timeout=60, so it can raise
subprocess.TimeoutExpired in addition to CalledProcessError. Without
catching this, a slow database during rollback would escape the handler
and bypass _resolve_specific_migration — the same class of bug.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: make all 85 migration SQL files idempotent, remove test cutoff
Fixed all existing migration files to use IF [NOT] EXISTS for DDL
statements (ADD COLUMN, DROP COLUMN, DROP INDEX, CREATE INDEX).
Removed the date cutoff from the idempotency tests so they now
validate all migrations, not just recent ones.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: make migration failure non-fatal by default, add --require_db_migration flag
By default the proxy now warns and continues when database migration
fails. Pass --require_db_migration (or set REQUIRE_DB_MIGRATION=true)
to restore the previous behavior of exiting with an error.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: wrap _resolve_specific_migration in try/except, guard RENAME COLUMN and ADD CONSTRAINT
Three fixes:
1. _resolve_specific_migration in the P3018 handler was not wrapped in
try/except, so failures there would bypass the return True and
propagate unexpectedly — partially defeating the rollback fix.
2. Bare RENAME COLUMN in 20260303000000_update_tool_table_policies was
non-idempotent. Wrapped in DO $$ IF EXISTS block. Also wrapped all
28 bare ADD CONSTRAINT statements across 9 migration files in
DO $$ IF NOT EXISTS (pg_constraint) blocks.
3. Added test_rename_column_is_guarded and test_add_constraint_is_guarded
to TestMigrationSQLIdempotency for full DDL coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: retry after resolving idempotent migration, guard DROP CONSTRAINT
Three fixes:
1. Both P3009 and P3018 idempotent handlers returned True after
resolving a single migration, exiting before remaining pending
migrations were applied. Now they continue the retry loop so
prisma migrate deploy runs again for any remaining migrations.
2. Two migration files had bare DROP CONSTRAINT without a DO $$ IF
EXISTS guard, which fails if the constraint was already dropped.
Wrapped both in idempotent DO $$ blocks.
3. Added test_drop_constraint_is_guarded to catch unguarded DROP
CONSTRAINT in future migrations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: P3009 try/except, CREATE TABLE IF NOT EXISTS, restore fail-fast default
Four fixes:
1. P3009 idempotent handler now has the same try/except around
_roll_back_migration and _resolve_specific_migration as the P3018
handler. Previously a rollback or resolve failure in the P3009 path
would propagate and leave the migration unresolved.
2. Added IF NOT EXISTS to all 57 bare CREATE TABLE statements across
34 migration files. Added test_create_table_uses_if_not_exists to
catch this pattern.
3. Reverted the backwards-incompatible default behavior change: the
proxy now fails fast on migration failure (original behavior).
Added --skip_db_migration_check / SKIP_DB_MIGRATION_CHECK to
opt into warn-and-continue instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
When scope_user_search_to_org flag is ON, team admins (non-org-admins) were
getting 403 because the code only checked for ORG_ADMIN role in org memberships.
Now checks all org memberships (any role) and falls back to the API key's team_id
to resolve the org.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add SecretRedactionFilter to scrub API keys, tokens, and credentials
from all log records (messages, args, tracebacks, extra fields).
- Enable redaction by default; opt out with LITELLM_DISABLE_REDACT_SECRETS=true
- Redact patterns: sk-*, Bearer tokens, x-api-key values, base64 creds
- Handle JSON formatter exception hooks and percent-style format args
- Snapshot dict iteration to avoid RuntimeError during concurrent logging
Asserts that GET /get/internal_user_settings returns
INTERNAL_USER_VIEW_ONLY on a fresh DB with no saved settings,
matching the runtime fallback in SSO/SCIM/JWT provisioning.
* feat: add sagemaker_nova provider for Nova models on SageMaker
Add support for custom/fine-tuned Amazon Nova models (Nova Micro, Nova Lite,
Nova 2 Lite) deployed on SageMaker Inference real-time endpoints.
Nova uses OpenAI-compatible request/response format with additional
Nova-specific parameters (top_k, reasoning_effort, allowed_token_ids,
truncate_prompt_tokens) and requires stream:true in the request body.
Nova endpoints also reject 'model' in the request body.
Changes:
- New provider: sagemaker_nova/<endpoint-name>
- SagemakerNovaConfig inherits from SagemakerChatConfig
- Override transform_request to strip 'model' from request body
- Override supports_stream_param_in_request_body (True for Nova)
- Extend get_supported_openai_params with Nova-specific params
- Refactored SagemakerChatConfig to use custom_llm_provider param
instead of hardcoded strings (backwards-compatible)
- Consolidated main.py routing for sagemaker_chat and sagemaker_nova
- 22 unit tests + 9 integration tests (skip-gated)
- Documentation with SDK, streaming, multimodal, and proxy examples
- All tests verified against live SageMaker Nova endpoint
* fix: move integration tests to tests/local_testing/ per test directory policy
* fix: remove unused module-level SagemakerNovaConfig instance
The sagemaker_nova_config singleton was never imported or used — the
ProviderConfigManager creates its own instance via the lambda registered
in utils.py. Removing this leftover boilerplate.
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
- Fix mypy arg-type error in background_streaming.py by adding proper
type annotation and cast for terminal_status
- Fix ruff F401 false positive for httpx import in vantage_destination.py
caused by from __future__ import annotations
- Fix flaky test_arouter_responses_api_bridge by providing a properly
structured mock response to prevent exception mapping errors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-request `httpx.AsyncClient` with `get_async_httpx_client`
to avoid the +500ms latency penalty from creating new clients per
request. Updates tests to mock the cached client factory.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix genuine regression in responses_api_bridge_check where the second
call assigned to `model_info` instead of `responses_api_model_info`,
preventing gpt-5.4 + tools + reasoning_effort from routing to the
Responses API bridge.
Also update outdated tests:
- Vantage tests: match "csv" file key and use supported column names
- Anthropic caching test: add "type": "custom" to expected tool payload
- Claude Agent SDK test: remove non-deterministic LLM content assertion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Map provider finish_reason "content_filtered" to the OpenAI-compatible "content_filter" and extend core_helpers tests to cover this case.
Made-with: Cursor
* fix: restore offline tiktoken cache for non-root envs
Made-with: Cursor
* chore: mkdir for custom tiktoken cache dir
Made-with: Cursor
* test: patch tiktoken.get_encoding in custom-dir test to avoid network
Made-with: Cursor
* test: clear CUSTOM_TIKTOKEN_CACHE_DIR in helper for test isolation
Made-with: Cursor
* test: restore default_encoding module state after custom-dir test
Made-with: Cursor
