* fix: Fixes https://github.com/BerriAI/litellm/issues/23185
* fix(responses/main.py): ensure litellm metadata custom cost works
* refactor: move all logging updates to a common function, to have just 1 place to update logging kwarg updates
The conftest fixtures were saving/restoring the current (potentially
contaminated) values of litellm globals like num_retries instead of
resetting to true defaults. Under xdist, module-level assignments
(e.g. `litellm.num_retries = 3` in 12+ test files) pollute the
shared module state and leak across tests in the same worker.
- Capture true litellm defaults at conftest import time and reset
before each test (local_testing + llm_translation)
- Make llm_translation/conftest.py xdist-safe (skip reload, add
state isolation)
- Replace asyncio.sleep(2) with polling in cooldown handler tests
- Add @pytest.mark.flaky to tests making real API calls under xdist
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Router tests: expand conftest save/restore to cover all globals mutated
by router tests (default_fallbacks, tag_budget_config, request_timeout,
enable_azure_ad_token_refresh, num_retries_per_request, model_cost,
token_counter). These were leaking across xdist workers.
Proxy tests: move test_proxy_utils.py (169 parametrized) and
test_proxy_server.py (72 parametrized) from part2 to part1, balancing
~370 vs ~360 tests (was ~129 vs ~600).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test intermittently fails in CI due to Redis cache write propagation
delays, causing the second call to miss the cache and hit OpenAI directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LLM responses are non-deterministic and ClaudeAgentOptions doesn't expose
temperature control. The model occasionally returns unexpected short responses
(e.g. just "!") instead of the expected greeting. Add up to 3 retries with a
more explicit prompt to make the test deterministic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test waited a fixed 3s for async callbacks to populate log_queue.
Under xdist -n 4, CPU contention can delay the GLOBAL_LOGGING_WORKER
background task beyond 3s. Replace fixed sleeps with polling loops
(up to 10s) that break as soon as the expected condition is met.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Module-level mutations (litellm.num_retries=3 in test_langfuse_e2e_test.py
and test_amazing_s3_logs.py, litellm.success_callback=['langfuse']) run
at import time, BEFORE any function fixture. The save/restore pattern
captured these polluted values as 'originals' and kept restoring them.
Fix: capture litellm defaults when conftest.py is first imported (before
test modules), then reset to those true defaults before each test instead
of saving/restoring the current (potentially polluted) state.
ThrottlingException is a transient AWS rate-limit error unrelated to code
correctness. Skip the test instead of failing the CI pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Clear _in_memory_loggers before/after each test to prevent cached logger
instances (LangsmithLogger, SlackAlerting, etc.) from leaking stale state
- Add pre_call_rules, post_call_rules to list attrs save/restore
- Add vector_store_registry to scalar attrs save/restore
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_avertex_batch_prediction: Add google.auth.default mock and env vars
so the test doesn't depend on real GCP credentials (was already a unit
test with mocked HTTP, just missing auth mock)
- test_async_create_batch[openai]: Add DNS pre-check that skips gracefully
when api.openai.com is unreachable instead of failing after 4 retries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The logging tests mutate many more litellm globals than guardrails tests
(turn_off_message_logging, s3_callback_params, datadog_params, service_callback,
etc.). The initial save/restore list only covered callbacks and a few basics,
causing state leaks like redaction settings bleeding across tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add LITELLM_LOG=WARNING to suppress verbose DEBUG log output
- Remove -s flag to stop capturing all stdout
- Bump xdist workers from -n 2 to -n 4
- Add --timeout=120 for safety
- Rewrite conftest.py to use save/restore pattern (matching guardrails_tests)
instead of per-function importlib.reload + event loop creation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The retrieve_batch endpoint sets batch status to "complete" but never set
batch_processed=True, permanently blocking file deletion. CheckBatchCost
(the safety net) also excluded completed batches from its primary query,
so batch_processed was never set by either path.
Three fixes:
1. update_batch_in_database sets batch_processed=True when status reaches
"complete", with old-schema fallback retry
2. CheckBatchCost primary query no longer excludes complete/completed
(batch_processed=False filter prevents reprocessing)
3. retrieve_batch early-return now includes "complete" (DB-normalized
spelling) to avoid unnecessary provider re-polls
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_parallel_function_call_anthropic_error_msg was flaky because other
tests set litellm.modify_params=True without resetting it. When True,
the validation adds a dummy tool instead of raising UnsupportedParamsError.
Fix: save/restore modify_params around the test.
test_vertex_ai_llama_tool_calling failed on intermittent 404 from the
Llama model endpoint in us-east5. Fix: skip on NotFoundError like
the existing RateLimitError handling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test used a fixed 40s sleep before checking spend logs, but async
spend logging in CI sometimes takes longer to flush. Replace with a
polling loop (10s interval, 120s max) that exits early on success.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The eager tiktoken tests were clearing all litellm modules from
sys.modules and re-importing, creating new module objects with different
class identities. This broke unittest.mock.patch for all subsequent
tests on the same xdist worker. Running these tests in subprocesses
provides perfect isolation.
Fixes: test_metadata_passed_to_custom_callback_codex_models,
test_oidc_github_success, test_oidc_google_cached,
test_oidc_google_failure,
test_encrypted_content_affinity_bypasses_rpm_limits, and 5 others.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test asserts that a ttl=0 cached entry expires immediately, so the
second call should not return cached content. Both calls used the same
mock_response text, making the content != assertion always fail. Use
different mock_response values so a cache hit is distinguishable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The isolate_litellm_state conftest fixture saved/restored litellm.callbacks
but never cleared it before each test, unlike the other callback lists. It
also didn't handle litellm.model_fallbacks. Leaked callbacks and fallback
config caused mocked tests to route through Router/fallback paths, hitting
real APIs with mock keys.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The check_batch_cost_job runs on a 10-40s interval and sets
batch_processed=True after sending the S3 callback. 30s (6×5s) wasn't
enough margin; 50s (10×5s) covers the worst-case poll interval plus
processing time, while still being 3.6x faster than the original 180s.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4 integration tests were flaky in CI because the AsyncHTTPHandler.post
mock was bypassed when aiohttp transport is used. Mock at the higher
BaseLLMHTTPHandler.async_response_api_handler level instead, which
bypasses the HTTP layer entirely while still exercising router deployment
selection, pre-call checks, and response post-processing (item ID rewriting).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace real OpenAI/Anthropic/Bedrock API calls with mock_response in
~20 cache tests to eliminate network-dependent flakiness
- Remove -x (fail-fast) from caching_unit_tests so all failures are reported
- Add parallelism: 2 with circleci tests run --split-by=timings
- Improve pip dependency cache key (v2-caching-deps) with fallback key
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests failed intermittently in CI (-n 8 workers) because cached
AsyncHTTPHandler instances from other tests bypassed the class-level
mock on AsyncHTTPHandler.post, causing real requests to OpenAI with
mock API keys. Router retries (default 2) masked the root cause.
- Add autouse fixture to flush litellm.in_memory_llm_clients_cache
before/after each test so mocks always apply to fresh clients
- Set num_retries=0 on all Router instances to surface mock failures
immediately instead of silently retrying
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a completed batch is served from the DB cache, _hidden_params was empty,
causing the managed files hook to skip output_file_id translation from raw
provider IDs to unified IDs. This fix populates unified_batch_id and model_id
on the early-return path, with a guard against double-encoding when the DB
already stores unified IDs.
Also reduces file deletion retry delay (20s→5s), reruns (5→2), and CI timeout
(30m→15m) to cut worst-case runtime from ~16min to ~4min.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Llama-3.2-3B-Instruct-Turbo is no longer available as a serverless model
on Together AI. Switch to Llama-3.3-70B-Instruct-Turbo which is still
available and has cost data in the model prices map.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add num_retries=0 to the async acompletion call to prevent retries
when the mock returns invalid response data. The test only validates
request payload format, not retry behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_router_context_window_check_pre_call_check_out_group: replace deprecated
gpt-3.5-turbo-1106 (removed from model_cost, returns max_input_tokens=0) with
gpt-4.1-mini + mock_response
- test_async_fallbacks: filter "Task was destroyed but it is pending" messages
that leak from parallel test execution in CI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_post_call_rule_streaming in test_rules.py sets
litellm.post_call_rules but never cleans up. Since
pytest_collection_modifyitems sorts tests by name across modules,
the leaked rule causes failures in test_streaming.py,
test_register_model.py, and test_sagemaker.py.
Add pre_call_rules and post_call_rules to the isolate_litellm_state
fixture's save/restore and clear lists.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, stream_chunk_builder only took annotations from the first
chunk that contained them, losing any annotations from later chunks.
This is a problem because providers like Gemini/Vertex AI send grounding
metadata (converted to annotations) in the final streaming chunk, while
other providers may spread annotations across multiple chunks.
Changes:
- Collect and merge annotations from ALL annotation-bearing chunks
instead of only using the first one
The test relied on a global side effect (customLogger initialization in
litellm_logging.py) from prior tests' success callbacks to dispatch
failure callbacks. When tests run in parallel by file, no prior test
initializes customLogger, so the Router's deployment_callback_on_failure
was never invoked and cooldowns were never set.
Rewrite to directly call deployment_callback_on_failure with a proper
RateLimitError containing retry-after headers, testing the cooldown
logic without depending on the logging callback chain.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_hanging_request_azure: mock httpx.AsyncClient.send to simulate slow
response instead of racing real network latency against a 10ms timeout.
The old non-existent deployment (gpt-4o-new-test) returned 404 faster
than the timeout, causing NotFoundError instead of APITimeoutError.
- test_completion_together_ai_llama: update model from deprecated
Meta-Llama-3.1-8B-Instruct-Turbo to Llama-3.2-3B-Instruct-Turbo
(Together AI removed the old model from serverless).
- conftest.py: clear litellm.callbacks list before each test to prevent
proxy hooks (SkillsInjectionHook, VirtualKeyModelMaxBudgetLimiter)
from leaking across tests via Router initialization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test was failing because it depended on real API calls to deprecated
models. Now uses mock_response to validate streaming through the router
without external dependencies.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_async_fallbacks_streaming: replace deprecated gpt-3.5-turbo fallback
with gpt-4o-mini, fix use of module-level kwargs variable
- test_ausage_based_routing_fallbacks: remove Redis dependency to prevent
shared state across parallel CI containers (test already uses mock_response)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Router testing: add CircleCI parallelism=4 with timing-based test splitting
- Guardrails testing: add pytest-xdist -n 4, suppress DEBUG logs with LITELLM_LOG=WARNING
- Rewrite conftest.py in both test dirs for xdist compatibility (save/restore pattern)
- Fix module-level Router instances in test_router_fallback_handlers, test_router_custom_routing, test_acooldowns_router
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix __main__ block: test returns None now, so always exited 1
- Close litellm_async_client in test_ssl_verification_with_aiohttp_transport
- Save/restore litellm.force_ipv4 and litellm.disable_aiohttp_transport
in test_force_ipv4_transport and test_aiohttp_disabled_transport
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
count_aiohttp_sessions() iterates every object in the Python GC,
which hangs in CI when xdist workers have millions of loaded objects.
Replace with direct session.closed checks — same coverage, no hang.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove real HTTP call to example.com in test_force_ipv4_transport
(hangs in CI when network is slow/unavailable)
- Close leaked aiohttp.ClientSession in test_ssl_verification_with_aiohttp_transport
- Add cleanup for transports in test_aiohttp_transport_trust_env_setting
- Add cleanup for handler in test_ssl_security_level
- Add cleanup for transport in test_ssl_context_transport
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: prisma migrate deploy failures on pre-existing instances
Fixes failed migrations due to idempotent schema changes on pre-existing litellm instances.
Problems:
1. P3018 recovery handler never returned True on successful resolution, causing "Database setup failed after multiple retries" even when the final recovery succeeded
2. _roll_back_migration exceptions escaped the P3018 handler, preventing _resolve_specific_migration from running
3. Migration SQL used ADD COLUMN/DROP COLUMN without IF [NOT] EXISTS, failing if schema was already modified
Changes:
- Add return True after successful P3018 idempotent error recovery
- Wrap _roll_back_migration in try/except to allow recovery continuation even if rollback fails
- Make migration.sql idempotent with IF NOT EXISTS / IF EXISTS clauses
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* test: add migration SQL idempotency safety tests
Adds TestMigrationSQLIdempotency test class that statically validates all
migration SQL files created after 2026-03-11 use idempotent DDL:
- ADD COLUMN must use IF NOT EXISTS
- DROP COLUMN must use IF EXISTS
- DROP INDEX must use IF EXISTS
- CREATE INDEX must use IF NOT EXISTS
This prevents the class of errors where prisma migrate deploy fails on
pre-existing instances because the schema was already modified.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: also catch TimeoutExpired in P3018 rollback handler
_roll_back_migration uses subprocess.run with timeout=60, so it can raise
subprocess.TimeoutExpired in addition to CalledProcessError. Without
catching this, a slow database during rollback would escape the handler
and bypass _resolve_specific_migration — the same class of bug.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: make all 85 migration SQL files idempotent, remove test cutoff
Fixed all existing migration files to use IF [NOT] EXISTS for DDL
statements (ADD COLUMN, DROP COLUMN, DROP INDEX, CREATE INDEX).
Removed the date cutoff from the idempotency tests so they now
validate all migrations, not just recent ones.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: make migration failure non-fatal by default, add --require_db_migration flag
By default the proxy now warns and continues when database migration
fails. Pass --require_db_migration (or set REQUIRE_DB_MIGRATION=true)
to restore the previous behavior of exiting with an error.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: wrap _resolve_specific_migration in try/except, guard RENAME COLUMN and ADD CONSTRAINT
Three fixes:
1. _resolve_specific_migration in the P3018 handler was not wrapped in
try/except, so failures there would bypass the return True and
propagate unexpectedly — partially defeating the rollback fix.
2. Bare RENAME COLUMN in 20260303000000_update_tool_table_policies was
non-idempotent. Wrapped in DO $$ IF EXISTS block. Also wrapped all
28 bare ADD CONSTRAINT statements across 9 migration files in
DO $$ IF NOT EXISTS (pg_constraint) blocks.
3. Added test_rename_column_is_guarded and test_add_constraint_is_guarded
to TestMigrationSQLIdempotency for full DDL coverage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: retry after resolving idempotent migration, guard DROP CONSTRAINT
Three fixes:
1. Both P3009 and P3018 idempotent handlers returned True after
resolving a single migration, exiting before remaining pending
migrations were applied. Now they continue the retry loop so
prisma migrate deploy runs again for any remaining migrations.
2. Two migration files had bare DROP CONSTRAINT without a DO $$ IF
EXISTS guard, which fails if the constraint was already dropped.
Wrapped both in idempotent DO $$ blocks.
3. Added test_drop_constraint_is_guarded to catch unguarded DROP
CONSTRAINT in future migrations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: P3009 try/except, CREATE TABLE IF NOT EXISTS, restore fail-fast default
Four fixes:
1. P3009 idempotent handler now has the same try/except around
_roll_back_migration and _resolve_specific_migration as the P3018
handler. Previously a rollback or resolve failure in the P3009 path
would propagate and leave the migration unresolved.
2. Added IF NOT EXISTS to all 57 bare CREATE TABLE statements across
34 migration files. Added test_create_table_uses_if_not_exists to
catch this pattern.
3. Reverted the backwards-incompatible default behavior change: the
proxy now fails fast on migration failure (original behavior).
Added --skip_db_migration_check / SKIP_DB_MIGRATION_CHECK to
opt into warn-and-continue instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
When scope_user_search_to_org flag is ON, team admins (non-org-admins) were
getting 403 because the code only checked for ORG_ADMIN role in org memberships.
Now checks all org memberships (any role) and falls back to the API key's team_id
to resolve the org.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add SecretRedactionFilter to scrub API keys, tokens, and credentials
from all log records (messages, args, tracebacks, extra fields).
- Enable redaction by default; opt out with LITELLM_DISABLE_REDACT_SECRETS=true
- Redact patterns: sk-*, Bearer tokens, x-api-key values, base64 creds
- Handle JSON formatter exception hooks and percent-style format args
- Snapshot dict iteration to avoid RuntimeError during concurrent logging
Asserts that GET /get/internal_user_settings returns
INTERNAL_USER_VIEW_ONLY on a fresh DB with no saved settings,
matching the runtime fallback in SSO/SCIM/JWT provisioning.
