_get_token_and_url_context_caching() was hardcoding model=None when
calling _check_custom_proxy(), which raises ValueError when api_base
is set because Gemini proxy URLs need the model name:
{api_base}/models/{model}:cachedContents
Fixes#23846
Fixes#23890 — Mistral's Voxtral transcription with `diarize=true` returns
`segments` (with speaker_id, timestamps) and `language`, but these fields
were dropped when mapping the response to TranscriptionResponse.
The count_tokens handler unconditionally overrode vertex_location to
us-central1 for Claude models, ignoring the user-configured
vertex_count_tokens_location parameter. Also, us-central1 is no longer
a supported region — Google now supports us-east5, europe-west1, and
asia-southeast1.
Now vertex_count_tokens_location takes precedence, vertex_location is
used as fallback, and us-east5 is the default only when neither is set.
Fixes#23872
* akto guardrails support in litellm
* docs(guardrails): add akto to supported values in types/guardrails.py
* frontend changes + fixes
* feat(akto): update Akto guardrail integration with new configuration options and modes
* docs(akto): enhance Akto documentation and configuration descriptions for clarity
* feat(tests): add proxy server request headers to sample request data
* refactor(akto): remove optional account and VXLAN IDs; update documentation and tests
* feat(akto): add event_type parameter for enhanced observability in guardrail logging
* refactor(akto): update environment variable references
* refactor the python codes
* refactor and fix linting
* refactor(akto): remove unused event hook and clean up imports
* refactor(akto): enhance AktoGuardrail with async support and improved logging
* fix: Register DynamoAI guardrail initializer and enum entry (#23752)
* fix: Register DynamoAI guardrail initializer and enum entry
Fix the "Unsupported guardrail: dynamoai" error by:
1. Adding DYNAMOAI to SupportedGuardrailIntegrations enum
2. Implementing initialize_guardrail() and registries in dynamoai/__init__.py
The DynamoAI guardrail was added in PR #15920 but never properly registered
in the initialization system. The __init__.py was missing the
guardrail_initializer_registry and guardrail_class_registry dictionaries
that the dynamic discovery mechanism looks for at module load time.
Fixes#22773
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Update litellm/proxy/guardrails/guardrail_hooks/dynamoai/__init__.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update litellm/proxy/guardrails/guardrail_hooks/dynamoai/__init__.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* test: Add tests for DynamoAI guardrail registration
Verifies enum entry, initializer registry, class registry,
instance creation, and global registry discovery.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* docs: add v1.82.3 release notes and update provider_endpoints_support.json (#23816)
* Revert "docs: add v1.82.3 release notes and update provider_endpoints_support…" (#23817)
This reverts commit 966124966f.
* Refactor Akto guardrail configuration and tests; update UI description and tags
* add account and vxlan ID parameters to Akto guardrail initialization; update Akto logo format
* enhance Akto guardrail documentation and improve error handling for non-JSON responses
* address greptile issues
* fix: update payload handling to use 'data' instead of 'json' in AktoGuardrail and adjust tests accordingly
---------
Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com>
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Joe Reyna <joseph.reyna@gmail.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Anthropic's 'refusal' stop_reason was missing from _FINISH_REASON_MAP,
causing it to fall through to the default 'stop' — hiding the fact that
the model refused to respond due to safety policies.
Fixes#23793
Models like gemini-3.1-flash-lite-preview send the final streaming chunk
with empty content (text:"") alongside finishReason:"STOP", instead of
omitting content entirely. The existing fix (PR #21577) only handled
chunks without content, so this case was missed.
Now, after processing candidates, if tool_calls were seen in earlier
chunks and a choice has finish_reason="stop", it is overridden to
"tool_calls" to match the OpenAI spec.
Fixes#22900
* fix(gemini): support images in tool_results for /v1/messages routing
convert_to_gemini_tool_call_result() dropped images in two cases:
- data-URL strings (data:image/...;base64,...) treated as plain text
- Anthropic image blocks in list content skipped
Add detection and convert both to Gemini inline_data BlobType so image
bytes are preserved.
Fixes#23712.
* fix(gemini): support images in tool_results for /v1/messages routing
convert_to_gemini_tool_call_result() dropped images in two cases:
- data-URL strings (data:image/...;base64,...) treated as plain text
- Anthropic image blocks in list content skipped
Add detection and convert both to Gemini inline_data BlobType so image
bytes are preserved.
Fixes#23712.
* fix(gemini): support images in tool_results for /v1/messages routing
convert_to_gemini_tool_call_result() dropped images in two cases:
- data-URL strings (data:image/...;base64,...) treated as plain text
- Anthropic image blocks in list content skipped
Add detection and convert both to Gemini inline_data BlobType so image
bytes are preserved.
Fixes#23712.
* fix(fireworks): skip #transform=inline for base64 data URLs
Closes#23583
* fix(fireworks): skip #transform=inline for base64 data URLs
Closes#23583
Appending #transform=inline to a data: URL corrupted the base64 payload,
causing binascii.Error (Incorrect padding) when Fireworks AI attempted to
decode the image. Data URLs are already inlined so the fragment is a no-op
anyway — guard both the str and dict image_url branches to skip the suffix
when the URL starts with "data:".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(fireworks): skip #transform=inline for base64 data URLs
Closes#23583
* fix(fireworks): skip #transform=inline for base64 data URLs
Closes#23583
* fix(fireworks): skip #transform=inline for base64 data URLs
Closes#23583
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* fix(test): add missing mocks for test_streamable_http_mcp_handler_mock
The test was missing mocks for extract_mcp_auth_context and set_auth_context,
causing the handler to fail silently in the except block instead of reaching
session_manager.handle_request. This mirrors the fix already applied to the
sibling test_sse_mcp_handler_mock.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): route OpenAI models through chat completions in pass-through tests
The test_anthropic_messages_openai_model_streaming_cost_injection test fails
because the OpenAI Responses API returns 400 for requests routed through the
Anthropic Messages endpoint. Setting LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES=true
routes OpenAI models through the stable chat completions path instead.
Cost injection still works since it happens at the proxy level.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): fix assemblyai custom auth and router wildcard test flakiness
1. custom_auth_basic.py: Add user_role='proxy_admin' so the custom auth
user can access management endpoints like /key/generate. The test
test_assemblyai_transcribe_with_non_admin_key was hidden behind an
earlier -x failure and was never reached before.
2. test_router_utils.py: Add flaky(retries=3) and increase sleep from 1s
to 2s for test_router_get_model_group_usage_wildcard_routes. The async
callback needs time to write usage to cache, and 1s is insufficient on
slower CI hardware.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* ci: retrigger CI pipeline
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): use LitellmUserRoles enum instead of raw string in custom_auth_basic
Fixes mypy error: Argument 'user_role' has incompatible type 'str'; expected 'LitellmUserRoles | None'
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: don't close HTTP/SDK clients on LLMClientCache eviction (#22926)
* fix: don't close HTTP/SDK clients on LLMClientCache eviction
Removing the _remove_key override that eagerly called aclose()/close()
on evicted clients. Evicted clients may still be held by in-flight
streaming requests; closing them causes:
RuntimeError: Cannot send a request, as the client has been closed.
This is a regression from commit fb72979432. Clients that are no longer
referenced will be garbage-collected naturally. Explicit shutdown cleanup
happens via close_litellm_async_clients().
Fixes production crashes after the 1-hour cache TTL expires.
* test: update LLMClientCache unit tests for no-close-on-eviction behavior
Flip the assertions: evicted clients must NOT be closed. Replace
test_remove_key_closes_async_client → test_remove_key_does_not_close_async_client
and equivalents for sync/eviction paths.
Add test_remove_key_removes_plain_values for non-client cache entries.
Remove test_background_tasks_cleaned_up_after_completion (no more _background_tasks).
Remove test_remove_key_no_event_loop variant that depended on old behavior.
* test: add e2e tests for OpenAI SDK client surviving cache eviction
Add two new e2e tests using real AsyncOpenAI clients:
- test_evicted_openai_sdk_client_stays_usable: verifies size-based eviction
doesn't close the client
- test_ttl_expired_openai_sdk_client_stays_usable: verifies TTL expiry
eviction doesn't close the client
Both tests sleep after eviction so any create_task()-based close would
have time to run, making the regression detectable.
Also expand the module docstring to explain why the sleep is required.
* docs(AGENTS.md): add rule — never close HTTP/SDK clients on cache eviction
* docs(CLAUDE.md): add HTTP client cache safety guideline
* [Fix] Install bsdmainutils for column command in security scans
The security_scans.sh script uses `column` to format vulnerability
output, but the package wasn't installed in the CI environment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: handle string callback values in prometheus multiproc setup
When callbacks are configured as a plain string (e.g., `callbacks: "my_callback"`)
instead of a list, the proxy crashes on startup with:
TypeError: can only concatenate str (not "list") to str
Normalize each callback setting to a list before concatenating.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* bump: version 1.82.2 → 1.82.3
* fix(test): update test_startup_fails_when_db_setup_fails for opt-in enforcement
The --enforce_prisma_migration_check flag is now required to trigger
sys.exit(1) on DB migration failure, after #23675 flipped the default
behavior to warn-and-continue.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(cost_calculator): use model name for per-request custom pricing when router_model_id has no pricing
When custom pricing is passed as per-request kwargs (input_cost_per_token/output_cost_per_token),
completion() registers pricing under the model name, but _select_model_name_for_cost_calc was
selecting the router deployment hash (which has no pricing data), causing response_cost to be 0.0.
Now checks whether the router_model_id entry actually has pricing before preferring it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add avideo_create_character, avideo_get_character, avideo_edit, and avideo_extension
to the skip condition since Azure video calls don't use initialize_azure_sdk_client.
Tests now properly skip with expected behavior instead of failing:
- test_ensure_initialize_azure_sdk_client_always_used[avideo_create_character] ✓
- test_ensure_initialize_azure_sdk_client_always_used[avideo_get_character] ✓
- test_ensure_initialize_azure_sdk_client_always_used[avideo_edit] ✓
- test_ensure_initialize_azure_sdk_client_always_used[avideo_extension] ✓
Made-with: Cursor
The xdist-conditional reload (manual reset in xdist mode) was missing
attributes that importlib.reload resets, causing Azure connection errors.
The original conftest used importlib.reload unconditionally (even under
xdist) and that worked on main. Restore that behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The old conftest never flushed HTTP client cache. Adding flush_cache() before
every test forces new TCP connections to external APIs, causing transient
connection failures under xdist parallelism. Global state isolation is already
handled by _SCALAR_DEFAULTS reset.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_rerank.py sets litellm.api_base = "http://localhost:4000" which leaked
to all subsequent tests on the same xdist worker, causing connection failures
across every provider (Cohere, Azure, OpenAI, etc.).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The revert of 9711e3adfe left xdist tests without proper state isolation.
Module-level assignments like `litellm.num_retries = 3` in 12+ test files
pollute shared globals, and the fixture was saving/restoring contaminated
values instead of resetting to true defaults.
- Capture true litellm defaults at conftest import time and reset before
each test (local_testing + llm_translation)
- Make llm_translation/conftest.py xdist-safe (skip reload under xdist,
add state isolation)
- Replace asyncio.sleep(2) with polling in cooldown handler tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Fixes https://github.com/BerriAI/litellm/issues/23185
* fix(responses/main.py): ensure litellm metadata custom cost works
* refactor: move all logging updates to a common function, to have just 1 place to update logging kwarg updates
The conftest fixtures were saving/restoring the current (potentially
contaminated) values of litellm globals like num_retries instead of
resetting to true defaults. Under xdist, module-level assignments
(e.g. `litellm.num_retries = 3` in 12+ test files) pollute the
shared module state and leak across tests in the same worker.
- Capture true litellm defaults at conftest import time and reset
before each test (local_testing + llm_translation)
- Make llm_translation/conftest.py xdist-safe (skip reload, add
state isolation)
- Replace asyncio.sleep(2) with polling in cooldown handler tests
- Add @pytest.mark.flaky to tests making real API calls under xdist
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Router tests: expand conftest save/restore to cover all globals mutated
by router tests (default_fallbacks, tag_budget_config, request_timeout,
enable_azure_ad_token_refresh, num_retries_per_request, model_cost,
token_counter). These were leaking across xdist workers.
Proxy tests: move test_proxy_utils.py (169 parametrized) and
test_proxy_server.py (72 parametrized) from part2 to part1, balancing
~370 vs ~360 tests (was ~129 vs ~600).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test intermittently fails in CI due to Redis cache write propagation
delays, causing the second call to miss the cache and hit OpenAI directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LLM responses are non-deterministic and ClaudeAgentOptions doesn't expose
temperature control. The model occasionally returns unexpected short responses
(e.g. just "!") instead of the expected greeting. Add up to 3 retries with a
more explicit prompt to make the test deterministic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test waited a fixed 3s for async callbacks to populate log_queue.
Under xdist -n 4, CPU contention can delay the GLOBAL_LOGGING_WORKER
background task beyond 3s. Replace fixed sleeps with polling loops
(up to 10s) that break as soon as the expected condition is met.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Module-level mutations (litellm.num_retries=3 in test_langfuse_e2e_test.py
and test_amazing_s3_logs.py, litellm.success_callback=['langfuse']) run
at import time, BEFORE any function fixture. The save/restore pattern
captured these polluted values as 'originals' and kept restoring them.
Fix: capture litellm defaults when conftest.py is first imported (before
test modules), then reset to those true defaults before each test instead
of saving/restoring the current (potentially polluted) state.
ThrottlingException is a transient AWS rate-limit error unrelated to code
correctness. Skip the test instead of failing the CI pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Clear _in_memory_loggers before/after each test to prevent cached logger
instances (LangsmithLogger, SlackAlerting, etc.) from leaking stale state
- Add pre_call_rules, post_call_rules to list attrs save/restore
- Add vector_store_registry to scalar attrs save/restore
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_avertex_batch_prediction: Add google.auth.default mock and env vars
so the test doesn't depend on real GCP credentials (was already a unit
test with mocked HTTP, just missing auth mock)
- test_async_create_batch[openai]: Add DNS pre-check that skips gracefully
when api.openai.com is unreachable instead of failing after 4 retries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The logging tests mutate many more litellm globals than guardrails tests
(turn_off_message_logging, s3_callback_params, datadog_params, service_callback,
etc.). The initial save/restore list only covered callbacks and a few basics,
causing state leaks like redaction settings bleeding across tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add LITELLM_LOG=WARNING to suppress verbose DEBUG log output
- Remove -s flag to stop capturing all stdout
- Bump xdist workers from -n 2 to -n 4
- Add --timeout=120 for safety
- Rewrite conftest.py to use save/restore pattern (matching guardrails_tests)
instead of per-function importlib.reload + event loop creation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The retrieve_batch endpoint sets batch status to "complete" but never set
batch_processed=True, permanently blocking file deletion. CheckBatchCost
(the safety net) also excluded completed batches from its primary query,
so batch_processed was never set by either path.
Three fixes:
1. update_batch_in_database sets batch_processed=True when status reaches
"complete", with old-schema fallback retry
2. CheckBatchCost primary query no longer excludes complete/completed
(batch_processed=False filter prevents reprocessing)
3. retrieve_batch early-return now includes "complete" (DB-normalized
spelling) to avoid unnecessary provider re-polls
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
