A database timeout (httpcore.ReadTimeout) during get_config() in
_update_llm_router would propagate and prevent ALL DB models from
loading into the router. This was the root cause of a customer issue
where 51 DB models were invisible despite valid data and correct
encryption keys.
Now get_config() failures are caught separately so model
add/delete operations still proceed. Similarly, _delete_deployment
catches get_config failures and safely skips cleanup rather than
crashing the entire sync cycle.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Greptile review caught a pagination bug: _get_db_only_models used
len(all_models) (full unfiltered router list) for the take calculation.
When searching with a large router (e.g. 50 models, page size 50), the
take would be 0 even though only 1 router model matched the search,
causing DB-only models to never be fetched.
Fix: accept filtered_router_count parameter and use it for pagination.
Added regression test with 50 router models where only 1 matches search.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When models are added via the UI, they are saved to the database and then
synced into the router. If the sync fails silently (e.g. due to decryption
errors, invalid params with ignore_invalid_deployments=True), the models
become invisible in the /v2/model/info response because the endpoint only
reads from llm_router.model_list.
The search path already queried the DB as a fallback, but the non-search
path did not. This fix extracts the DB-query logic into _get_db_only_models()
and calls it in all cases, ensuring DB models that failed to load into the
router still appear in the model management UI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds TestProxyMcpStatelessBehavior to test_proxy_mcp_e2e.py with a test
that verifies two independent MCP clients can connect, initialize, and
call tools without sharing session state. This catches the regression
from PR #19809 where stateless=False broke clients that don't manage
mcp-session-id headers.
Regression test for #20242
The tests were mocking `filter_server_ids_by_ip` but the production
code in server.py now calls `filter_server_ids_by_ip_with_info` which
returns a (server_ids, blocked_count) tuple. Update all 8 mock sites
to use the correct method name and return signature.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The old test assumed ArizePhoenixLogger reused the global TracerProvider.
With the nested traces fix, Phoenix now creates its own dedicated provider
and produces litellm_proxy_request + litellm_request + raw_gen_ai_request
spans independently.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- ArizePhoenixLogger now creates spans on its own dedicated TracerProvider
instead of trying to reuse parent spans from the global otel TracerProvider
(which were invisible in Phoenix since they go to a different exporter)
- Auto-initialize ArizePhoenixLogger when otel callback is configured and
Phoenix env vars (PHOENIX_API_KEY, PHOENIX_COLLECTOR_*) are detected
- Use exact type check in get_custom_logger_compatible_class to prevent
ArizePhoenixLogger (subclass) from being returned when looking up otel
- Fix tool_permission guardrail to check non-function tools like
code_interpreter (previously skipped with `type != "function"`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(adapter): populate cache_read_input_tokens from prompt_tokens_details
The Anthropic adapter's translate_openai_response_to_anthropic checked
only the private _cache_read_input_tokens attr (set by Anthropic/DeepSeek)
but not prompt_tokens_details.cached_tokens (set by OpenAI/Azure).
Use prompt_tokens_details.cached_tokens directly — it is already extracted
and is the standard field populated by all providers.
Fixes#22089
* fix(adapter): apply same cache_read_input_tokens fix to streaming path
The streaming path in translate_streaming_openai_response_to_anthropic
had the same bug — relying on _cache_read_input_tokens instead of
prompt_tokens_details.cached_tokens.
* fix(proxy): improve auth exception logging levels and add structured context
Downgrade expected auth failures (ProxyException, HTTPException < 500,
BudgetExceededError) from ERROR to WARNING log level to reduce noise from
routine rejected requests (e.g. missing/invalid API keys on polled endpoints
like /schedule/model_cost_map_reload/status).
Unexpected exceptions and HTTPException with status >= 500 still log at
ERROR with full traceback.
Enrich log messages with structured context: route, HTTP method, masked
API key (using existing abbreviate_api_key), error type, and error code.
All fields also passed via log extra dict for log aggregation tools.
Fixes#21293
* Update tests/test_litellm/proxy/auth/test_auth_exception_handler.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(realtime guardrails): end_session_after_n_fails + Endpoint Settings wizard step
Adds per-session violation thresholds and an optional endpoint-settings step
to the guardrail wizard for /v1/realtime.
Backend:
- Add end_session_after_n_fails, on_violation, realtime_violation_message fields
to BaseLitellmParams (no DB migration — stored in existing JSON column)
- Store same fields on CustomGuardrail instance attrs
- Pass through in litellm_content_filter initializer
- Track _violation_count per RealTimeStreaming session; close backend_ws when
on_violation=end_session OR violation count >= end_session_after_n_fails
- Use realtime_violation_message as the spoken text (falls back to guardrail
error string if not configured)
UI (add_guardrail_form.tsx):
- Rename "Default Categories" step to "Topics"
- Add step 5 "Endpoint Settings (Optional)" for content filter guardrails
- Call type dropdown shows /v1/realtime
- Settings are in a collapsed accordion (closed by default)
- "End session after X violations" + on_violation radio + spoken message field
Tests: 2 new tests in test_realtime_streaming.py
- test_end_session_after_n_fails_closes_connection
- test_on_violation_end_session_closes_on_first_fail
* fix(test): move inline imports to module level in realtime streaming tests
* Update ui/litellm-dashboard/src/components/guardrails/add_guardrail_form.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(realtime): guardrails with pre_call/post_call mode now work on realtime WebSocket; return error directly to consumer
* fix(realtime guardrails): address code review feedback
- Restore session.update injection for audio/VAD path, but only when
realtime_input_transcription guardrails are configured (not pre_call).
Forward session.created to the client first so no error arrives before
the client sees the session.
- Change _swallow_next_response_create bool to int counter so consecutive
blocked items are handled correctly.
- Extract _build_litellm_metadata() helper to eliminate duplicated
metadata-building logic across OpenAI/Azure/XAI provider branches.
- Plumb litellm_metadata and user_api_key_dict to Azure and XAI handlers
so guardrails work for those providers too.
- Add tests for session.update injection, no-inject for pre_call-only,
and consecutive-block counter.
* simplify: remove response.create swallowing after guardrail block
When an item is blocked, the error event is already sent to the client.
The subsequent response.create from the client is fine to forward through —
the LLM may respond to previous context which is acceptable behavior.
Removing the swallow counter eliminates unnecessary state tracking.
* feat(vertex_ai): add Vertex AI Gemini Live support via unified /realtime endpoint
Adds VertexAIRealtimeConfig which translates the OpenAI Realtime WebSocket
protocol to Vertex AI BidiGenerateContent. Supports voice in/voice out
(16 kHz mic → 24 kHz speaker) and text in/text out through the proxy's
/realtime endpoint.
Key changes:
- New litellm/llms/vertex_ai/realtime/transformation.py with VertexAIRealtimeConfig
- Builds correct wss:// URL (regional + global)
- OAuth2 Bearer token auth (not API key)
- Full model path (projects/.../publishers/google/models/...)
- Ignores session.update (Vertex AI only accepts one setup message)
- realtime_api/main.py: vertex_ai branch resolves OAuth token + constructs config
- llm_http_handler.py: auto-sends session setup before bidirectional_forward
- gemini/realtime/transformation.py: fix crashes on empty turnComplete events
- realtime_streaming.py: try/except guard so bad messages don't kill the loop
- proxy_server.py: add missing websockets.exceptions import
* docs: add vertex_realtime to sidebars
* fix: drop unknown event types in Gemini transform; add vertex_ai health check
* fix: propagate UUID fallback IDs from transform_content_done_event to return_additional_content_done_events
* fix: route guardrail backend sends through provider transform; fix str.strip misuse for model prefix
* fix: handle Vertex AI full resource path in session.created; route guardrail block sends through _send_to_backend
* fix: remove unused VertexBase in transformation.py; apply UUID fallback in return_additional_content_done_events
Adds a new block_code_execution guardrail that detects markdown fenced code blocks
in request/response content and blocks or masks them by language. Includes full
UI integration, type definitions, compliance test dataset, and 26 unit tests.
Key guardrail capabilities:
- Regex-based fenced code block detection with configurable blocked languages
- Confidence scoring with tunable threshold
- Execution-intent heuristics (request-side only) with conflict resolution
- Block or mask actions for detected code
- Support for pre_call, post_call, and during_call event hooks
Security hardening:
- Response-side blocking skips intent heuristics (LLM output doesn't contain
user intent phrases, so checking would silently disable post_call blocking)
- No-execution short-circuit includes conflict resolution: if both no-execution
and execution phrases match, execution intent wins
- Tightened overly broad phrases to prevent trivial bypass
- _normalize_escaped_newlines only applies to pure-escaped payloads to avoid
corrupting content that discusses escape sequences
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Replace Prisma ORM count/find_many calls with two query_raw calls that
only project the key_alias column. The Prisma client wrapper does not
support SELECT projection via find_many, so raw SQL is used to keep
memory usage proportional to the page size rather than total key count.
Update tests to mock query_raw instead of count/find_many.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
LiteLLM_VerificationTokenActions.find_many() does not support the
select keyword argument. Remove it and drop the corresponding test.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Add select={"key_alias": True} to the find_many call so only the alias
column is fetched from the database instead of full token rows. Add
five unit tests in test_key_management_endpoints.py covering response
shape, pagination skip/take computation, search filter injection,
absence of contains filter when no search term is given, and the
select-only-alias optimization.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
When calling non-text-embedding-3 models routed through the openai provider
(e.g. nvidia/llama-3.2-nv-embedqa-1b-v2), passing `dimensions` previously
raised an UnsupportedParamsError unconditionally. This fix threads
`allowed_openai_params` through the embedding call stack so that providers
can opt-in to passing `dimensions` by including it in the list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
