Root cause: LLMClientCache._remove_key() was calling aclose()/close() on
httpx clients when their cache entries expired (TTL) or were evicted (LRU).
However, callers of get_async_httpx_client() hold direct references to
these client objects (e.g. litellm.module_level_aclient, streaming handlers,
passthrough request handlers). Closing the underlying httpx client while
these references are still in use causes:
RuntimeError: Cannot send a request, as the client has been closed.
This was introduced in commits 6931fea9 and bfeed0e5 (Feb 20, 2026) which
added close-on-eviction to prevent connection pool leaks. The fix removes
the close-on-eviction behavior: when a cache entry is evicted, only the
cache's reference is dropped. Python GC will finalize transport/connection
pool resources when all external references are released.
The bug manifests after ~1 hour of traffic (_DEFAULT_TTL_FOR_HTTPX_CLIENTS
= 3600s) when cache entries expire, or earlier under high concurrency if
the 200-entry LRU limit is reached. It affects all providers using the
shared httpx client cache (Vertex AI, Bedrock, etc.).
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Adds TestProxyMcpStatelessBehavior to test_proxy_mcp_e2e.py with a test
that verifies two independent MCP clients can connect, initialize, and
call tools without sharing session state. This catches the regression
from PR #19809 where stateless=False broke clients that don't manage
mcp-session-id headers.
Regression test for #20242
The tests were mocking `filter_server_ids_by_ip` but the production
code in server.py now calls `filter_server_ids_by_ip_with_info` which
returns a (server_ids, blocked_count) tuple. Update all 8 mock sites
to use the correct method name and return signature.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The old test assumed ArizePhoenixLogger reused the global TracerProvider.
With the nested traces fix, Phoenix now creates its own dedicated provider
and produces litellm_proxy_request + litellm_request + raw_gen_ai_request
spans independently.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- ArizePhoenixLogger now creates spans on its own dedicated TracerProvider
instead of trying to reuse parent spans from the global otel TracerProvider
(which were invisible in Phoenix since they go to a different exporter)
- Auto-initialize ArizePhoenixLogger when otel callback is configured and
Phoenix env vars (PHOENIX_API_KEY, PHOENIX_COLLECTOR_*) are detected
- Use exact type check in get_custom_logger_compatible_class to prevent
ArizePhoenixLogger (subclass) from being returned when looking up otel
- Fix tool_permission guardrail to check non-function tools like
code_interpreter (previously skipped with `type != "function"`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(adapter): populate cache_read_input_tokens from prompt_tokens_details
The Anthropic adapter's translate_openai_response_to_anthropic checked
only the private _cache_read_input_tokens attr (set by Anthropic/DeepSeek)
but not prompt_tokens_details.cached_tokens (set by OpenAI/Azure).
Use prompt_tokens_details.cached_tokens directly — it is already extracted
and is the standard field populated by all providers.
Fixes#22089
* fix(adapter): apply same cache_read_input_tokens fix to streaming path
The streaming path in translate_streaming_openai_response_to_anthropic
had the same bug — relying on _cache_read_input_tokens instead of
prompt_tokens_details.cached_tokens.
* Adjust input and output cost per token for mistral-small-2503
Cost per million for mistral-small-2503 is not correct.
In Azure Documentation:
Pay-as-you-go (per 1,000 tokens)
$0.0001
Model input
$0.0003
Model output
* Update input and output cost per token for model
* fix(proxy): improve auth exception logging levels and add structured context
Downgrade expected auth failures (ProxyException, HTTPException < 500,
BudgetExceededError) from ERROR to WARNING log level to reduce noise from
routine rejected requests (e.g. missing/invalid API keys on polled endpoints
like /schedule/model_cost_map_reload/status).
Unexpected exceptions and HTTPException with status >= 500 still log at
ERROR with full traceback.
Enrich log messages with structured context: route, HTTP method, masked
API key (using existing abbreviate_api_key), error type, and error code.
All fields also passed via log extra dict for log aggregation tools.
Fixes#21293
* Update tests/test_litellm/proxy/auth/test_auth_exception_handler.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(realtime guardrails): end_session_after_n_fails + Endpoint Settings wizard step
Adds per-session violation thresholds and an optional endpoint-settings step
to the guardrail wizard for /v1/realtime.
Backend:
- Add end_session_after_n_fails, on_violation, realtime_violation_message fields
to BaseLitellmParams (no DB migration — stored in existing JSON column)
- Store same fields on CustomGuardrail instance attrs
- Pass through in litellm_content_filter initializer
- Track _violation_count per RealTimeStreaming session; close backend_ws when
on_violation=end_session OR violation count >= end_session_after_n_fails
- Use realtime_violation_message as the spoken text (falls back to guardrail
error string if not configured)
UI (add_guardrail_form.tsx):
- Rename "Default Categories" step to "Topics"
- Add step 5 "Endpoint Settings (Optional)" for content filter guardrails
- Call type dropdown shows /v1/realtime
- Settings are in a collapsed accordion (closed by default)
- "End session after X violations" + on_violation radio + spoken message field
Tests: 2 new tests in test_realtime_streaming.py
- test_end_session_after_n_fails_closes_connection
- test_on_violation_end_session_closes_on_first_fail
* fix(test): move inline imports to module level in realtime streaming tests
* Update ui/litellm-dashboard/src/components/guardrails/add_guardrail_form.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(realtime): guardrails with pre_call/post_call mode now work on realtime WebSocket; return error directly to consumer
* fix(realtime guardrails): address code review feedback
- Restore session.update injection for audio/VAD path, but only when
realtime_input_transcription guardrails are configured (not pre_call).
Forward session.created to the client first so no error arrives before
the client sees the session.
- Change _swallow_next_response_create bool to int counter so consecutive
blocked items are handled correctly.
- Extract _build_litellm_metadata() helper to eliminate duplicated
metadata-building logic across OpenAI/Azure/XAI provider branches.
- Plumb litellm_metadata and user_api_key_dict to Azure and XAI handlers
so guardrails work for those providers too.
- Add tests for session.update injection, no-inject for pre_call-only,
and consecutive-block counter.
* simplify: remove response.create swallowing after guardrail block
When an item is blocked, the error event is already sent to the client.
The subsequent response.create from the client is fine to forward through —
the LLM may respond to previous context which is acceptable behavior.
Removing the swallow counter eliminates unnecessary state tracking.
* feat: add pretty view for realtime API logs in dashboard
- Create RealtimePrettyView component that renders structured session
config, conversation turns with transcripts, and token breakdowns
- Update PrettyMessagesView to detect realtime responses (via
isRealtimeResponse helper) and delegate to the new component
- Session card shows model, voice, modalities, temperature, instructions
in a collapsible panel
- Conversation turns show status, per-turn token usage, and audio/text
transcripts with appropriate icons
- Add 24 tests for RealtimePrettyView and 3 tests for PrettyMessagesView
- All 75 LogDetailsDrawer tests pass
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* chore: remove dev_config.yaml from tracked files
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat: show turn count in realtime pretty view session header and output header
- Add purple 'N turns' tag to Session card header for at-a-glance turn count
- Add 'Turns: N' to the Output section header next to tokens/cost
- Extend SectionHeader to accept optional turnCount prop
- Add 3 new tests for turn count display (singular, plural, output header)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: address Greptile review feedback
- Remove response.audio.done and conversation.item.created from
isRealtimeResponse() detection since the view doesn't render them;
prevents misleading fallback for responses with only those events
- Remove dead code: index >= 0 is always true in .map() callback
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(vertex_ai): add Vertex AI Gemini Live support via unified /realtime endpoint
Adds VertexAIRealtimeConfig which translates the OpenAI Realtime WebSocket
protocol to Vertex AI BidiGenerateContent. Supports voice in/voice out
(16 kHz mic → 24 kHz speaker) and text in/text out through the proxy's
/realtime endpoint.
Key changes:
- New litellm/llms/vertex_ai/realtime/transformation.py with VertexAIRealtimeConfig
- Builds correct wss:// URL (regional + global)
- OAuth2 Bearer token auth (not API key)
- Full model path (projects/.../publishers/google/models/...)
- Ignores session.update (Vertex AI only accepts one setup message)
- realtime_api/main.py: vertex_ai branch resolves OAuth token + constructs config
- llm_http_handler.py: auto-sends session setup before bidirectional_forward
- gemini/realtime/transformation.py: fix crashes on empty turnComplete events
- realtime_streaming.py: try/except guard so bad messages don't kill the loop
- proxy_server.py: add missing websockets.exceptions import
* docs: add vertex_realtime to sidebars
* fix: drop unknown event types in Gemini transform; add vertex_ai health check
* fix: propagate UUID fallback IDs from transform_content_done_event to return_additional_content_done_events
* fix: route guardrail backend sends through provider transform; fix str.strip misuse for model prefix
* fix: handle Vertex AI full resource path in session.created; route guardrail block sends through _send_to_backend
* fix: remove unused VertexBase in transformation.py; apply UUID fallback in return_additional_content_done_events
