The tests were mocking `filter_server_ids_by_ip` but the production
code in server.py now calls `filter_server_ids_by_ip_with_info` which
returns a (server_ids, blocked_count) tuple. Update all 8 mock sites
to use the correct method name and return signature.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When calling non-text-embedding-3 models routed through the openai provider
(e.g. nvidia/llama-3.2-nv-embedqa-1b-v2), passing `dimensions` previously
raised an UnsupportedParamsError unconditionally. This fix threads
`allowed_openai_params` through the embedding call stack so that providers
can opt-in to passing `dimensions` by including it in the list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The old test assumed ArizePhoenixLogger reused the global TracerProvider.
With the nested traces fix, Phoenix now creates its own dedicated provider
and produces litellm_proxy_request + litellm_request + raw_gen_ai_request
spans independently.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- ArizePhoenixLogger now creates spans on its own dedicated TracerProvider
instead of trying to reuse parent spans from the global otel TracerProvider
(which were invisible in Phoenix since they go to a different exporter)
- Auto-initialize ArizePhoenixLogger when otel callback is configured and
Phoenix env vars (PHOENIX_API_KEY, PHOENIX_COLLECTOR_*) are detected
- Use exact type check in get_custom_logger_compatible_class to prevent
ArizePhoenixLogger (subclass) from being returned when looking up otel
- Fix tool_permission guardrail to check non-function tools like
code_interpreter (previously skipped with `type != "function"`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Adjust input and output cost per token for mistral-small-2503
Cost per million for mistral-small-2503 is not correct.
In Azure Documentation:
Pay-as-you-go (per 1,000 tokens)
$0.0001
Model input
$0.0003
Model output
* Update input and output cost per token for model
* feat(realtime guardrails): end_session_after_n_fails + Endpoint Settings wizard step
Adds per-session violation thresholds and an optional endpoint-settings step
to the guardrail wizard for /v1/realtime.
Backend:
- Add end_session_after_n_fails, on_violation, realtime_violation_message fields
to BaseLitellmParams (no DB migration — stored in existing JSON column)
- Store same fields on CustomGuardrail instance attrs
- Pass through in litellm_content_filter initializer
- Track _violation_count per RealTimeStreaming session; close backend_ws when
on_violation=end_session OR violation count >= end_session_after_n_fails
- Use realtime_violation_message as the spoken text (falls back to guardrail
error string if not configured)
UI (add_guardrail_form.tsx):
- Rename "Default Categories" step to "Topics"
- Add step 5 "Endpoint Settings (Optional)" for content filter guardrails
- Call type dropdown shows /v1/realtime
- Settings are in a collapsed accordion (closed by default)
- "End session after X violations" + on_violation radio + spoken message field
Tests: 2 new tests in test_realtime_streaming.py
- test_end_session_after_n_fails_closes_connection
- test_on_violation_end_session_closes_on_first_fail
* fix(test): move inline imports to module level in realtime streaming tests
* Update ui/litellm-dashboard/src/components/guardrails/add_guardrail_form.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(realtime): guardrails with pre_call/post_call mode now work on realtime WebSocket; return error directly to consumer
* fix(realtime guardrails): address code review feedback
- Restore session.update injection for audio/VAD path, but only when
realtime_input_transcription guardrails are configured (not pre_call).
Forward session.created to the client first so no error arrives before
the client sees the session.
- Change _swallow_next_response_create bool to int counter so consecutive
blocked items are handled correctly.
- Extract _build_litellm_metadata() helper to eliminate duplicated
metadata-building logic across OpenAI/Azure/XAI provider branches.
- Plumb litellm_metadata and user_api_key_dict to Azure and XAI handlers
so guardrails work for those providers too.
- Add tests for session.update injection, no-inject for pre_call-only,
and consecutive-block counter.
* simplify: remove response.create swallowing after guardrail block
When an item is blocked, the error event is already sent to the client.
The subsequent response.create from the client is fine to forward through —
the LLM may respond to previous context which is acceptable behavior.
Removing the swallow counter eliminates unnecessary state tracking.
* feat: add pretty view for realtime API logs in dashboard
- Create RealtimePrettyView component that renders structured session
config, conversation turns with transcripts, and token breakdowns
- Update PrettyMessagesView to detect realtime responses (via
isRealtimeResponse helper) and delegate to the new component
- Session card shows model, voice, modalities, temperature, instructions
in a collapsible panel
- Conversation turns show status, per-turn token usage, and audio/text
transcripts with appropriate icons
- Add 24 tests for RealtimePrettyView and 3 tests for PrettyMessagesView
- All 75 LogDetailsDrawer tests pass
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* chore: remove dev_config.yaml from tracked files
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat: show turn count in realtime pretty view session header and output header
- Add purple 'N turns' tag to Session card header for at-a-glance turn count
- Add 'Turns: N' to the Output section header next to tokens/cost
- Extend SectionHeader to accept optional turnCount prop
- Add 3 new tests for turn count display (singular, plural, output header)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: address Greptile review feedback
- Remove response.audio.done and conversation.item.created from
isRealtimeResponse() detection since the view doesn't render them;
prevents misleading fallback for responses with only those events
- Remove dead code: index >= 0 is always true in .map() callback
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(adapter): populate cache_read_input_tokens from prompt_tokens_details
The Anthropic adapter's translate_openai_response_to_anthropic checked
only the private _cache_read_input_tokens attr (set by Anthropic/DeepSeek)
but not prompt_tokens_details.cached_tokens (set by OpenAI/Azure).
Use prompt_tokens_details.cached_tokens directly — it is already extracted
and is the standard field populated by all providers.
Fixes#22089
* fix(adapter): apply same cache_read_input_tokens fix to streaming path
The streaming path in translate_streaming_openai_response_to_anthropic
had the same bug — relying on _cache_read_input_tokens instead of
prompt_tokens_details.cached_tokens.
* fix(proxy): improve auth exception logging levels and add structured context
Downgrade expected auth failures (ProxyException, HTTPException < 500,
BudgetExceededError) from ERROR to WARNING log level to reduce noise from
routine rejected requests (e.g. missing/invalid API keys on polled endpoints
like /schedule/model_cost_map_reload/status).
Unexpected exceptions and HTTPException with status >= 500 still log at
ERROR with full traceback.
Enrich log messages with structured context: route, HTTP method, masked
API key (using existing abbreviate_api_key), error type, and error code.
All fields also passed via log extra dict for log aggregation tools.
Fixes#21293
* Update tests/test_litellm/proxy/auth/test_auth_exception_handler.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(vertex_ai): add Vertex AI Gemini Live support via unified /realtime endpoint
Adds VertexAIRealtimeConfig which translates the OpenAI Realtime WebSocket
protocol to Vertex AI BidiGenerateContent. Supports voice in/voice out
(16 kHz mic → 24 kHz speaker) and text in/text out through the proxy's
/realtime endpoint.
Key changes:
- New litellm/llms/vertex_ai/realtime/transformation.py with VertexAIRealtimeConfig
- Builds correct wss:// URL (regional + global)
- OAuth2 Bearer token auth (not API key)
- Full model path (projects/.../publishers/google/models/...)
- Ignores session.update (Vertex AI only accepts one setup message)
- realtime_api/main.py: vertex_ai branch resolves OAuth token + constructs config
- llm_http_handler.py: auto-sends session setup before bidirectional_forward
- gemini/realtime/transformation.py: fix crashes on empty turnComplete events
- realtime_streaming.py: try/except guard so bad messages don't kill the loop
- proxy_server.py: add missing websockets.exceptions import
* docs: add vertex_realtime to sidebars
* fix: drop unknown event types in Gemini transform; add vertex_ai health check
* fix: propagate UUID fallback IDs from transform_content_done_event to return_additional_content_done_events
* fix: route guardrail backend sends through provider transform; fix str.strip misuse for model prefix
* fix: handle Vertex AI full resource path in session.created; route guardrail block sends through _send_to_backend
* fix: remove unused VertexBase in transformation.py; apply UUID fallback in return_additional_content_done_events
Adds a new block_code_execution guardrail that detects markdown fenced code blocks
in request/response content and blocks or masks them by language. Includes full
UI integration, type definitions, compliance test dataset, and 26 unit tests.
Key guardrail capabilities:
- Regex-based fenced code block detection with configurable blocked languages
- Confidence scoring with tunable threshold
- Execution-intent heuristics (request-side only) with conflict resolution
- Block or mask actions for detected code
- Support for pre_call, post_call, and during_call event hooks
Security hardening:
- Response-side blocking skips intent heuristics (LLM output doesn't contain
user intent phrases, so checking would silently disable post_call blocking)
- No-execution short-circuit includes conflict resolution: if both no-execution
and execution phrases match, execution intent wins
- Tightened overly broad phrases to prevent trivial bypass
- _normalize_escaped_newlines only applies to pure-escaped payloads to avoid
corrupting content that discusses escape sequences
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Replace the non-paginated Key Alias filter with a new PaginatedKeyAliasSelect component that mirrors the existing PaginatedModelSelect pattern. This aligns the UI with the paginated /key/aliases endpoint from PR #22137.
Changes:
- Added useInfiniteKeyAliases hook for paginated key alias fetching
- Created PaginatedKeyAliasSelect component with infinite scroll (80% threshold)
- Updated keyAliasesCall in networking to accept page/size/search params
- Replaced Key Alias filter in Request Logs and Virtual Keys tables to use customComponent
- Removed fetchAllKeyAliases helper and related upfront fetching logic
- Added 22 tests for new component and hook; all existing tests pass (54 tests)
Fixes the issue where the UI was fetching all key aliases at once, causing performance issues with large key sets.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat(realtime): add guardrails query param to /v1/realtime WebSocket endpoint
- Add 'guardrails' query param (comma-separated) to realtime_websocket_endpoint
- Import websockets and websockets.exceptions at module level (fixes NameError in except clause)
- Split try/except into Phase 1 (pre-call) and Phase 2 (routing) so guardrail
errors send back a typed error event before closing, while upstream errors
close silently with 1011
* feat(ui): pass selectedGuardrails from sidebar to RealtimePlayground WebSocket URL
* docs(realtime): add guardrails section with dynamic passing examples
* Add claims agent guardrails with 243-case eval dataset
5 new category guardrails for healthcare claims agent chatbots:
- claims_fraud_coaching: fraud coaching, exaggeration, document forgery
- claims_phi_disclosure: unauthorized PHI access, bulk data extraction
- claims_prior_auth_gaming: code manipulation, medical necessity misrepresentation
- claims_system_override: system injection, rule bypass, role impersonation
- claims_medical_advice: medical advice (claims-context-aware)
Plus claims_agent_safety.yaml policy template combining all 5.
All 5 eval suites pass at 100% precision/recall/F1 (243 test cases).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add claims agent chatbot safety policy template
Combines the 5 claims guardrails into a single deployable policy template:
fraud coaching, PHI disclosure, prior-auth gaming, system override, and medical advice.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add guardrail benchmark results and UI compliance prompts
Adds benchmark results for claims, discrimination, and content filter guardrails.
Updates UI compliance prompt data.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove duplicate "file an appeal" exception in claims_prior_auth_gaming.yaml
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove unused claims_agent_safety.yaml policy template
The claims-agent-safety template in policy_templates.json references
individual category files in categories/, not this combined file.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add post-call hook for Lakera guardrail and mask PII in responses
* Add post-call hook for Lakera and mask PII in responses
* Fix post-call hook: pass event_type to call_v2_guard
* Address Greptile review: return ModelResponse, fix mutation, add header, test location, mask order
- PII masking path: return ModelResponse instead of dict so deployment hook accepts it
- Avoid mutating request data: deep copy original_messages and messages in _mask_pii_in_messages
- Add guardrail header in PII-only return path
- Add test in tests/test_litellm/ (test_lakera_ai_v2.py) per PR checklist
- Sort PII payload spans by (start,end) descending so multiple spans in one message mask correctly
Co-authored-by: Cursor <cursoragent@cursor.com>
* Updated ponteital for index mismatch when choices have null content and inconsistent on_flagged access pattern
* Update litellm/proxy/guardrails/guardrail_hooks/lakera_ai_v2.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update to explicitly state supported endpoints - chat completions
* Fix minor lint error on masked_entity_count
---------
Co-authored-by: Steve <steve.giguere@lakera.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
