* fix: Fixes https://github.com/BerriAI/litellm/issues/23185
* fix(responses/main.py): ensure litellm metadata custom cost works
* refactor: move all logging updates to a common function, to have just 1 place to update logging kwarg updates
- realtime_api/main.py: Revert param types to Dict, explicitly construct
RealtimeSessionConfig/RealtimeExpiresAfter before passing to
RealtimeClientSecretRequest
- presidio.py: Move type:ignore[override] to def line where mypy reports it
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- realtime_api/main.py: Widen param types to accept both Dict and Pydantic models
- proxy/realtime_endpoints/endpoints.py: Widen return type to Union[..., Response]
- proxy/guardrails/guardrail_hooks/presidio.py: Add type:ignore[override] for bytes in streaming return
- proxy/_experimental/mcp_server/rest_endpoints.py: Annotate _oauth2_flow with Literal type
- proxy/management_endpoints/ui_sso.py: Add httpx import under TYPE_CHECKING, remove invalid timeout kwarg from AsyncHTTPHandler.get()
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Thread api_version through HTTP handlers to Azure realtime endpoints
- Make expires_at optional in RealtimeClientSecretResponse
- Fix test token expiry times to be in the future
- Populate user_id and team_id in minimal_auth for spend tracking
Made-with: Cursor
* fix(image_generation): propagate extra_headers to OpenAI image generation
Add headers parameter to image_generation() and aimage_generation() methods
in OpenAI provider, and pass headers from images/main.py to ensure custom
headers like cf-aig-authorization are properly forwarded to the OpenAI API.
Aligns behavior with completion() method and Azure provider implementation.
* test(image_generation): add tests for extra_headers propagation
Verify that extra_headers are correctly forwarded to OpenAI's
images.generate() in both sync and async paths, and that they
are absent when not provided.
* Add Prometheus child_exit cleanup for gunicorn workers
When a gunicorn worker exits (e.g. from max_requests recycling), its
per-process prometheus .db files remain on disk. For gauges using
livesum/liveall mode, this means the dead worker's last-known values
persist as if the process were still alive. Wire gunicorn's child_exit
hook to call mark_process_dead() so live-tracking gauges accurately
reflect only running workers.
* docs: update AssemblyAI docs with Universal-3 Pro, Speech Understanding, and LLM Gateway (#21130)
* docs: update AssemblyAI docs with Universal-3 Pro, Speech Understanding, and LLM Gateway provider config
* feat: add AssemblyAI LLM Gateway as OpenAI-compatible provider
* fix(mcp): update test mocks to use renamed filter_server_ids_by_ip_with_info
Tests were mocking the old method name `filter_server_ids_by_ip` but production
code at server.py:774 calls `filter_server_ids_by_ip_with_info` which returns
a (server_ids, blocked_count) tuple. The unmocked method on AsyncMock returned
a coroutine, causing "cannot unpack non-iterable coroutine object" errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): update realtime guardrail test assertions for voice violation behavior
Tests were asserting no response.create/conversation.item.create sent to
backend when guardrail blocks, but the implementation intentionally sends
these to have the LLM voice the guardrail violation message to the user.
Updated assertions to verify the correct guardrail flow:
- response.cancel is sent to stop any in-progress response
- conversation.item.create with violation message is injected
- response.create is sent to voice the violation
- original blocked content is NOT forwarded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bedrock): restore parallel_tool_calls mapping in map_openai_params
The revert in 8565c70e53 removed the parallel_tool_calls handling from
map_openai_params, and the subsequent fix d0445e1e33 only re-added the
transform_request consumption but forgot to re-add the map_openai_params
producer that sets _parallel_tool_use_config. This meant parallel_tool_calls
was silently ignored for all Bedrock models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): update Azure pass-through test to mock litellm.completion
Commit 99c62ca40e removed "azure" from _RESPONSES_API_PROVIDERS,
routing Azure models through litellm.completion instead of
litellm.responses. The test was not updated to match, causing it
to assert against the wrong mock.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add in_flight_requests metric to /health/backlog + prometheus (#22319)
* feat: add in_flight_requests metric to /health/backlog + prometheus
* refactor: clean class with static methods, add tests, fix sentinel pattern
* docs: add in_flight_requests to prometheus metrics and latency troubleshooting
* fix(db): add missing migration for LiteLLM_ClaudeCodePluginTable
PR #22271 added the LiteLLM_ClaudeCodePluginTable model to
schema.prisma but did not include a corresponding migration file,
causing test_aaaasschema_migration_check to fail.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update stale docstring to match guardrail voicing behavior
Addresses Greptile review feedback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(caching): store background task references in LLMClientCache._remove_key to prevent unawaited coroutine warnings
Fixes#22128
* [Feat] Agent RBAC Permission Fix - Ensure Internal Users cannot create agents (#22329)
* fix: enforce RBAC on agent endpoints — block non-admin create/update/delete
- Add /v1/agents/{agent_id} to agent_routes so internal users can
access GET-by-ID (previously returned 403 due to missing route pattern)
- Add _check_agent_management_permission() guard to POST, PUT, PATCH,
DELETE agent endpoints — only PROXY_ADMIN may mutate agents
- Add user_api_key_dict param to delete_agent so the role check works
- Add comprehensive unit tests for RBAC enforcement across all roles
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: mock prisma_client in internal user get-agent-by-id test
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(ui): hide agent create/delete controls for non-admin users
Match MCP servers pattern: wrap '+ Add New Agent' button in
isAdmin conditional so internal users see a read-only agents view.
Delete buttons in card and table were already gated.
Update empty-state copy for non-admin users.
Add 7 Vitest tests covering role-based visibility.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: Add PROXY_ADMIN role to system user for key rotation (#21896)
* fix: Add PROXY_ADMIN role to system user for key rotation
The key rotation worker was failing with 'You are not authorized to regenerate this key'
when rotating team keys. This was because the system user created by
get_litellm_internal_jobs_user_api_key_auth() was missing the user_role field.
Without user_role=PROXY_ADMIN, the system user couldn't bypass team permission checks
in can_team_member_execute_key_management_endpoint(), causing authorization failures
for team key rotation.
This fix adds user_role=LitellmUserRoles.PROXY_ADMIN to the system user, allowing
it to bypass team permission checks and successfully rotate keys for all teams.
* test: Add unit test for system user PROXY_ADMIN role
- Verify internal jobs system user has PROXY_ADMIN role
- Critical for key rotation to bypass team permission checks
- Regression test for PR #21896
* fix: populate user_id and user_info for admin users in /user/info (#22239)
* fix: populate user_id and user_info for admin users in /user/info endpoint
Fixes#22179
When admin users call /user/info without a user_id parameter, the endpoint
was returning null for both user_id and user_info fields. This broke
budgeting tooling that relies on /user/info to look up current budget and spend.
Changes:
- Modified _get_user_info_for_proxy_admin() to accept user_api_key_dict parameter
- Added logic to fetch admin's own user info from database
- Updated function to return admin's user_id and user_info instead of null
- Updated unit test to verify admin user_id is populated
The fix ensures admin users get their own user information just like regular users.
* test: make mock get_data signature match real method
- Updated MockPrismaClientDB.get_data() to accept all parameters that the real method accepts
- Makes mock more robust against future refactors
- Added datetime and Union imports
- Mock now returns None when user_id is not provided
* [Fix] Pass MCP auth headers from request into tool fetch for /v1/responses and chat completions (#22291)
* fixed dynamic auth for /responses with mcp
* fixed greptile concern
* fix(bedrock): filter internal json_tool_call when mixed with real tools
Fixes#18381: When using both tools and response_format with Bedrock
Converse API, LiteLLM internally adds json_tool_call to handle structured
output. Bedrock may return both this internal tool AND real user-defined
tools, breaking consumers like OpenAI Agents SDK.
Changes:
- Non-streaming: Added _filter_json_mode_tools() to handle 3 scenarios:
only json_tool_call (convert to content), mixed (filter it out), or
no json_tool_call (pass through)
- Streaming: Added json_mode tracking to AWSEventStreamDecoder to suppress
json_tool_call chunks and convert to text content
- Fixed optional_params.pop() mutation issue
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* refactor: extract duplicated JSON unwrapping into helper method
Addresses review comment from greptile-apps:
https://github.com/BerriAI/litellm/pull/21107#pullrequestreview-3796085353
Changes:
- Added `_unwrap_bedrock_properties()` helper method to eliminate code duplication
- Replaced two identical JSON unwrapping blocks (lines 1592-1601 and 1612-1620)
with calls to the new helper method
- Improves maintainability - single source of truth for Bedrock properties unwrapping logic
The helper method:
- Parses JSON string
- Checks for single "properties" key structure
- Unwraps and returns the properties value
- Returns original string if unwrapping not needed or parsing fails
No functional changes - pure refactoring.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: use correct class name AmazonConverseConfig in helper method calls
Fixed MyPy errors where BedrockConverseConfig was used instead of
AmazonConverseConfig in the _unwrap_bedrock_properties() calls.
Errors:
- Line 1619: BedrockConverseConfig -> AmazonConverseConfig
- Line 1631: BedrockConverseConfig -> AmazonConverseConfig
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* fix: shorten guardrail benchmark result filenames for Windows long path support
Fixes#21941
The generated result filenames from _save_confusion_results contained
parentheses, dots, and full yaml filenames, producing paths that exceed
the Windows 260-char MAX_PATH limit. Rework the safe_label logic to
produce short {topic}_{method_abbrev} filenames (e.g. insults_cf.json)
while preserving the full label inside the JSON content.
Rename existing tracked result files to match the new naming convention.
* Update litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/guardrail_benchmarks/test_eval.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Remove Apache 2 license from SKILL.md (#22322)
* fix(mcp): default available_on_public_internet to true (#22331)
* fix(mcp): default available_on_public_internet to true
MCPs were defaulting to private (available_on_public_internet=false) which
was a breaking change. This reverts the default to public (true) across:
- Pydantic models (AddMCPServerRequest, UpdateMCPServerRequest, LiteLLM_MCPServerTable)
- Prisma schema @default
- mcp_server_manager.py YAML config + DB loading fallbacks
- UI form initialValue and setFieldValue defaults
* fix(ui): add forceRender to Collapse.Panel so toggle defaults render correctly
Ant Design's Collapse.Panel lazy-renders children by default. Without
forceRender, the Form.Item for 'Available on Public Internet' isn't
mounted when the useEffect fires form.setFieldValue, causing the Switch
to visually show OFF even though the intended default is true.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mcp): update remaining schema copies and MCPServer type default to true
Missed in previous commit per Greptile review:
- schema.prisma (root)
- litellm-proxy-extras/litellm_proxy_extras/schema.prisma
- litellm/types/mcp_server/mcp_server_manager.py MCPServer class
* ui(mcp): reframe network access as 'Internal network only' restriction
Replace scary 'Available on Public Internet' toggle with 'Internal network only'
opt-in restriction. Toggle OFF (default) = all networks allowed. Toggle ON =
restricted to internal network only. Auth is always required either way.
- MCPPermissionManagement: new label/tooltip/description, invert display via
getValueProps/getValueFromEvent so underlying available_on_public_internet
value is unchanged
- mcp_server_view: 'Public' → 'All networks', 'Internal' → 'Internal only' (orange)
- mcp_server_columns: same badge updates
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(jwt): OIDC discovery URLs, roles array handling, dot-notation error hints (#22336)
* fix(jwt): support OIDC discovery URLs, handle roles array, improve error hints
Three fixes for Azure AD JWT auth:
1. OIDC discovery URL support - JWT_PUBLIC_KEY_URL can now be set to
.well-known/openid-configuration endpoints. The proxy fetches the
discovery doc, extracts jwks_uri, and caches it.
2. Handle roles claim as array - when team_id_jwt_field points to a list
(e.g. AAD's "roles": ["team1"]), auto-unwrap the first element instead
of crashing with 'unhashable type: list'.
3. Better error hint for dot-notation indexing - when team_id_jwt_field is
set to "roles.0" or "roles[0]", the 401 error now explains to use
"roles" instead and that LiteLLM auto-unwraps lists.
* Add integration demo script for JWT auth fixes (OIDC discovery, array roles, dot-notation hints)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add demo_servers.py for manual JWT auth testing with mock JWKS/OIDC endpoints
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add demo screenshots for PR comment
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add integration test results with screenshots for PR review
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* address greptile review feedback (greploop iteration 1)
- fix: add HTTP status code check in _resolve_jwks_url before parsing JSON
- fix: remove misleading bracket-notation hint from debug log (get_nested_value does not support it)
* Update tests/test_litellm/proxy/auth/test_handle_jwt.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove demo scripts and assets
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* perf: streaming latency improvements — 4 targeted hot-path fixes (#22346)
* perf: raise aiohttp connection pool limits (300→1000, 50/host→500)
* perf: skip model_copy() on every chunk — only copy usage-bearing chunks
* perf: replace list+join O(n²) with str+= O(n) in async_data_generator
* perf: cache model-level guardrail lookup per request, not per chunk
* test: add comprehensive Vitest coverage for CostTrackingSettings
Add 88 tests across 9 test files for the CostTrackingSettings component directory:
- provider_display_helpers.test.ts: 9 tests for helper functions
- how_it_works.test.tsx: 9 tests for discount calculator component
- add_provider_form.test.tsx: 7 tests for provider form validation
- add_margin_form.test.tsx: 9 tests for margin form with type toggle
- provider_discount_table.test.tsx: 12 tests for table editing and interactions
- provider_margin_table.test.tsx: 13 tests for margin table with sorting
- use_discount_config.test.ts: 11 tests for discount hook logic
- use_margin_config.test.ts: 12 tests for margin hook logic
- cost_tracking_settings.test.tsx: 15 tests for main component and role-based rendering
All tests passing. Coverage includes form validation, user interactions, API calls, state management, and conditional rendering.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* [Feature] Key list endpoint: Add project_id and access_group_id filters
Add filtering capabilities to /key/list endpoint for project_id and access_group_id parameters. Both filters work globally across all visibility rules and stack with existing sort/pagination params. Added comprehensive unit tests for the new filters.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* [Feature] UI - Projects: Add Project Details page with Edit modal
- Add ProjectDetailsPage with header, details card, spend/budget progress,
model spend bar chart, keys placeholder, and team info card
- Refactor CreateProjectModal into base form pattern (ProjectBaseForm)
shared between Create and Edit flows
- Add EditProjectModal with pre-filled form data from backend
- Add useProjectDetails and useUpdateProject hooks
- Add duplicate key validation for model limits and metadata
- Wire project ID click in table to navigate to detail view
- Move pagination inline with search bar
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Update ui/litellm-dashboard/src/components/Projects/ProjectModals/CreateProjectModal.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(azure): forward realtime_protocol from config and relax api_version check for GA path
The realtime_protocol parameter set in config.yaml litellm_params was
not reliably reaching the Azure realtime handler. Add fallback chain:
kwargs → litellm_params → LITELLM_AZURE_REALTIME_PROTOCOL env var → beta.
Also relax the api_version validation to only require it for the beta
protocol path, since the GA/v1 path does not use api_version in the URL.
Make protocol matching case-insensitive so 'ga', 'GA', 'v1', 'V1' all
work consistently. Fix _construct_url type signature to accept Optional
api_version.
Fixes#22127
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Zero Clover <zero@root.me>
Co-authored-by: Ryan Crabbe <rcrabbe@berkeley.edu>
Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com>
Co-authored-by: Dylan Duan <dylan.duan@assemblyai.com>
Co-authored-by: Julio Quinteros Pro <jquinter@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Shivaang <shivaang.05@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com>
Co-authored-by: Brian Caswell <bcaswell@microsoft.com>
Co-authored-by: Brian Caswell <bcaswell@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: rasmi <rrelasmar@gmail.com>
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(gemini): enable inputAudioTranscription and handle transcription events for realtime guardrails
Gemini sends inputTranscription/outputTranscription inside serverContent separately from modelTurn/turnComplete. This adds handling to convert them into OpenAI-compatible events so the guardrail pipeline can inspect voice input, and enables inputAudioTranscription in the session setup config.
Made-with: Cursor
* fix(vertex_ai): enable inputAudioTranscription in realtime session config
Add inputAudioTranscription to the Vertex AI realtime setup so the backend returns transcripts of user speech, allowing guardrails to inspect voice input.
Made-with: Cursor
* fix(realtime): pass user_api_key_dict and guardrail metadata through async_realtime handler
The base LLM HTTP handler's async_realtime method was not accepting or forwarding user_api_key_dict and litellm_metadata to RealTimeStreaming. This meant guardrails configured with default_on=false were silently skipped for all provider_config-based realtime connections (Gemini, Vertex AI, etc). Also fixes wss:// connections when SSL_VERIFY=False by overriding ssl=False for secure WebSocket URLs.
Made-with: Cursor
* fix(realtime): forward guardrail metadata for generic provider_config and vertex_ai paths
The _arealtime function was not passing user_api_key_dict or litellm_metadata to base_llm_http_handler.async_realtime() for the generic provider_config path and the vertex_ai-specific path. This broke guardrail resolution since RealTimeStreaming.request_data was empty, causing should_run_guardrail to return False.
Made-with: Cursor
* fix(realtime): voice guardrail responses and block duplicate response.create on text input
When a guardrail blocks voice input, send a conversation.item.create + response.create to the backend so the LLM voices the guardrail message as audio instead of only returning text. Also adds pending_guardrail_message tracking to suppress the automatic response.create the client sends after a blocked text message, and broadens _has_audio_transcription_guardrails to match pre_call/post_call modes.
Made-with: Cursor
* test(realtime): update guardrail tests for broadened audio transcription check and add integration tests
Update existing tests to reflect that pre_call guardrails now correctly trigger the audio/VAD session.update injection. Add integration test file for live OpenAI realtime guardrail testing.
Made-with: Cursor
* fix(realtime): instruct LLM to say exact guardrail message verbatim
The previous prompt gave the LLM creative freedom to paraphrase the guardrail violation message. Now it instructs the LLM to repeat the exact configured message word for word.
Made-with: Cursor
* fix(realtime): preserve wss ssl semantics and move live guardrail test
Keep TLS enabled for wss realtime sessions while honoring SSL_VERIFY=False via a no-verify SSLContext, move the OpenAI live guardrail test into llm_translation, and dedupe duplicated guardrail-detection helpers to prevent drift.
Made-with: Cursor
* fix(realtime): guardrails with pre_call/post_call mode now work on realtime WebSocket; return error directly to consumer
* fix(realtime guardrails): address code review feedback
- Restore session.update injection for audio/VAD path, but only when
realtime_input_transcription guardrails are configured (not pre_call).
Forward session.created to the client first so no error arrives before
the client sees the session.
- Change _swallow_next_response_create bool to int counter so consecutive
blocked items are handled correctly.
- Extract _build_litellm_metadata() helper to eliminate duplicated
metadata-building logic across OpenAI/Azure/XAI provider branches.
- Plumb litellm_metadata and user_api_key_dict to Azure and XAI handlers
so guardrails work for those providers too.
- Add tests for session.update injection, no-inject for pre_call-only,
and consecutive-block counter.
* simplify: remove response.create swallowing after guardrail block
When an item is blocked, the error event is already sent to the client.
The subsequent response.create from the client is fine to forward through —
the LLM may respond to previous context which is acceptable behavior.
Removing the swallow counter eliminates unnecessary state tracking.
* feat(vertex_ai): add Vertex AI Gemini Live support via unified /realtime endpoint
Adds VertexAIRealtimeConfig which translates the OpenAI Realtime WebSocket
protocol to Vertex AI BidiGenerateContent. Supports voice in/voice out
(16 kHz mic → 24 kHz speaker) and text in/text out through the proxy's
/realtime endpoint.
Key changes:
- New litellm/llms/vertex_ai/realtime/transformation.py with VertexAIRealtimeConfig
- Builds correct wss:// URL (regional + global)
- OAuth2 Bearer token auth (not API key)
- Full model path (projects/.../publishers/google/models/...)
- Ignores session.update (Vertex AI only accepts one setup message)
- realtime_api/main.py: vertex_ai branch resolves OAuth token + constructs config
- llm_http_handler.py: auto-sends session setup before bidirectional_forward
- gemini/realtime/transformation.py: fix crashes on empty turnComplete events
- realtime_streaming.py: try/except guard so bad messages don't kill the loop
- proxy_server.py: add missing websockets.exceptions import
* docs: add vertex_realtime to sidebars
* fix: drop unknown event types in Gemini transform; add vertex_ai health check
* fix: propagate UUID fallback IDs from transform_content_done_event to return_additional_content_done_events
* fix: route guardrail backend sends through provider transform; fix str.strip misuse for model prefix
* fix: handle Vertex AI full resource path in session.created; route guardrail block sends through _send_to_backend
* fix: remove unused VertexBase in transformation.py; apply UUID fallback in return_additional_content_done_events
* feat(realtime): add guardrail hook for voice transcription in Realtime API
Adds a new `realtime_input_transcription` guardrail event hook that fires
after Whisper transcription completes, before the LLM generates a response.
When a guardrail blocks, a synthetic warning is sent to the client and
`response.create` is never forwarded — the LLM never responds.
Also rewrites `create_response: true` → `false` in client `session.update`
so the proxy controls when responses are triggered.
* feat(realtime): speak guardrail block message as audio via TTS
Instead of sending synthetic text events when a guardrail blocks,
send response.create with forced instructions so OpenAI's TTS speaks
the warning message — user hears the block instead of just seeing text.
* fix(realtime): speak exact content filter error message via TTS
Extract the human-readable error string from HTTPException.detail
so the spoken warning says e.g. "Content blocked: keyword 'system update'
detected" instead of the raw str(e) repr.
* fix(realtime): reliably enforce create_response=false for guardrails
- Proxy now injects session.update with create_response=false immediately
on session.created (when guardrails are active), instead of rewriting
the client's session.update — works regardless of what the client sends
- Add response.cancel before the warning response.create to kill any
in-flight LLM response that snuck through before the guardrail fired
* refactor(realtime): call apply_guardrail directly, remove dedicated hook method
The async_realtime_input_transcription_hook in CustomGuardrail and
ContentFilterGuardrail was just a thin wrapper that called apply_guardrail —
the same interface used by /chat and /messages. Remove the wrapper and call
apply_guardrail directly from run_realtime_guardrails, keeping the pattern
consistent across all endpoints.
* docs: add Realtime API guardrails tutorial and flow diagram
* fix: address Greptile review comments
- Forward user_api_key_dict through realtime_api/main.py (_arealtime) so
it actually reaches RealTimeStreaming instead of always being None
- Run guardrail interception in provider_config path too (e.g. Gemini),
not only the OpenAI direct path
- Narrow exception catch to HTTPException/ValueError only; re-raise
unexpected errors so programming bugs surface in logs rather than
silently appearing as guardrail blocks
- Update tests: mock apply_guardrail directly (hook method was removed),
replace session.update client-rewrite test with session.created
injection test matching the new server-side approach
* fix: address latest Greptile review comments
- Remove fastapi import from SDK-layer file; check for status_code/detail
attrs instead to identify guardrail-block exceptions vs programming errors
- Add store_message() before continue in transcription interception so
transcription events are logged in the non-provider_config path
- Inject create_response=false on session.created in provider_config path
(Gemini etc.) to match the OpenAI path — prevents LLM auto-responding
before guardrail runs on VAD-detected turns
* Cache realtime websocket request body
Move the realtime request payload builder out of the websocket handler and wrap it with an LRU cache so repeated connections reuse the same bytes object. This keeps the JSON formatting cost down while bounding memory usage.
* Optimize realtime websocket caching
Refactored /v1/realtime to use cached helpers for both the JSON body and query params, introduced a reusable request-scope template, and optimized header handling to avoid redundant work.
* Refine realtime websocket header handling
* Reuse websocket scope headers in auth
* Refactor realtime request body helper
Move the realtime request body formatter into proxy common utils so it can be reused across modules. Reuse it in the websocket auth flow to share LRU caching and avoid ad hoc byte builders.
* fix: revert to old pattern
The old pattern was necessary, we can just return the optimized function instead.
* Reuse SSL context for realtime
Create a shared SSLContext for OpenAI realtime websocket dials and pass it into websockets.connect so we stop re-reading verify paths on every session.
* feat: reuse shared TLS context for realtime websockets
- add `SHARED_REALTIME_SSL_CONTEXT` helper so all realtime websocket clients share the same TLS settings
- wire the shared context into OpenAI, Azure, custom HTTPX handlers, and realtime health checks
- update realtime tests to assert that the expected SSL context is passed to `websockets.connect`
This keeps TLS configuration consistent and avoids recreating SSL contexts per connection.
* Reuse HTTP SSL context for realtime
Remove the standalone realtime SSL helper, expose a shared context directly from the HTTP handler, and point all realtime websocket clients and tests to it. Add the websocket header comparison tool.
* Lazy-load shared realtime SSL context
Fix circular imports introduced by eagerly instantiating the shared TLS context. Make the HTTP handler lazily create the context and have realtime clients/tests fetch it on demand, keeping configuration consistent without breaking startup.
* add: unit test for realtime LRU caches
* fix: merge conflict with imports
* fix intent params
* Add responses
* fix unrelated test
* test fix - fireworks API endpoint is down
* test fix fireworks ai is having an active outage
* test_completion_cost_databricks
* dbrx fix test API currently not responding
* Update OpenAI Realtime handler to use the correct endpoint and include all query parameters. Adjusted error messages for missing API base and key. Updated health check URL construction to pass model as a query parameter.
* Enhance OpenAI Realtime handler tests to ensure model parameter inclusion in WebSocket URL. Added new tests to verify correct URL construction with model and additional parameters, preventing 'missing_model' errors. Updated existing tests for consistency.
* Remove debug print statements for API base and key in OpenAIRealtime handler to clean up the code.
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* refactor(litellm_logging.py): refactor realtime cost tracking to use common code as rest
Ensures basic features like base model just work
* feat(realtime/): support 'base_model' cost tracking on realtime api
Fixes issue where base model was not working on realtime
* fix: fix ruff linting error
* test: fix test
* feat(azure/realtime): initial working commit for proxy azure openai realtime endpoint support
Adds support for passing /v1/realtime calls via litellm proxy
* feat(realtime_api/main.py): abstraction for handling openai realtime api calls
* feat(router.py): add `arealtime()` endpoint in router for realtime api calls
Allows using `model_list` in proxy for realtime as well
* fix: make realtime api a private function
Structure might change based on feedback. Make that clear to users.
* build(requirements.txt): add websockets to the requirements.txt
* feat(openai/realtime): add openai /v1/realtime api support
