- Gemini tool-call response.done now includes conversation_id so clients
can match it against the preceding response.created.
- Vertex AI setup no longer overrides an explicit guardrail-injected
create_response: False back to disabled: False; the guardrail's intent
to disable VAD auto-response is now respected.
- Modality handler is now passed the locally-updated response/item IDs
rather than the original input snapshot, preventing stale IDs after a
prior tool-call/response.done in the same JSON message resets them.
- Skip emitting orphaned response.created/response.done events when
Gemini sends an empty functionCalls array.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Update test_gemini_realtime_function_call_output_transformation to pre-load
the call_id → name mapping by transforming a Gemini toolCall first, then
assert that the resulting Gemini toolResponse functionResponses entry
carries the function name. This pins the production round-trip rather
than the degenerate 'name missing' branch.
- Skip the force-override block when the injection block already ran for
the same session.update to avoid redundant JSON re-serialization.
- Normalize non-dict client-provided turn_detection values (flat and
nested audio.input.turn_detection) to a dict before enforcing
create_response=False, matching the injection block's behavior and
preventing potential bypass on backends that accept non-dict values.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Gemini Live treats a second BidiGenerateContentSetup as a full session
replacement, not a partial merge. The guardrail-driven turn_detection-only
session.update was emitting a setup containing only model + realtimeInputConfig,
which would silently drop tools, generationConfig, inputAudioTranscription, and
systemInstruction from the original setup. Carry forward the cached original
setup and only override realtimeInputConfig.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The function exceeded the 50-statement limit (64 > 50) after recent
realtime guardrail additions. Matches the existing project pattern for
inherently complex event/message-mapping methods (see _process_event,
translate_messages_to_responses_input, transform_realtime_response,
_arealtime, etc.).
Prevents a client from re-enabling Gemini/GA VAD auto-response (and thereby
bypassing the audio transcription guardrail) by sending a later
session.update with turn_detection.create_response: true.
Previously _build_vertex_ai_setup_config only lifted nested turn_detection
back to the top level. GA clients' output_modalities and
audio.input.transcription were silently dropped because map_openai_params
only recognises the flat OpenAI-beta keys. Use the parent's
_normalize_session_payload_for_mapping so modalities, transcription, and
turn_detection are all surfaced before mapping.
map_openai_params only recognises the flat OpenAI-beta keys (modalities,
input_audio_transcription, turn_detection). For GA clients the upstream
shim renames these into the nested GA schema (output_modalities,
audio.input.transcription, audio.input.turn_detection), causing them to
be silently dropped in _handle_session_update. Add a normalization helper
that surfaces the GA-remapped values back at the top level so the
existing mapping logic picks them up. Without this, a GA client
explicitly requesting modalities=['text'] would still default to audio
output.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bedrock): strip bedrock/ prefix and URL-encode ARNs in get_bedrock_model_id for invoke path
The invoke path (used by /v1/messages → Anthropic SDK / Claude Code) called
get_bedrock_model_id() which, when falling back to the raw model string, did
not strip the 'bedrock/' routing prefix and did not URL-encode ARNs.
For a model like:
bedrock/arn:aws:bedrock:us-east-1:<ACCOUNT>:inference-profile/global.anthropic...
the URL built was:
/model/bedrock/arn:aws:bedrock:…/invoke-with-response-stream ❌
Bedrock returned a JSON error body. LiteLLM's AWSEventStreamDecoder passed
those bytes into botocore's EventStreamBuffer which expects binary event-stream
framing. Checksum validation failed on the JSON prelude (0x223a7b22 == ':{"')
producing a misleading botocore.eventstream.ChecksumMismatch instead of the
actual Bedrock error.
Fix: strip 'bedrock/' (and 'invoke/') routing prefix from model string, then
URL-encode if the result is an ARN — matching what the converse path already
does in converse_handler.py.
Fixes: LIT-3274
* fix(bedrock): use strip_bedrock_routing_prefix to handle compound prefixes
Address greptile review: the original fix used a loop with break, so
bedrock/invoke/arn:... only stripped bedrock/ leaving invoke/arn:...
which is not an ARN → fell through to .replace('invoke/','',1) →
bare unencoded ARN → same malformed-URL bug.
strip_bedrock_routing_prefix() iterates without break, correctly
stripping bedrock/ then invoke/ in sequence. Also adds test case
for the compound-prefix scenario.
* style: apply black formatting to fix lint CI (LIT-3274)
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: LiteLLM Bot <bot@berri.ai>
map_automatic_turn_detection sets disabled=True whenever create_response is
absent OR False. Transcription guardrails inject create_response: False to
suppress auto-responses while expecting VAD to stay active, but the previous
override in _build_vertex_ai_setup_config only fired when create_response was
absent, leaving disabled=True and silently breaking speech detection and
transcription events. Vertex Live has no 'VAD on, no auto-response' mode, so
always keep VAD active in the setup config.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(sagemaker): use Cohere embed payload for Marketplace endpoints
SageMaker embedding only special-cased Voyage; every other endpoint received
HuggingFace TGI `{"inputs": [...]}`. AWS Marketplace Cohere containers expect
the native Cohere embed payload (`texts`, `input_type`) and reject the HF
shape with `422 EmbedReqV2.inputs is of type string but should be of type
Object`.
Add `SagemakerCohereEmbeddingConfig` that reuses Bedrock/Cohere request and
response transforms, and route SageMaker endpoint names containing `cohere`
or a Cohere embed model fragment (`embed-multilingual`, `embed-english`,
`embed-v3`, `embed-v4`) to it. Supports `input_type`, `dimensions`, and
`encoding_format`. Voyage and HuggingFace SageMaker endpoints are unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(sagemaker): simplify cohere detection and align with file conventions
- Detect Cohere SageMaker endpoints with a single `"cohere" in model.lower()`
check, mirroring the existing Voyage branch instead of a separate helper
function and marker constant.
- Drop instance caches of sub-configs; instantiate `BedrockCohereEmbeddingConfig`
/ `CohereEmbeddingConfig` per call to match the existing pattern in
`BedrockCohereEmbeddingConfig._transform_request`.
- Match `SagemakerEmbeddingConfig`'s signatures, defaults, and `Any` typing for
`logging_obj`; collapse the input-normalization helper inline.
- Inline `transform_embedding_response` input lookup; no behavior change.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(sagemaker): restore provider-supported embedding params after map
Cohere input_type is advertised in get_supported_openai_params but was
filtered out of non_default_params by OPENAI_EMBEDDING_PARAMS before
map_openai_params ran. Merge supported params from passed_params after
map (same path Greptile flagged). Handle input_type explicitly in
SagemakerCohereEmbeddingConfig.map_openai_params and add an integration
test through get_optional_params_embeddings.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(embeddings): only restore non-OpenAI supported params after map
The post-map restore loop must skip OPENAI_EMBEDDING_PARAMS so mapped
fields (e.g. dimensions -> output_dimension) are not duplicated under
their OpenAI names. Align SageMaker embedding import order with sibling
files and add a regression test for dimensions mapping.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(sagemaker): avoid double post_call on Cohere embedding response
Greptile review on #28613 caught that `CohereEmbeddingConfig._transform_response`
calls `logging_obj.post_call` internally. The SageMaker embedding handler
already calls `post_call` once before invoking the transform, so the Cohere
SageMaker path fired callbacks, cost calculators, and log handlers twice
per request.
Extract the parsing body of `_transform_response` into
`_populate_embedding_response` (pure extract-method, no behavior change
for existing Cohere direct or Bedrock Cohere paths, which keep calling
`_transform_response`). Have `SagemakerCohereEmbeddingConfig` call the
new helper directly so it parses the response without re-logging.
Add a regression test asserting `logging_obj.post_call` is not invoked
by the SageMaker Cohere transform.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Previously the _guardrail_turn_detection_update_sent flag was set inline
during message rewriting in client_ack_messages, before the modified
session.update was forwarded to the backend. If _send_to_backend raised
(e.g. backend WebSocket disconnect), the exception was caught and the
loop continued, but the flag remained True — permanently disabling the
guardrail create_response=False injection for the rest of the session.
Neither the client_ack_messages path nor the
_maybe_send_guardrail_turn_detection_update backup path would retry.
Track the injection locally and only set the flag after _send_to_backend
returns a truthy sent result, matching the pattern used by
_maybe_send_guardrail_turn_detection_update.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The Playwright suite under tests/proxy_admin_ui_tests/e2e_ui_tests/ is no
longer wired into CI (only test_*.py is globbed) and every active spec is
duplicated by ui/litellm-dashboard/e2e_tests/tests/ (login, auth redirect,
search users, internal user list). team_admin.spec.ts was entirely
commented out. Removing the directory plus its only-used-here playwright
config, package.json/lock, and utils/login.ts keeps the canonical suite
under ui/litellm-dashboard/e2e_tests/ as the single source of truth.
* test(proxy): add create_scratch_actor harness helper
Adds create_scratch_actor() to the management behavior-suite conftest and
extends create_scratch_team() with team_member_permissions / models kwargs,
needed by the PR3 team-key-permission and team-model matrices. The new
helper mints a scratch-prefixed user + verification token (+ org
memberships), all reclaimed by the existing scratch-prefix teardown.
* test(proxy): pin /key block, unblock, health, aliases behavior
Adds behavior-pinning matrices for POST /key/block, POST /key/unblock,
POST /key/health, and GET /key/aliases. Pins that the management-route gate
401s ORG_ADMIN-role callers before _check_key_admin_access runs, the
block/unblock round-trip on the blocked column, missing-key 404, and the
_apply_non_admin_alias_scope visibility rules for /key/aliases.
* test(proxy): pin /key/bulk_update + /team/key/bulk_update behavior
Adds behavior-pinning matrices for POST /key/bulk_update (PROXY_ADMIN-only;
ORG_ADMIN stopped 401 at the route gate, INTERNAL_USER-role 403 at the
handler) and POST /team/key/bulk_update (team-member-permission gate keyed
on KEY_UPDATE). Pins batch semantics: empty/over-cap 400, per-key failure
isolation into failed_updates, all_keys_in_team broadcast, and no-keys 404.
Adds an optional key_alias arg to create_scratch_key for multi-key scenarios.
* test(proxy): pin /key SA-generate, v2-info, reset-spend behavior
Adds behavior-pinning matrices for POST /key/service-account/generate
(team-membership + team-member-permission gating; SA keys carry no user_id),
POST /v2/key/info (per-key _can_user_query_key_info silently drops invisible
keys), and POST /key/{key}/reset_spend (PROXY_ADMIN or team admin only;
missing key 404, reset-value 400). Pins that ORG_ADMIN-role callers are
stopped 401 at the management-route gate on the two non-info routes.
* test(proxy): close PR1/PR2 key-side deferred coverage gaps
Closes the four key-side gaps deferred from PR1/PR2:
- 404 on missing key for /key/update and /key/delete (not 401/403)
- denied /key/update leaves max_budget/tpm_limit/rpm_limit untouched
- /key/regenerate enforces litellm.upperbound_key_generate_params (#26340)
- /key/list key_alias substring vs exact (admin-only) + team_id filter,
and a non-admin filtering a foreign team is 403
* test(proxy): pin /team block, unblock, available, filter/ui, members/me
Adds behavior-pinning matrices for POST /team/block + /team/unblock
(management-route gate fronts _verify_team_access; reachable only by
PROXY_ADMIN and an org admin of the team's own org), GET /team/available
(default empty path), GET /team/filter/ui (route-gated PROXY-ADMIN-only
despite the handler having no gate), and GET /team/{team_id}/members/me
(caller resolves its own membership; non-member 404, no-user_id key 400).
* test(proxy): pin /team model add/delete + permissions endpoints
Adds behavior-pinning matrices for POST /team/model/add + /team/model/delete
(route-gated PROXY-ADMIN-only; missing team 404), GET /team/permissions_list +
POST /team/permissions_update (self-managed; proxy/team/org admin pass), and
POST /team/permissions_bulk_update (PROXY_ADMIN-only). Pins the deliberate
divergence that the available-team self-join grants read access via
permissions_list but never write access via permissions_update.
* test(proxy): pin /team delete, bulk_member_add, v2/list, daily/activity
Adds behavior-pinning matrices for POST /team/delete (per-team
_verify_team_access; batch aborts whole on a missing id), POST
/team/bulk_member_add (route-gated PROXY-ADMIN-only; empty/over-cap 400),
GET /v2/team/list (_enforce_list_team_v2_access — bare query 401s regular
users, org-scoped for org admins) and GET /team/daily/activity (non-member
team_ids filter 404, the VERIA-43 fix).
* test(proxy): add route-coverage gate + close team org-relocation gap
Adds test_route_coverage.py (PR3.M1): parses every @router route literal
from the two management-endpoint source files and asserts each is exercised
by >=1 behavior-suite scenario — a permanent regression guard for future
routes. Closes the last PR1/PR2 deferred gap: the /team/update org-relocation
allowed branch, exercised by a dual-org-admin minted via create_scratch_actor.
test_team_model uses literal route URLs so the coverage parser resolves them.
* test(proxy): bound plain route params to one path segment in coverage gate
Plain path params ({team_id}) now compile to [^/?]+ instead of [^?]+, so a
parameter cannot span '/'. Starlette ':path' params still match across '/'.
Keeps the route-coverage guard from falsely reporting a future multi-segment
route as covered. All 37 routes remain covered.
Align return_new_content_delta_events with the AUDIO defaults used in
_handle_session_update and transform_session_created_event so deferred
session config does not produce TEXT-typed delta events for audio data.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The dependency license checker only read the legacy free-text
`info.license` field from PyPI. Packages that adopt PEP 639 publish
their license as an SPDX expression in `info.license_expression` and
leave the legacy field null, so the checker reported "Unknown license"
and failed CI for every newly-bumped PEP 639 dependency.
`get_package_license_from_pypi` now resolves the license in order:
`license_expression`, then legacy `license`, then the
`License :: OSI Approved :: ...` trove classifiers.
`is_license_acceptable` splits compound SPDX expressions on the
uppercase OR/AND operators (case-sensitive, so the lowercase
`-or-later` inside an identifier is not mistaken for an operator) and
strips `WITH <exception>` suffixes, requiring every component to be
acceptable. Free-text license blobs are detected and fall back to the
original whole-string matching.
The `black` and `pydantic-settings` entries in liccheck.ini that
existed solely to work around this now resolve correctly on their own
and have been removed.
Vertex AI Live treats setup as a first-and-only client message; emitting a
second setup with realtimeInputConfig only closes the websocket with a 1007
policy error. Reverting the follow-up-setup branch restores the pre-existing
no-op behavior for subsequent session.update messages.
The synthetic session.created event emitted in deferred setup mode used
TEXT as the default for responseModalities, while _handle_session_update
defaults to AUDIO. Align the default so clients reading modalities from
the initial session.created see the correct value for live sessions.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
When a client sends a session.update whose turn_detection field is None or
a non-dict value (e.g. "auto"), the guardrail injection used setdefault
followed by item assignment on the returned value, raising TypeError. The
inner except only caught JSONDecodeError/AttributeError, so the TypeError
escaped to the outer Exception handler that wraps the entire client_ack
loop, killing the connection. Replace non-dict turn_detection with a
fresh dict carrying create_response=False so the guardrail still applies
without crashing the loop.
- Vertex AI _build_vertex_ai_setup_config now lifts nested
audio.input.turn_detection to the top level before calling
map_openai_params, mirroring the parent GeminiRealtimeConfig
behavior. Without this, guardrail-injected create_response: False
was silently dropped for GA-protocol Vertex AI clients.
- realtime_streaming session.created handling now sends the
(possibly re-typed) event first and then triggers the guardrail
turn-detection update for both first and duplicate cases, removing
the inconsistent guardrail-then-event ordering for duplicates.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic): handle empty streaming tool calls (#28549)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* [Feature][Bug Fix] Decouple Azure OpenAI Deployment ID from model name via base_model to fix gpt5 model routing (#28490)
* feat(azure): decouple deployment ID from model name via base_model
Azure OpenAI deployments have arbitrary names (deployment IDs) that may
not match the underlying model. Previously, model-type detection
(o-series, gpt-5, etc.) relied on substring matching against the
deployment name, causing misrouted configs and rejected params when
deployment names were non-standard (e.g. 'my-deployment-id' for gpt-5.2).
This change extends the existing base_model field to drive model-type
detection, config selection, supported param resolution, and param
mapping throughout the Azure call path:
- _get_azure_config() uses base_model for is_o_series/is_gpt_5 checks
- get_provider_chat_config() threads base_model for Azure
- get_supported_openai_params() accepts and uses base_model
- get_optional_params() accepts base_model and passes it to all Azure
config method calls (get_supported_openai_params, map_openai_params)
- azure.py completion handler uses base_model for GPT-5 detection
- Config internal methods (e.g. is_model_gpt_5_2_model) now receive
base_model so features like logprobs are correctly enabled
Fully backward compatible - when base_model is unset, behavior is
identical. Existing o_series/ and gpt5_series/ prefix workarounds
continue to work.
Usage in proxy config:
model_list:
- model_name: my-gpt5
litellm_params:
model: azure/my-deployment-id
model_info:
base_model: azure/gpt-5.2
Fixes: non-standard deployment names like 'prefix-gpt-5.2' rejecting
logprobs/top_logprobs despite the underlying model supporting them.
* Addressing Greptile comments.
* gemini-3.1-flash-lite pricing (#27933)
* feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers
* fix pricing
* add service tier
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
* fix(openai-responses): strip Anthropic cache_control from Responses API requests (#28431)
Squash-merged by litellm-agent from cwang-otto's PR.
* Treat None litellm_provider as wildcard in _check_provider_match (#28523)
Squash-merged by litellm-agent from adityasingh2400's PR.
* fix greptile
* fix: use _azure_detection_model in default Azure branch of get_supported_openai_params
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(openai-responses): strip cache_control on compact endpoint as well
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: Felipe Garé <90070734+FelipeRodriguesGare@users.noreply.github.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: withomasmicrosoft <withomas@microsoft.com>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com>
Co-authored-by: Aditya Singh <60082699+adityasingh2400@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(responses): use OpenAI SSEDecoder for Responses API streaming
httpx aiter_lines() uses str.splitlines(), which splits on U+2028 inside
JSON payloads and silently drops response.completed (no spend log). Use
openai._streaming.SSEDecoder (bytes.splitlines before decode) instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(responses): drop redundant SSE prefix strip after SSEDecoder switch
SSEDecoder already strips the 'data:' field prefix from each event, so the
extra call to _strip_sse_data_from_chunk on sse.data was redundant and could
incorrectly mangle payloads whose actual content starts with 'data:'.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
After the GA remap moves session.turn_detection into session.audio.input.turn_detection,
Gemini's map_openai_params only looks at top-level keys and silently drops it. Normalize
the extracted turn_detection back to the top level on first session.update so the guardrail
create_response:False (and any client-provided VAD settings) reach the Gemini setup.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(proxy): persist allowlisted OIDC claims in CLI SSO poll
Map CLI_SSO_CLAIM_MAP sources into user metadata and return scalar
attribution_metadata from /sso/cli/poll. Build SSOUserDefinedValues in
cli_sso_callback so first-time CLI logins can upsert users. Add mock OIDC
scripts and tests for claim extraction and poll exposure.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(proxy): document CLI SSO attribution_metadata in client README
Co-authored-by: Cursor <cursoragent@cursor.com>
* Delete scripts/mock_oidc_server_for_cli_sso.py
* Delete scripts/test_cli_sso_claims_e2e.py
* fix(ui_sso): preserve claim types and avoid metadata. prefix stripping
- Replace _update_dictionary with a local recursive merge so string
OIDC claim values that happen to look numeric are not silently coerced
to int/float when persisting CLI SSO attribution metadata.
- Use a local dot-path resolver in _extract_sso_claim_value so that
source claim paths beginning with 'metadata.' are not silently stripped
by get_nested_value (which is designed for LiteLLM JWT metadata, not
arbitrary OIDC claims).
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Remove redundant metadata. prefix strip in _set_nested_metadata_value
The _parse_cli_sso_claim_map already strips the metadata. prefix from
dest keys before reaching the setter. The duplicate strip in
_set_nested_metadata_value was a no-op in normal flow but could
mis-place values for dest keys like metadata.metadata.foo.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Fix greptile
* Fix ruff
* Move CLI SSO user defined values build inside try/except for consistent error handling
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(proxy): enforce restricted SSO group on CLI SSO callback
Apply verify_user_in_restricted_sso_group before CLI session completion
and user upsert, matching the UI SSO path. Re-raise ProxyException so
restricted-group denials return 403 instead of 500.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): replace recursive CLI SSO metadata helpers with iterative merge
Use stack-based flatten/merge to satisfy recursive_detector CI. Fix mypy
types for UserApiKeyCache and user_id on CLI SSO session completion.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: resolve nested CustomOpenID extra_fields in CLI SSO claim extraction
When GENERIC_USER_EXTRA_ATTRIBUTES captures a parent object (e.g. org_info),
extra_fields stores it as {"org_info": {"department": "..."}}. A CLI claim
map entry using a dotted path like org_info.department would silently fail
because the lookup only checked the exact flat key. Fall back to dotted-path
resolution on extra_fields before model_dump().
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(sso): update CLI SSO test for new received_response kwarg and remove redundant 'token' secret fragment
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- Convert provider's real session.created to session.updated when a synthetic
one was already forwarded so clients receive the authoritative modalities
derived from their session.update instead of the synthetic placeholder.
- Reset current_response_id / current_output_item_id after Gemini RESPONSE_DONE
so a toolCall arriving in a later frame starts a fresh response instead of
reusing the completed response's ID and emitting a duplicate response.done.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(mcp): cache OAuth token client-side so Tools tab loads without re-auth
After a user creates an OAuth MCP server and completes the authorization
flow, the resulting access token is now stored in sessionStorage keyed by
server_id. The MCP Tools tab reads this cached token and includes it as
an MCP auth header when listing and invoking tools, so the user never sees
an empty tool list. When the session ends (tab close / new browser) an
Authorize button re-triggers the flow without leaving the Tools screen.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(ui/mcp): surface listMCPTools 401 errors so auth gate reappears
listMCPTools previously swallowed all errors (including HTTP 401) by
returning a synthetic { tools: [], error: 'network_error', ... } payload.
That made the useQuery retry-on-401 guard and mcpToolsError dead code,
so expired OAuth tokens never re-triggered the auth gate.
- Throw an enhanced Error with .status attached on non-2xx responses
(still preserves the legacy shape for true network failures so the
caller can render a generic message without crashing).
- Clear the cached OAuth session token when the tools query fails with
401, mirroring callMCPTool's onError handler so the Authorize button
is shown again.
- Surface mcpToolsError in the existing error banner.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp-tools): stable onSuccess + reuse parsed flow state
- Pass stable setOauthToken setter directly as onSuccess to avoid
recreating useToolsOAuthFlow's resumeOAuthFlow on every render.
- Reuse the already-parsed FLOW_STATE_KEY value (peeked) instead of
re-reading and re-parsing sessionStorage in resumeOAuthFlow.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(ui/mcp): restore listMCPTools never-throws contract
The previous fix made listMCPTools throw on HTTP errors while still
returning a synthetic object on network errors. This inconsistent
contract broke existing callers (MCPToolPermissions, MCPAppsPanel,
MCPConnectPicker) which inspect result.error / result.message and
expect the function to never throw.
- Return a normalized { tools: [], error, message, status, ... }
object on HTTP errors (instead of throwing) so all callers see a
consistent shape and the user-visible error text from
result.message is preserved.
- Convert the returned error object into a thrown Error inside the
one caller that needs it — the useQuery in mcp_tools.tsx — so the
401 retry/onError handlers still trigger and clear the cached
OAuth token.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix greptile
* fix(mcp): align OAuth header alias lookup with dashboard sanitization
Backend auth header resolution now matches x-mcp-{alias} keys produced by
the dashboard sanitizer, and the Tools tab re-syncs OAuth tokens when
serverId changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): widen auth header lookup types for list_tools
Accept legacy str | dict server auth maps and annotate list_tools
server_auth_header as Union[str, dict] for mypy.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(ui): extract shared buildCallbackUrl/clearStorage for MCP OAuth hooks
Hoist the duplicate buildCallbackUrl and clearStorage helpers out of
useToolsOAuthFlow and useUserMcpOAuthFlow into a new shared module
src/hooks/mcpOAuthUtils.ts so the two hooks cannot drift if the URL
construction or storage cleanup logic needs to change.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(ui): don't gate M2M OAuth MCP servers behind interactive authorize
M2M (client_credentials) OAuth servers share auth_type="oauth2" with
interactive PKCE servers, but the backend fetches their token internally
and they typically lack a user authorization endpoint. Gating tool
listing on them rendered an Authorize button that would fail or redirect
incorrectly. Detect M2M via the presence of token_url (matching the
existing heuristic in mcp_server_edit.tsx) and skip the auth gate.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(ui/mcp): return error shape when listMCPTools JSON parse fails
Restore the never-throws contract when response.json() fails on a 2xx
body so callers do not receive null and crash on result.tools.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Add error_description and hint for oauth flows
* Fix tests
* fix(mcp-oauth): improve redirect_uri errors without leaking internal config
Use NoReturn on _oauth_invalid_request, structured errors for BYOK loopback
validation, and refactor validate_trusted_redirect_uri to satisfy PLR0915.
Keep PROXY_BASE_URL and raw proxy_base_url in server logs only, not in the
HTTP 400 body returned to unauthenticated callers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp-oauth): stop leaking internal proxy origin in redirect_uri 400 body
The trusted-redirect-uri rejection helper included the proxy's
resolved scheme/host/port (e.g. http://litellm-internal:4000) in both
the error_description and as a top-level proxy_origin field. Since
the OAuth /authorize endpoint is unauthenticated, any caller could
probe with a crafted redirect_uri and enumerate the internal network
topology behind a reverse proxy.
Keep full diagnostic detail in the server-side warning log
(including the computed proxy base) but omit proxy-side values from
the HTTP 400 body. Also drop the duplicated origin computation in
_raise_trusted_redirect_uri_rejected now that those values are no
longer needed by the response.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp-oauth): remove dead userinfo check in redirect_uri validation
The first check combined missing netloc with userinfo presence, making
the second userinfo-only check unreachable. Split into two distinct
checks so each error message reflects the actual failure mode.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- Replace duplicated transform/send/cache logic in client_ack_messages with a call to _send_to_backend so future changes stay in one place.
- VertexAIRealtimeConfig.transform_realtime_request now uses .get('session') or {} for the first session.update so a malformed client payload no longer crashes the connection.
- Move the audio-transcription guardrail turn_detection injection to run BEFORE the beta->GA session remap. This lets the injected create_response ride along with any client-provided turn_detection fields (e.g. silence_duration_ms) into the nested audio.input.turn_detection path produced by the remap instead of being stranded as a separate root-level dict.
- Update the deferred-mode injection test to assert the GA-shaped location.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- Move store_input() after the guardrail turn_detection injection in
client_ack_messages so audit logs reflect what is actually forwarded
to the backend (previously the unmodified pre-injection message was
logged).
- In Gemini's _handle_session_update, allow a session.update that only
carries a turn_detection change to be forwarded as a follow-up Gemini
setup with realtimeInputConfig.automaticActivityDetection set, even
after the initial setup. This restores the guardrail layer's ability
to disable VAD auto-response in non-deferred mode (the default Gemini
flow), which was a regression after _handle_session_update started
silently dropping subsequent session.update messages. Both flat
beta-style and nested GA-style turn_detection payloads are accepted.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
After closing a tool-call response, clear current_output_item_id and
current_response_id so post-tool model turns emit a fresh response.created
preamble. Add regression tests and align guardrail turn_detection test with
GA session shape; apply Black formatting.
Co-authored-by: Cursor <cursoragent@cursor.com>
- realtime_streaming: only mark _guardrail_turn_detection_update_sent
when the message was actually delivered to the backend. The provider
transformation (e.g. Gemini after initial setup) may silently drop
session.update; previously we set the flag anyway, falsely claiming
the disable was sent and preventing any retry on subsequent
session.created events. _send_to_backend now returns whether at
least one transformed message was sent.
- gemini realtime transformation: avoid shadowing the outer
openai_event variable in map_openai_event's fallback loop. With
the new toolCall entry now last in MAP_GEMINI_FIELD_TO_OPENAI_EVENT,
an unmatched key would otherwise leak FUNCTION_CALL_ARGUMENTS_DONE
and skip the ValueError raise. Use a distinct loop variable so the
is-None check correctly raises for unknown Gemini messages.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Take litellm_internal_staging version of e1a670efcb966aaa.js after
incomplete merge left conflict markers in the committed artifact.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Store call_id → name mapping when receiving toolCall from Gemini
- Look up and include name in functionResponses when sending tool results
- Required by Gemini Live API spec for proper tool call routing
- Add test to verify name field is included in round-trip
Made-with: Cursor
- Add response.done event after tool-call loop to signal response completion
- Required by OpenAI SDK clients to submit tool results
- Without this, clients stall indefinitely waiting for response completion
- Update test to verify complete 6-event sequence including response.done
Made-with: Cursor
- Emit response.output_item.done between function_call_arguments.done and conversation.item.created
- Required by OpenAI Realtime spec to finalize function-call items
- Update test to verify complete event sequence
Made-with: Cursor
- Emit response.created, output_item.added, and conversation.item.created for function calls
- Ensures OpenAI Realtime API spec compliance
- Add test for preamble emission
Made-with: Cursor
- Instead of sending turn_detection as separate message (which gets dropped), inject it into the first client session.update
- This ensures guardrails work correctly in deferred mode
- Add test for turn_detection injection in deferred mode
Made-with: Cursor
- Remove early guardrail turn-detection update that consumed first setup slot
- Add inputAudioTranscription default in Gemini deferred-mode setup
- Add tests for both fixes
Made-with: Cursor
* Add granian as a ASGI compliant web server. Provides better stability, 10-20 RPS improvement under standard LT conditions.
TODO: Verify poetry lock details and add locust numbers to PR
* Update granian version in license_cache.json and pyproject.toml to 2.5.7
* Enhance proxy CLI tests by adding SSL initialization checks for Granian server. Remove Python version skip conditions and implement tests to ensure SSL certificate and key are required for server initialization.
* update uv lock to fix granian import error
* test(e2e): forward LITELLM_LICENSE to UI e2e proxy
The UI e2e job ran without LITELLM_LICENSE, so premium_user was always
false in the issued login JWT and premium-gated UI surfaces (Team-BYOK
Model switch, etc.) couldn't be driven through the UI. Forward the env
var from run_e2e.sh and the CircleCI e2e_ui_testing job, and add a
sanity test that decodes the admin storage state token and asserts
premium_user=true so the wiring fails loudly if it ever regresses.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Update ui/litellm-dashboard/e2e_tests/tests/proxy-admin/license.spec.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
