The purge takes an injectable invalidate_token_cache callable defaulting to the manager's shared
invalidation, and MCPServerManager takes an injectable per_user_token_cache alongside the existing
per_user_oauth_token_store, so tests inject fakes instead of monkeypatching the global manager and
the module-level cache. The new identity helpers drop Any for object throughout
Review follow-ups on the stale-token invalidation. The backend identity now decrypts client_id and
client_secret before comparing: the stored values are NaCl-encrypted with a fresh nonce on every
write, so comparing ciphertext flagged every routine save as a mint-relevant change and purged
per-user tokens that were still valid. The identity also gains spec_path, the audience for OpenAPI
servers, and parses credentials stored as a JSON string
The purge now routes each (user, server) through the manager's invalidate_user_oauth_token_cache,
which becomes the single invalidation point covering both the legacy per-user token cache and the
v2 per-user OAuth token store; previously the purge evicted only the legacy cache while the revoke
path evicted only the v2 store, so each path left the other cache serving a replaced token until
its TTL. A credential row racing in between the find and the delete is now detected via the
delete_many count and logged; its cache entry expires by TTL
On the dashboard, CLEARED_ON_INVALIDATION and the staleness check move to types.tsx as the single
shared implementation for both forms. The edit form's transport handler now rechecks the identity
after its programmatic setFieldsValue calls, which antd does not report through onValuesChange, so
a token no longer survives a transport switch that clears the mint target. The create form rebuilds
formValues from the post-reset form state after an invalidation instead of publishing the pre-reset
snapshot, so the tool preview can no longer refetch with the discarded DCR client. Both transport
handlers now share the recheck, which also stops the create form from over-invalidating on an
http to sse swap that keeps the same url and therefore the same audience
An admin who ran Authorize & Fetch and then changed a field that determines which upstream OAuth
token gets minted kept using the stale token for tool preview, sessionStorage, and (on the backend)
the stored per-user credential and its cache. Grounded in RFC 8707/8693 and the MCP auth spec, a token
is bound to one tuple: resource/audience (url), OAuth mode/grant (auth_type, oauth_flow_type), the
authorization-server endpoints, and the OAuth client + scopes. A shared getOAuthAuthorizationIdentity
captures exactly those fields; transport (http/sse on the same url is the same audience) and
delegate_auth_to_upstream (a downstream-usage toggle never sent to the authorize request) are excluded.
UI: both the create and edit forms now discard the held token (React state / sessionStorage / hook,
plus the fetched token + DCR client in form.credentials) whenever the identity diverges from the one it
was authorized against, re-applying the admin's in-flight edit so it is never wiped. The check lives in
one shared helper so the two forms cannot drift.
Backend: editing an MCP server now compares the pre/post identity and, on a mint-relevant change, purges
every stored per-user OAuth credential for the server (DB row + per-user token cache) so no user
forwards a token minted for a resource/AS/client that no longer matches. Best-effort; a purge failure
never fails the update.
The redacted resource kept the path, but hosted MCP servers routinely embed the credential in the
path (for example /mcp/s/<token>/mcp), and mcp_tool_call_metadata is readable by a caller who can
invoke the tool, so the path leaked the upstream credential into spend logs. Only scheme, host, and
port are logged now
Authorization doubles as the admission fallback when x-litellm-api-key is absent, so a caller who
authenticated the preview request that way had their LiteLLM key forwarded to the upstream as the
oauth2/client-forwarded token. The preview now forwards Authorization only when the primary
admission header is present, which is how the dashboard has always sent it; with no primary header
there is no upstream token on the request at all. Applies to oauth2 and both client-forwarded
modes; parametrized regression test plus the admission header added to the existing extraction
tests to mirror the real UI request shape
* fix(bedrock-invoke): retain clear_tool_uses_20250919 context_management edits and emit context-management-2025-06-27 beta (LIT-3393)
Copy of #29206 by oss-agent-shin, rebased onto litellm_internal_staging so CircleCI can run.
Bedrock InvokeModel supports automatic tool-call clearing (clear_tool_uses_20250919) under the context-management-2025-06-27 beta, but LiteLLM stripped the edit and dropped the beta header, causing a Bedrock 400. This maps bedrock.context-management-2025-06-27 to itself in anthropic_beta_headers_config.json (bedrock_converse stays null) and rewrites _filter_context_management_for_bedrock_invoke around an allowlist of supported edit types that keeps each supported edit and adds its matching beta.
* test(bedrock-invoke): restore beta-headers config cache with a shared fixture in LIT-3393 tests
Greptile flagged that three of the four new tests reloaded the module-level
beta-headers config into local mode without restoring it on teardown, leaking
state into later tests in the same process. Move setup/teardown into a
local_beta_headers_config fixture used by all four tests.
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
* fix: rust ocr tests finally pass
* fix: move realtime dir
* fix(realtime): normalize azure realtime api_base to host for Foundry endpoints
The azure realtime handler appended the realtime path to api_base verbatim, so a
Foundry base carrying a project path (.../api/projects/<name>) produced an invalid
realtime URL and the websocket handshake hung. Normalize api_base to scheme and host
before building the realtime path so both Azure OpenAI and Foundry bases connect
Point the e2e realtime azure deployment at the GA gpt-realtime model and stop passing
the os.environ refs the realtime path never unwraps, resolving them from the gateway
env by name instead. Drop the local docker-compose scaffolding from the tree
* test(e2e): add Gateway.list_files and list_fine_tuning_jobs for the discovery suite
The discovery endpoints suite calls client.gateway.list_files and
list_fine_tuning_jobs, which did not exist on Gateway, so both tests errored with
AttributeError before reaching the proxy. Add the two GET wrappers using the
existing FileListResponse / FineTuningJobsResponse models
* revert(realtime): drop azure realtime api_base host-normalization
The azure realtime handshake failure was a config issue, not a litellm bug: the
realtime base was set to the Azure AI Foundry project endpoint (.../api/projects/<p>),
but the OpenAI-compatible realtime route lives at the resource root. litellm correctly
appends the realtime path to whatever base it is given, so pointing the realtime
deployment at the resource root is the fix and no core change is needed
* fix(ocr): route azure_ai doc-intelligence to its own endpoint at the source
get_llm_provider inherits AZURE_AI_API_BASE into api_base for every azure_ai/* OCR
model, but Azure Document Intelligence is a separate resource reached via
AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT, so doc-intelligence requests went to the wrong
host. Stop inheriting the azure_ai base for doc-intelligence models so api_base stays
unset and both the rust bridge and the python get_complete_url fall back to the
document-intelligence endpoint. This drops the earlier _rust_bridge_api_base reorder,
which only covered the rust path and let the env silently override an explicit api_base
* refactor(ocr): consolidate azure doc-intelligence detection; keep explicit api_base
Extract is_azure_document_intelligence_model as the single source of truth for the azure_ai doc-intelligence sub-route so the check is no longer duplicated across _prepare_ocr_request and _rust_bridge_api_base, and gate the dynamic_api_base suppression on the caller not supplying an api_base so an explicit endpoint is always honoured. Restore xai to the realtime PROVIDERS as a documented disabled entry instead of dropping it silently, and add a regression test pinning doc-intelligence api_base resolution.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException
The Bedrock-specific GuardrailInterventionNormalStringError predates the
unified guardrails refactor and no proxy code path handles it, so a block
with the flag set surfaced as an uncaught Exception -> HTTP 500 in pre_call
mode and was silently discarded in during_call mode (model call proceeded
in the parallel asyncio.gather; the block hook's data["mock_response"]
mutation happened after route_request had already unpacked kwargs).
Convert the block to ModifyResponseException at the raise site inside
make_bedrock_api_request. That exception is the industry-standard proxy
contract already caught in proxy_server, anthropic_endpoints, response_api
_endpoints, and pass_through_endpoints; it turns into a 200 response with
finish_reason=content_filter and the block message as content, which is
exactly what the flag was documented to yield. Post-call blocks attach
the LLM response to original_response so the synthetic reply reports the
upstream call's real token usage instead of zero.
Deletes the now-orphaned GuardrailInterventionNormalStringError class and
the dead create_guardrail_blocked_response / mock_response plumbing in the
Bedrock hooks; updates the existing tests that had locked in the buggy
contract.
Resolves LIT-4186
* chore(guardrails/bedrock): drop dead str branch in _update_messages_with_updated_bedrock_guardrail_response
Follow-up to the disable_exception_on_block fix. That method used to
receive either a BedrockGuardrailResponse or a plain string (the block
message, when the flag was set). Now that a block always raises
ModifyResponseException before this method runs, the string branch is
unreachable; tighten the type to BedrockGuardrailResponse and delete
the guard.
* fix(guardrails/bedrock): streaming post_call block yields synthetic stream instead of surfacing as SSE 500
Regression from the LIT-4186 refactor: pre-refactor, the streaming
post_call iterator caught GuardrailInterventionNormalStringError locally
and replaced the assembled response with a synthetic content-filter
message, then re-emitted it as chunks via MockResponseIterator. After
the refactor the exception was re-raised as ModifyResponseException,
which async_streaming_data_generator serializes as a proxy 500 error
frame because the SSE response headers are already flushed by the time
the block fires.
Non-streaming paths still let ModifyResponseException propagate to the
endpoint handler (which converts it into a 200). Streaming can't do
that, so keep the local synthesis: on the exception, rebind the
assembled response to a ModelResponse whose single choice carries the
block message as content and finish_reason=content_filter, and let the
downstream MockResponseIterator emit it as chunks. Same shape a
non-streaming block produces.
Adds a mapped-file regression test that mutation-kills the raise
behavior and locks in the synthetic-stream contract.
* fix(guardrails/bedrock): preserve upstream usage on streaming post_call block
Non-streaming post_call blocks report the upstream LLM call's real
token usage via ModifyResponseException.original_response, which the
endpoint handler unwraps through _blocked_response_usage. Streaming
post_call synthesizes its own ModelResponse locally (the exception
can't escape the SSE generator), and previously left .usage unset,
so the client saw accurate billing on non-streaming blocks and zero
on streaming blocks -- silent revenue leak.
Copy the assembled response's .usage onto the synthetic block
response before yielding. Pre-refactor code had the same gap
(create_guardrail_blocked_response never set usage); this is a net
improvement, not a regression fix.
* fix(proxy): capture logging_obj before post_call_failure_hook pops it in ModifyResponseException streaming path
post_call_failure_hook removes litellm_logging_obj from request_data before
iterating callbacks (it's not serialisable). The streaming branch of the
ModifyResponseException handler read it from _data after that call, so it
always received None and CustomStreamWrapper.__init__ crashed with
AttributeError: NoneType has no attribute model_call_details.
Capture it before the hook runs so the streaming path gets a valid object.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(proxy): add regression for streaming ModifyResponseException logging_obj capture
Covers the bug where logging_obj was read from request_data after
post_call_failure_hook had already popped it, causing CustomStreamWrapper
to crash with AttributeError.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(proxy): drive real chat_completion in ModifyResponseException streaming logging_obj regression
The original test inlined the fix pattern (capture before pop) in its
own body rather than calling the actual chat_completion handler in
proxy_server.py, so a revert of the fix left the test passing.
Confirmed via mutation check: reverting the two-line source fix and
re-running left the test green.
Rewrite the test to drive chat_completion directly:
- patch _read_request_body so chat_completion sees the seeded dict
- patch ProxyBaseLLMRequestProcessing.base_process_llm_request to
raise ModifyResponseException with the same request_data
- patch proxy_logging_obj so post_call_failure_hook mutates the dict
the way production does (pops litellm_logging_obj)
- intercept CustomStreamWrapper.__init__ and assert logging_obj is
the non-None object seeded in request_data
Mutation-verified: reverting the source fix now surfaces the exact
production crash inside CustomStreamWrapper's __init__
(AttributeError: NoneType has no attribute model_call_details) rather
than a silently-passing test.
Addresses Greptile P1 on PR #32665.
---------
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException
The Bedrock-specific GuardrailInterventionNormalStringError predates the
unified guardrails refactor and no proxy code path handles it, so a block
with the flag set surfaced as an uncaught Exception -> HTTP 500 in pre_call
mode and was silently discarded in during_call mode (model call proceeded
in the parallel asyncio.gather; the block hook's data["mock_response"]
mutation happened after route_request had already unpacked kwargs).
Convert the block to ModifyResponseException at the raise site inside
make_bedrock_api_request. That exception is the industry-standard proxy
contract already caught in proxy_server, anthropic_endpoints, response_api
_endpoints, and pass_through_endpoints; it turns into a 200 response with
finish_reason=content_filter and the block message as content, which is
exactly what the flag was documented to yield. Post-call blocks attach
the LLM response to original_response so the synthetic reply reports the
upstream call's real token usage instead of zero.
Deletes the now-orphaned GuardrailInterventionNormalStringError class and
the dead create_guardrail_blocked_response / mock_response plumbing in the
Bedrock hooks; updates the existing tests that had locked in the buggy
contract.
Resolves LIT-4186
* chore(guardrails/bedrock): drop dead str branch in _update_messages_with_updated_bedrock_guardrail_response
Follow-up to the disable_exception_on_block fix. That method used to
receive either a BedrockGuardrailResponse or a plain string (the block
message, when the flag was set). Now that a block always raises
ModifyResponseException before this method runs, the string branch is
unreachable; tighten the type to BedrockGuardrailResponse and delete
the guard.
* fix(guardrails/bedrock): streaming post_call block yields synthetic stream instead of surfacing as SSE 500
Regression from the LIT-4186 refactor: pre-refactor, the streaming
post_call iterator caught GuardrailInterventionNormalStringError locally
and replaced the assembled response with a synthetic content-filter
message, then re-emitted it as chunks via MockResponseIterator. After
the refactor the exception was re-raised as ModifyResponseException,
which async_streaming_data_generator serializes as a proxy 500 error
frame because the SSE response headers are already flushed by the time
the block fires.
Non-streaming paths still let ModifyResponseException propagate to the
endpoint handler (which converts it into a 200). Streaming can't do
that, so keep the local synthesis: on the exception, rebind the
assembled response to a ModelResponse whose single choice carries the
block message as content and finish_reason=content_filter, and let the
downstream MockResponseIterator emit it as chunks. Same shape a
non-streaming block produces.
Adds a mapped-file regression test that mutation-kills the raise
behavior and locks in the synthetic-stream contract.
* fix(guardrails/bedrock): preserve upstream usage on streaming post_call block
Non-streaming post_call blocks report the upstream LLM call's real
token usage via ModifyResponseException.original_response, which the
endpoint handler unwraps through _blocked_response_usage. Streaming
post_call synthesizes its own ModelResponse locally (the exception
can't escape the SSE generator), and previously left .usage unset,
so the client saw accurate billing on non-streaming blocks and zero
on streaming blocks -- silent revenue leak.
Copy the assembled response's .usage onto the synthetic block
response before yielding. Pre-refactor code had the same gap
(create_guardrail_blocked_response never set usage); this is a net
improvement, not a regression fix.
Two review findings on the passthrough modes.
The tool-call log records the upstream MCP server URL as mcp_server_resource,
which is persisted in spend-log metadata and sent to logging callbacks. A URL
carrying embedded userinfo or a secret query parameter would leak into logs, so
the value is now redacted to its bare resource identifier (scheme + host + path);
userinfo, query string, and fragment are stripped before it is logged.
The listing fan-out withholds the request-wide Authorization from a
true_passthrough / oauth_delegate server when another server in scope also
consumes it, so one bearer is not replayed across upstreams. The later
server.extra_headers copy loop did not honor that decision: a server listing
Authorization in extra_headers would re-copy the withheld bearer from raw_headers.
The withhold decision is now computed once and applied to both the forwarding
branch and the extra_headers loop.
Two correctness fixes for the client-forwarded token modes.
The preemptive-401 connect gate for true_passthrough and oauth_delegate
only inspected the request-wide Authorization, so a caller who bound the
upstream token via the per-server x-mcp-{alias}-authorization header (the
mandatory shape in a multi-server aggregate, where the request-wide
Authorization is withheld) was spuriously 401'd at connect even though
egress already honors that header. The gate now recognizes the per-server
header for both modes via a shared helper, mode-correctly: true_passthrough
treats any Authorization or the per-server header as the upstream token,
oauth_delegate keeps requiring a distinct x-litellm-api-key so a lone
Authorization consumed for admission is never mistaken for an upstream
token. The preemptive raise is also gated to single-server scopes so a
multi-server aggregate degrades gracefully (the listing absorbs a
per-server failure) instead of one missing token 401-ing the whole connect.
The browser-only Authorize flow was writing the upstream access and refresh
token to LiteLLM_MCPUserCredentials, contradicting the modes' persist-nothing
contract: the temp OAuth-relay server was cached with a hardcoded oauth2
auth_type, so needs_user_oauth_token was true and the token exchange stored
it. The create and edit forms now send the real auth_type for these modes,
so the temp server is not oauth2, needs_user_oauth_token is false, and the
exchange skips storage while still returning the token to the browser
session.
The server detail page's Tool Testing Playground gated its browser-held
token handling on the legacy PKCE-passthrough shape, so a
true_passthrough or oauth_delegate server listed tools unauthenticated
and surfaced 'Failed to fetch MCP tools' with no way to authorize. The
playground now treats both modes as browser-held-token servers: it
reads the sessionStorage token established by the create/edit
browser-only Authorize, forwards it via the x-mcp-{alias}-authorization
header, evicts it on a 401, and shows its own Authorize gate when the
token is absent.
That gate's flow uses the gateway's relayed authorize/register/token
endpoints with the real server id, which previously 400ed for anything
but oauth2. Those endpoints now also accept the client-forwarded token
modes (the minted token is upstream-audienced and browser-held; DCR
persistence stays off on this path), and registry builds run the same
RFC 9728/8414 endpoint discovery for these modes that oauth2 rows get,
since their rows never store an authorization_url.
true_passthrough and oauth_delegate persist no upstream credentials, so
the create/edit forms had no way to preview tools or configure the tool
allowlist: tools/list went upstream unauthenticated and came back 401.
This reuses the existing OAuth authorize machinery in browser-only mode
for those two auth types: the admin authorizes against the upstream
(DCR/PKCE, with optional client credentials for IdPs without dynamic
registration), the token lands in sessionStorage exactly like the
legacy PKCE-passthrough path, and the tools preview forwards it via the
per-server x-mcp-{alias}-authorization header, which the passthrough
resolver arm already accepts. Nothing is written to the server row or
the per-user credential store; the create payload keeps excluding
credentials for these auth types via AUTH_TYPES_REQUIRING_CREDENTIALS.
The tools preview endpoint now also extracts the Authorization header
for the two new auth types so the browser-held token reaches the
passthrough arm during create-time previews.
When an IdP denies SSO access it redirects back to /sso/callback with
error and error_description query params and no code param. The callback
previously fell through to the provider token exchange, which failed
with a generic "'code' parameter was not found in callback request"
400 that hides the real denial reason. Raise a 401 that surfaces the
IdP's error and description instead.
Ported from #26640 with conflicts resolved against current staging
The v2 OTel integration stamped litellm-specific error details as
error.code, error.stack_trace, and error.llm_provider, squatting on the
semconv-owned error.* namespace. They now live at
litellm.provider.error.code, litellm.provider.error.stack_trace, and
litellm.provider.error.llm_provider alongside the other vendor-extension
keys. error.type and error.message stay on the semconv keys.
* fix(guardrails): walk Responses-API text taxonomy in shared content helpers
Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently
drops all text on the /v1/responses path. AIM turns it into a loud 422 (
{"error":"No messages in the request"}); every other guardrail (Lakera v2,
Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret
detection) scans an empty payload and lets the request through unscanned.
Three defects, all in _content_utils.py:
1. _iter_text_parts_in_content recognised only part.type == "text", but the
Responses API uses input_text (request) and output_text (assistant).
2. _coerce_input_to_messages gated on "every item has a role key"; any
Responses input list containing a function_call or function_call_output
item failed the check and was wrapped as one opaque blob.
3. build_inspection_messages forwarded any role through, including a bare
tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze
reject with a schema error.
Fix walks the actual Responses item taxonomy (message, function_call,
function_call_output, bare content parts and strings), recognises
{text, input_text, output_text} everywhere, and coerces any role outside
{system, user, assistant} to user in the outbound inspection payload.
* style: ruff-format changed guardrail files
* test(guardrails): cover function_call_output string form; drop em-dash in new docstring
* fix(guardrails): map function_call_output straight to user role
Avoids ever materialising a schema-invalid bare tool message. The
downstream role-safety coercion in build_inspection_messages still
guards genuinely caller-supplied non-standard roles (developer,
function, custom values); add a regression test covering that path
so the coercion has real coverage after this simplification.
* test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages
* docs(test): soften AIM-specific claims in LIT-4294 test docstrings
Ryan's review flagged that several test docstrings assert AIM's
/fw/v1/analyze validates + rejects specific schema violations. That
behavior is customer-reported in the LIT-4294 writeup, not directly
verified by us. Rephrase to attribute the AIM 422 to the customer's
writeup and describe the underlying constraint as the OpenAI chat
schema; any downstream API that validates against that schema rejects
the same shape.
* refactor(guardrails): move unsupported-role coercion into AIM only
The generic coercion in build_inspection_messages collapsed any role
outside {system, user, assistant} to user for every caller of the
helper. Combined with the pre-existing apply_redacted_messages_back
write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400
on chat-completions tool-message masking into a silent semantic
corruption of the outbound request (role tool with tool_call_id got
rewritten to bare role user, dropping the assistant + tool_calls
sibling).
AIM specifically requires the coercion because its /fw/v1/analyze
validates the payload against the OpenAI chat schema; other guardrails
either do not validate roles or do their own reconstruction. Move the
coercion to AimGuardrail._build_aim_inspection_messages so the shared
helper keeps caller roles intact and no new cross-guardrail role
corruption is introduced. The pre-existing apply_redacted_messages_back
structural flatten remains as separate follow-up work.
function_call_output items still synthesise role user in the shared
helper because they have no natural role field, which is a different
concern from coercing a caller-supplied role.
* refactor(guardrails): preserve role fidelity in shared _content_utils
Shared inspection helpers should extract text and preserve semantic
role signals; role coercion for third-party schema safety stays inside
the guardrail that needs it (AIM).
Three shared-helper changes:
- Bare content-part dicts (input_text/output_text) with an explicit role
keep it; only role-less parts default to user.
- Responses message items already had their role preserved; the
behavior is now covered by an explicit test.
- function_call_output items default to role tool (semantic equivalent
of the chat-completions tool message shape) instead of role user, so
Responses and chat completions produce symmetric inspection payloads.
A caller-supplied role on the item is still preserved.
AIM's schema-safe coercion in _build_aim_inspection_messages already
handles the resulting role tool: it collapses to user before the POST
to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the
bare tool message (no tool_call_id can survive the flatten). Added a
regression test in test_aim.py covering that path.
Bedrock Converse supports cachePoint ttl (1h GA for Claude 4.5+), and
_get_cache_point_block maps cache_control.ttl -> cachePoint.ttl, but the
model parameter its allow-list gate requires was only threaded through
the system-message path. Every message-level path either called
_get_cache_point_block without model= (8 call sites in
_bedrock_converse_messages_pt / _pt_async) or hardcoded
CachePointBlock(type="default") (tool-result blocks and
_convert_to_bedrock_tool_call_invoke), so a requested 1h ttl silently
degraded to the 5-minute default - exactly on the conversation-tail
breakpoint that long-running agents need to survive tool calls longer
than 5 minutes.
- pass model= at the 8 _get_cache_point_block call sites
- tool-result blocks: capture the cache_control dict (was a boolean)
and route through _get_cache_point_block so ttl survives
- _convert_to_bedrock_tool_call_invoke: accept optional model and route
per-tool-call cache_control through _get_cache_point_block
Completes the ttl support added for system messages (#19848, #20326):
message-level cache_control now behaves identically.
Note: message-level cache_control on a content-less assistant message
emits no cachePoint at all today; that pre-existing gap is orthogonal
to ttl and left out of scope (per-tool-call placement covers it).
Co-authored-by: Arash <arashne@glia-ai.com>
* refactor(ui): point invitation links at the dedicated /onboarding route
Invitation and reset-password links were built as /ui?invitation_id=..., which lands on the dashboard index and renders the onboarding form inline. They now point at the standalone /ui/onboarding route, so the index no longer has to special-case invitations. Old links keep working unchanged; the index still renders onboarding inline for ?invitation_id until the migration closeout removes that branch.
Updates the three generators (the enterprise email builder, bulk user create, and the invitation/reset-password modal) and extracts the modal's URL building into a pure, unit-tested buildOnboardingUrl
Refs LIT-3687
* refactor(ui): guard buildOnboardingUrl against a missing invitation id
Return "" instead of emitting an invitation_id=undefined link when the id is not yet available, matching the existing empty-baseUrl guard. Placed after the SSO branch so the SSO link, which does not use the id, is unaffected
Refs LIT-3687
An org admin updating a team budget from the Hub UI was rejected with 401, because the route gate only recognizes an org admin when the request body carries organization_id while the UI sends team_id. For /team/update, resolve the target team's organization_id from team_id before the gate runs, so an org admin of the team's own org clears the org-scoped branch without the client passing organization_id. Team admins and cross-org admins stay denied at the gate, and callers that already pass organization_id are unaffected, so the existing /team/update authorization matrix is unchanged
* fix(rerank): log optional_rerank_params at debug not info to avoid leaking request content
* test(rerank): exercise sync rerank path so coverage counts the log line
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex): forward realtime health check params
* refactor(vertex): resolve realtime health check params via VertexBase helpers
Address review feedback on the vertex param forwarding: pass model_params
through to _realtime_health_check and resolve vertex credentials, project,
and location inside the vertex_ai branch using the existing
VertexBase.safe_get_vertex_ai_* helpers, so provider-specific key extraction
no longer lives in litellm_core_utils and dict-typed vertex_credentials are
supported
* test(vertex): move realtime health check test to mapped unit test path
codecov/patch reported the vertex branch of _realtime_health_check as
uncovered because tests/litellm_utils_tests is not part of the unit test
groups that upload coverage. Move the test into
tests/test_litellm/litellm_core_utils/test_health_check_helpers.py, which
the core-utils group runs, keeping the same end-to-end assertions through
litellm.ahealth_check
---------
Co-authored-by: Aleksandr Liadov <72351793+AleksandrLiadov@users.noreply.github.com>
The connect gate resolved x-mcp-{alias}-authorization by matching the raw
lowercased alias/server_name/name only, but dashboard clients send
x-mcp-{sanitize_mcp_alias_for_header(alias)}-authorization, and egress resolves
those through lookup_mcp_server_auth_in_headers, which also tries the sanitized
alias. So a per-server token bound with a sanitized alias (e.g. alias 'pt-server'
arriving as header key 'pt_server') was forwarded at egress but still triggered a
preemptive 401 at connect. _client_has_per_server_auth_header now resolves through
the same lookup_mcp_server_auth_in_headers egress uses, so connect and egress
agree on which header names match.
The listing fan-out withholds the request-wide Authorization from a
true_passthrough / oauth_delegate server when another server in scope also
consumes it, so one bearer is not replayed across upstreams. The later
server.extra_headers copy loop did not honor that decision: a server listing
Authorization in extra_headers would re-copy the withheld bearer from
raw_headers. The withhold decision is now computed once and applied to both
the forwarding branch and the extra_headers loop.
The preemptive 401 for true_passthrough and oauth_delegate only inspected the
request-wide Authorization, so a caller who bound the upstream token via the
per-server x-mcp-{alias}-authorization header (the required shape in a
multi-server aggregate, where the request-wide Authorization is withheld) was
spuriously 401'd at initialize even though egress already honors that header.
The gate now recognizes the per-server header for both modes via a shared
helper, mode-correctly: true_passthrough treats any Authorization or the
per-server header as the upstream token, oauth_delegate keeps requiring a
distinct x-litellm-api-key so a lone Authorization consumed for admission is
never mistaken for an upstream token. The preemptive raise is also gated to
single-server scopes so a multi-server aggregate degrades gracefully instead
of one missing token 401-ing the whole connect.
For the client-forwarded token modes the gateway relays the caller's
Authorization to the upstream, so a request carrying more than one
Authorization header would make which token is forwarded ambiguous (the
ASGI header list collapses to last-wins) and could diverge from what
admission inspected. Multiple Authorization headers is malformed for
bearer auth anyway (RFC 9110: not a comma-combinable field), so the
ingress header converter now fails closed with a 400 instead of silently
keeping one. Applies to every MCP request, not just passthrough.
* fix(guardrails): forward grayswan scan id header
* test(guardrails): cover grayswan scan id forwarding
* fix(guardrails): prevent overwriting existing metadata headers when extracting scan id
* test(guardrails): cover header merging logic
* chore(guardrails): fix formatting
* test(guardrails): enforce case preservation
* chore(guardrails): corrected grayswan type annotations
* fix(guardrails): sanitized grayswan header metadata
* test(guardrails): covered grayswan logging headers
* fix(guardrails): guard grayswan header lookup against None and drop dead comment
- Fall back to {} when proxy_server_request is explicitly None so
request_data.get(...).get('headers') never raises AttributeError.
- Remove the commented-out user_api_key_auth pop; it was inert and
greptile called it out as ambiguous.
---------
Co-authored-by: Theodore Drzewinski <93957989+tediferJones@users.noreply.github.com>
The v2 emitter has never stamped error.message / error.code /
error.stack_trace / error.llm_provider as span attributes; only error.type
reached the wire. Backends that flatten span attributes into label
indexes (Elastic APM labels.error_*, Datadog span tags) lost these
four fields when v2 became the active integration on v1.90+ for
otel_v2-flagged deployments. The pre-existing exception span event
carrying the full message (LIT-3758) is unchanged; the message now
rides both places at once, matching v1s shape.
SpanError grows three optional detail fields; _parse_error threads
them from StandardLoggingPayloadErrorInformation; the emitters error
branch stamps them via a new module-level helper, guarded per field so
guardrail-shape errors are not polluted with empty attributes. New
semconv constants mirror open_inference.ErrorAttributes byte-for-byte,
so v1 and v2 consumers read the same keys.
Regression tests extend the mapped test files under
tests/test_litellm/integrations/otel/. pytest reports 243 passed.
* fix(health): bridge litellm_metadata into logging object in _batch_health_check
* Update litellm/litellm_core_utils/health_check_helpers.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(health): address review - share single metadata copy, conditional api_base, add tests
- Only set api_base in litellm_params when a value actually exists;
providers like bedrock/vertex/gemini resolve it implicitly and an
empty string overwrites their resolution.
- Use a single .copy() for both metadata and litellm_metadata to
prevent downstream drift between the two references.
- Add 6 unit tests covering metadata bridging, api_base omission,
guard conditions, and dispatch routing.
Signed-off-by: pramod <pramod.b@pfizer.com>
* refactor(health): use update_from_kwargs helper for metadata bridge
Collapses the manual metadata/litellm_metadata plumbing in
_batch_health_check into a single update_from_kwargs call, matching
how the sibling batch/image/rerank/ocr surfaces bridge metadata onto
the pre-injected logging object. Drops the bare Dict typing and the
inline comment, and switches the tests to assert against the helper.
---------
Signed-off-by: pramod <pramod.b@pfizer.com>
Co-authored-by: pramod <pramod.b@pfizer.com>
Co-authored-by: Pramod B <155433727+BPRMD18@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
On re-auth against a server with a persisted DCR client, register_client_with_server
short-circuits and returns a placeholder client_secret ("dummy") that the browser
echoes back to /token. exchange_token_with_server overrode the caller's client_id
with the persisted one but still fell back to the caller's secret when the server
had none stored, so a persisted public PKCE client (which has no secret) was paired
with the literal string "dummy" and the IdP rejected the exchange with 401 on every
re-authorization; the proxy surfaced that as a 500. First connects and brand-new
servers worked because a real DCR registration ran and no placeholder existed.
Resolve the secret from the server whenever the server's client_id wins, so a
secretless public client sends no client_secret at all
key_alias can become the secret name used by external secret manager
integrations (HashiCorp Vault, CyberArk Conjur) when store_virtual_keys is
enabled. Add raise_if_unsafe_secret_name, a shared validation check applied
unconditionally before a secret name reaches either integration or the
/key/generate, /key/update, and /key/regenerate API boundary, independent
of the existing enable_key_alias_format_validation opt-in flag.
Also hardens the Vault URL builder to percent-encode reserved characters
in secret_name (preserving "/" and "@"), and switches the Conjur policy
body to a real YAML serializer instead of raw string interpolation.
An MCP tool call that completes with CallToolResult.isError=true correctly
returns HTTP 200 per the MCP spec, but the shared post-call logging helper
always fired async_success_handler, so the standard logging payload carried
status=success and OTel (whose _parse_error only marks ERROR on
status=failure) showed green spans for failed tools.
The helper now checks the result after async_post_mcp_tool_call_hook runs
(guardrails may flip isError there) and routes error results to the failure
path: success gates are consumed so the @client wrapper cannot enqueue a
success log, failure_handler and async_failure_handler fire with a new
MCPToolResultError carrying the tool's first text content, and
post_call_failure_hook records the failure the same way raised exceptions
already do. Raised exceptions never reach the helper, so no double failure
logging. HTTP wire behavior is unchanged
Resolves LIT-4081
Follow-up to #31411 (superseded and merged as #31997). Two related fixes
so LiteLLM callers see what TinyFish actually returns, plus small
correctness cleanups.
## Response headers surfaced on _hidden_params
TinyFish sets useful response headers (x-request-id on every response,
retry-after and x-ratelimit-limit on 429s). Previously these were only
accessible via BaseLLMException.headers on error paths; on the success
path they were dropped entirely.
Fix: stash headers on both LiteLLM-conventional channels, matching the
pattern used by Gemini / Volcengine / Manus / ChatGPT / OpenAI-responses
providers.
- `_hidden_params["headers"]` -- raw dict from httpx, all keys lowercased.
- `_hidden_params["additional_headers"]` -- passed through
process_response_headers, which prefixes any x-litellm-* provider
header with `llm_provider-` so downstream LiteLLM code that trusts
bare x-litellm-* markers can't be spoofed (values still survive
under the prefixed key for observability).
## Top-level response extras (query, total_results, page, future fields)
transform_search_response was building a fresh SearchResponse from just
`results`, silently dropping every top-level field TinyFish's response
carries beyond `results` / `object`.
Fix: mutate parsed.results to its truncated slice and return the same
SearchResponse instance rather than reconstructing. Every field pydantic
populated during model_validate -- declared attributes AND extras
(query, total_results, page, parameter_warnings, and any future TinyFish
additions) -- survives regardless of which storage bucket holds it.
Robust against upstream schema evolution: if LiteLLM later promotes a
field from extras to declared, this code needs no change.
## Code cleanup
- List-valued custom params JSON-encoded on the wire (matching the
existing dict handling), so callers can pass a natural Python list
for JSON-array wire params.
- URL-encodable-params adapter accepts float in addition to
str / int / bool; server-side rejection of a wrong-typed float now
surfaces cleanly with `TinyFish Search:` attribution + docs link.
- Assorted comment / docstring / test-fixture hygiene (no logic changes).
## Tests
70 unit + integration tests pass locally. Live-tested against
production TinyFish with 6 diverse queries (basic / max_results /
country=US / language=ja / domain filter / fetch={"format":"html"}) --
all 6 pass every expected-behavior check.
* fix(mcp): drop the cached per-user OAuth token when the credential row changes
The v2 authorization_code chain Cached(Refreshing(V2PerUserTokenStore)) caches a positive token
until its expires_at (or 300s without one), and CachedOAuthTokenStore.invalidate had no callers,
so a re-authorization or revocation wrote the DB while egress kept serving the replaced token
from the in-process cache until its TTL. LazyPerUserOAuthTokenStore now exposes invalidate,
MCPServerManager threads it to the write side, and the three credential write sites (the OAuth
callback, the Tools-tab persist endpoint, and the revoke endpoint) drop the cache entry after
the row changes. The v2 refresher's own persist stays untouched; RefreshingTokenStore already
feeds the rotated token back into the cache in the same fetch
* test(mcp): pin cache invalidation on the revoke already-gone branch
Greptile's review flagged that only the happy-path delete asserted the invalidate; a refactor
moving the call inside the try block would silently skip the cache drop when the row was
already deleted by a concurrent request while the cache still held the revoked token. The new
test fails on exactly that mutation
* test(mcp): cover invalidate on the redis-backed lazy store path
Codecov flagged the redis fast path of LazyPerUserOAuthTokenStore.invalidate as unexercised;
the existing invalidate tests only ran the no-redis chain. The new test builds the redis chain
via a fetch and asserts a subsequent invalidate reaches the same store instance without a
rebuild
Add a tag_rpm_limit field to virtual keys so each request tag gets its own independent RPM counter on the v3 rate limiter. A key configured with per-tag limits tracks each tag/group separately, and requests whose tag has no configured limit fall back to the key-level limit. Includes the dashboard UI to manage per-tag limits on key create and edit.
Resolves LIT-3147
* feat(ui): expose MCP max_concurrent_requests in server create and edit forms
The proxy has enforced a per-server outbound tool-call concurrency cap
(max_concurrent_requests) across every MCP egress path since #31641, and the
management API has accepted the field on create and update all along, but the
dashboard offered no way to set it. Add an optional Max Concurrent Requests
input to the MCP server create and edit forms; it applies to every auth type
and transport, so it renders unconditionally rather than gated on auth mode.
Clearing the field on edit sends null so the stored limit is unset.
Also rebuild the per-server semaphore when the configured limit changes.
Previously the semaphore was created once per server_id and never resized, so
an edited limit only took effect after a proxy restart even though the new
value was persisted and reloaded into the registry.
* feat(ui): mark MCP max concurrent requests field label as optional
* test(ui): stop OBO create-form tests from timing out on CI
The token-exchange payload test and the Entra scope-required test filled five
text fields with user.type, which dispatches a full keystroke sequence per
character; every input event runs the antd form onValuesChange handler and
re-renders the whole CreateMCPServer tree, roughly 120 renders per test. As
the form grew the two tests reached 8s and 18s locally, which crosses the 30s
vitest timeout on slower CI containers; ui_unit_tests failed twice this way.
Switch the plain text fields to fireEvent.change (one input event per field),
matching the existing stdio test pattern. Both tests assert form output, not
keystroke behavior, and now run in about 3s each.
