* fix(spend-logs): honor store_prompts_in_spend_logs for guardrail_information (LIT-4314)
_get_spend_logs_metadata passed guardrail_information entries through
verbatim, so guardrail hooks that echo the LLM request into
guardrail_response leaked the raw prompt into LiteLLM_SpendLogs.metadata
regardless of store_prompts_in_spend_logs. This mirrored the pre-existing
gap for the other prompt-carrying fields (vector_store_request_metadata,
error_information, etc.), which already sanitize via
_should_store_prompts_and_responses_in_spend_logs.
Add _sanitize_guardrail_information_for_spend_logs alongside the other
per-field sanitizers and wire it into _get_spend_logs_metadata. When the
flag is False the sanitizer replaces guardrail_request and
guardrail_response with REDACTED_BY_LITELM_STRING while preserving every
other typed field on the entry (name, provider, mode, status, timings,
action, violation_categories, risk_score, masked_entity_count, ...) so
guardrail dashboards keep working. When the flag is True (or the field
is None) the entries pass through unchanged.
Widen StandardLoggingGuardrailInformation.guardrail_request from
Optional[dict] to Optional[Union[dict, str]] so the redacted sentinel
satisfies the TypedDict without needing a cast; guardrail_response
already accepted str.
Regression tests cover the three cases (flag=False redacts,
flag=True passes through, None passes through) plus an end-to-end
get_logging_payload path that fails if the wire-in at line 139 is
reverted.
* chore(spend-logs): review nits (one-shot dict build, scrub identifier in tests)
- _redact_prompt_fields_in_guardrail_entry now returns the redacted
dict in one expression instead of seed-then-mutate (TYPE-3)
- swap the illustrative guardrail_name in the new test fixtures for
a generic 'demo-echo-guard' identifier
* chore(spend-logs): only redact guardrail prompt fields when caller supplied them
Greptile P2: the sanitizer was unconditionally writing REDACTED_BY_LITELM
into both guardrail_request and guardrail_response on the copy, so entries
that never carried one of those fields (e.g. a guardrail that only emits
a guardrail_response) came out with a phantom guardrail_request key added.
Guard both assignments with an in-check so the output shape is stable.
Add a mutation-checked regression test that fails if either guard is
removed.
* fix(spend-logs): also redact match_details and classification in guardrail_information
The initial LIT-4314 fix redacted guardrail_request and guardrail_response,
but two other typed fields on StandardLoggingGuardrailInformation also
carry raw prompt content when a first-party guardrail populates them:
- litellm_content_filter/content_filter.py:1676 sets classification =
dict(CompetitorIntentDetection), whose evidence[*].match is a substring
taken directly from the user's normalized prompt (see
litellm_content_filter/competitor_intent/base.py:184-194).
- block_code_execution/block_code_execution.py:571 sets match_details =
guardrail_response = [dict(d) for d in detections], where detections
carry the fenced-code-block content extracted from the user's message.
Reproduced live against localhost:4000 with store_prompts_in_spend_logs
false and a custom guardrail passing tracing_detail with both fields:
before this commit the raw prompt shows up in metadata.guardrail_information[0]
under match_details and classification; after, both are the sentinel.
Widen the two TypedDict fields to Optional[Union[..., str]] so the
sentinel string satisfies the schema without a cast, and consolidate
the redaction set into a tuple so future prompt-carrying additions are
one-line changes.
* fix(spend-logs): normalize non-list guardrail_information shapes in sanitizer
xecguard's logging hook (xecguard.py:246) assigns a bare dict to
standard_logging_object['guardrail_information'] instead of a list,
violating the typed contract Optional[List[StandardLoggingGuardrailInformation]].
Without defensive normalization, _sanitize_guardrail_information_for_spend_logs
iterates the dict's string keys and _redact_prompt_fields_in_guardrail_entry
raises TypeError on {**'guardrail_name'}, which get_logging_payload's
downstream update_database catches with a broad except and silently drops
the entire spend-log write for that request.
Normalize a bare-dict input to a single-item list at the sanitizer's
entry point, and skip any non-dict entries defensively (matching OTEL's
existing isinstance filter at opentelemetry.py:1751-1753 for the same
field). Downstream readers already model this defensively; make the
spend-log write path match.
The root cause is xecguard's writer, not the sanitizer. That is being
tracked as a separate ticket; this PR keeps xecguard-enabled deploys
from silently losing spend logs when store_prompts_in_spend_logs=false.
* fix(types): declare guardrail Union members str-first to avoid poisoning typing cache
CPython's typing module caches Union[...] order-insensitively (first-
construction wins), and litellm/types/utils.py has no 'from __future__
import annotations', so its unions are constructed eagerly at import
time -- before any proxy model. Declaring guardrail_request,
classification, and match_details with dict-first ordering seeds the
typing cache with a dict-first tuple, and later proxy models that
declare custom_llm_provider / model_aliases / vertex_credentials as
Optional[Union[str, dict]] pick up the same dict-first object.
Downstream, Pydantic's get_args() then reports anyOf in dict-first
order, FastAPI emits the OpenAPI accordingly, and 'npm run gen:api'
produces a schema.d.ts diff on unrelated fields, tripping the schema-
sync CI check.
Behaviorally identical in Python and at the wire; the flip only reorders
the union members so the first construction matches how the codebase
had always declared these unions, and 'npm run gen:api' now produces a
zero diff against the committed schema.d.ts.
The multi-server list path already relays an upstream 401 from a client-forwarded
server (true_passthrough / oauth_delegate) as an MCPUpstreamAuthError so the caller
re-runs its own upstream OAuth. The single-server REST call path did not: an upstream
401 was masked as a graceful isError result, so an MCP client holding an expired
upstream token never learned it had to re-authenticate
Relay the upstream 401 on the call path too. For these modes the manager calls the
client with raise_on_error=True, extracts the WWW-Authenticate through the existing
upstream-auth exception walk, and raises MCPUpstreamAuthError; the REST endpoint turns
it into a real 401 + WWW-Authenticate. Only 401 is treated as a re-auth signal (a 403 is
a genuine authorization failure that re-auth will not fix, so it stays a masked isError
with a visible warning), matching the list path and MCPUpstreamAuthError's contract. The
legacy oauth2 + delegate_auth_to_upstream mode is deliberately left off the call-path
relay since it is being removed
To keep this expected caller-must-reauth signal from tripping error-rate alerts, the
client layer logs at debug when the caller opted into raise_on_error and therefore owns
the exception (both call_tool/list_tools and the run_with_session helper they share, so an
expected re-auth emits no warning per call either), the manager's non-auth branch logs the
exception type only (never str(e), which for an httpx error embeds the upstream URL a
credential can hide in), and the streamable and REST handlers log the relayed 401 at info
rather than as an error with a traceback
Tests cover the manager raising on a client-forwarded 401 while keeping a 403/503 as a
masked isError, the client-layer debug-vs-error logging split, the streamable handler's
informational isError, and the REST endpoint relaying both the direct and virtual
mcp_tool_call branches as a real 401 + WWW-Authenticate; each was mutation-checked to fail
when the corresponding behavior is broken
Keep the full provider exception in server-side logs only; the client
receives a fixed actionable message. Also follow implicit exception
context when detecting context window overflows and pin the detection
variants plus the redaction in tests
Resolves LIT-4284
When the embedding model exceeded its context window, the MCP semantic
tool filter silently passed all tools through and reported N->N success
in the filter header; when the overflow happened while embedding tool
descriptions at router build time, the hook was never registered at all
and filtering was silently disabled
Semantic filtering now fails closed on context window overflows: the
request is rejected with HTTP 400 and a message that names the embedding
model and advises switching to one with a larger context window or
disabling the filter. Build time overflows are recorded on the filter so
the hook still registers and blocks MCP tool requests with the same
actionable error while leaving native-only requests untouched. The
dashboard test panel renders the backend message in an error banner
instead of a success state. OpenAI's embedding overflow message
(maximum input length is N tokens) now maps to ContextWindowExceededError
The per-row delete_many loop becomes a single delete filtered to the enumerated OAuth users'
(user_id IN, server_id) pairs; same rows deleted, same BYOK-sparing precision, same count-mismatch
detection, one round-trip instead of N
LiteLLM_MCPUserCredentials stores BYOK API keys in the same column as per-user
OAuth tokens, so the purge on a mint-relevant config change now deletes only
rows whose payload decodes as an OAuth2 credential, each by its
(user_id, server_id) pair, instead of every row for the server. An api_key
server whose url changes purges nothing. delete_mcp_server now also
invalidates each enumerated user's cached token so a re-created server reusing
the id cannot serve tokens minted for the deleted one, and both cache drops
are best-effort
The snapshot read only feeds the stale-token purge decision; leaving it unguarded meant a failed
read would 500 an edit whose update would have succeeded, and it broke
test_edit_mcp_server_redacts_credentials, whose mocked prisma is not awaitable on the un-patched
get_mcp_server path. A failure now logs and skips the purge, consistent with the purge half already
being best-effort. Adds the first endpoint-level coverage of the edit purge wiring: purge on a
mint-relevant change, no purge when the identity is unchanged, and edit success with purge skipped
when the snapshot read raises
The purge takes an injectable invalidate_token_cache callable defaulting to the manager's shared
invalidation, and MCPServerManager takes an injectable per_user_token_cache alongside the existing
per_user_oauth_token_store, so tests inject fakes instead of monkeypatching the global manager and
the module-level cache. The new identity helpers drop Any for object throughout
Review follow-ups on the stale-token invalidation. The backend identity now decrypts client_id and
client_secret before comparing: the stored values are NaCl-encrypted with a fresh nonce on every
write, so comparing ciphertext flagged every routine save as a mint-relevant change and purged
per-user tokens that were still valid. The identity also gains spec_path, the audience for OpenAPI
servers, and parses credentials stored as a JSON string
The purge now routes each (user, server) through the manager's invalidate_user_oauth_token_cache,
which becomes the single invalidation point covering both the legacy per-user token cache and the
v2 per-user OAuth token store; previously the purge evicted only the legacy cache while the revoke
path evicted only the v2 store, so each path left the other cache serving a replaced token until
its TTL. A credential row racing in between the find and the delete is now detected via the
delete_many count and logged; its cache entry expires by TTL
On the dashboard, CLEARED_ON_INVALIDATION and the staleness check move to types.tsx as the single
shared implementation for both forms. The edit form's transport handler now rechecks the identity
after its programmatic setFieldsValue calls, which antd does not report through onValuesChange, so
a token no longer survives a transport switch that clears the mint target. The create form rebuilds
formValues from the post-reset form state after an invalidation instead of publishing the pre-reset
snapshot, so the tool preview can no longer refetch with the discarded DCR client. Both transport
handlers now share the recheck, which also stops the create form from over-invalidating on an
http to sse swap that keeps the same url and therefore the same audience
An admin who ran Authorize & Fetch and then changed a field that determines which upstream OAuth
token gets minted kept using the stale token for tool preview, sessionStorage, and (on the backend)
the stored per-user credential and its cache. Grounded in RFC 8707/8693 and the MCP auth spec, a token
is bound to one tuple: resource/audience (url), OAuth mode/grant (auth_type, oauth_flow_type), the
authorization-server endpoints, and the OAuth client + scopes. A shared getOAuthAuthorizationIdentity
captures exactly those fields; transport (http/sse on the same url is the same audience) and
delegate_auth_to_upstream (a downstream-usage toggle never sent to the authorize request) are excluded.
UI: both the create and edit forms now discard the held token (React state / sessionStorage / hook,
plus the fetched token + DCR client in form.credentials) whenever the identity diverges from the one it
was authorized against, re-applying the admin's in-flight edit so it is never wiped. The check lives in
one shared helper so the two forms cannot drift.
Backend: editing an MCP server now compares the pre/post identity and, on a mint-relevant change, purges
every stored per-user OAuth credential for the server (DB row + per-user token cache) so no user
forwards a token minted for a resource/AS/client that no longer matches. Best-effort; a purge failure
never fails the update.
The redacted resource kept the path, but hosted MCP servers routinely embed the credential in the
path (for example /mcp/s/<token>/mcp), and mcp_tool_call_metadata is readable by a caller who can
invoke the tool, so the path leaked the upstream credential into spend logs. Only scheme, host, and
port are logged now
Authorization doubles as the admission fallback when x-litellm-api-key is absent, so a caller who
authenticated the preview request that way had their LiteLLM key forwarded to the upstream as the
oauth2/client-forwarded token. The preview now forwards Authorization only when the primary
admission header is present, which is how the dashboard has always sent it; with no primary header
there is no upstream token on the request at all. Applies to oauth2 and both client-forwarded
modes; parametrized regression test plus the admission header added to the existing extraction
tests to mirror the real UI request shape
* fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException
The Bedrock-specific GuardrailInterventionNormalStringError predates the
unified guardrails refactor and no proxy code path handles it, so a block
with the flag set surfaced as an uncaught Exception -> HTTP 500 in pre_call
mode and was silently discarded in during_call mode (model call proceeded
in the parallel asyncio.gather; the block hook's data["mock_response"]
mutation happened after route_request had already unpacked kwargs).
Convert the block to ModifyResponseException at the raise site inside
make_bedrock_api_request. That exception is the industry-standard proxy
contract already caught in proxy_server, anthropic_endpoints, response_api
_endpoints, and pass_through_endpoints; it turns into a 200 response with
finish_reason=content_filter and the block message as content, which is
exactly what the flag was documented to yield. Post-call blocks attach
the LLM response to original_response so the synthetic reply reports the
upstream call's real token usage instead of zero.
Deletes the now-orphaned GuardrailInterventionNormalStringError class and
the dead create_guardrail_blocked_response / mock_response plumbing in the
Bedrock hooks; updates the existing tests that had locked in the buggy
contract.
Resolves LIT-4186
* chore(guardrails/bedrock): drop dead str branch in _update_messages_with_updated_bedrock_guardrail_response
Follow-up to the disable_exception_on_block fix. That method used to
receive either a BedrockGuardrailResponse or a plain string (the block
message, when the flag was set). Now that a block always raises
ModifyResponseException before this method runs, the string branch is
unreachable; tighten the type to BedrockGuardrailResponse and delete
the guard.
* fix(guardrails/bedrock): streaming post_call block yields synthetic stream instead of surfacing as SSE 500
Regression from the LIT-4186 refactor: pre-refactor, the streaming
post_call iterator caught GuardrailInterventionNormalStringError locally
and replaced the assembled response with a synthetic content-filter
message, then re-emitted it as chunks via MockResponseIterator. After
the refactor the exception was re-raised as ModifyResponseException,
which async_streaming_data_generator serializes as a proxy 500 error
frame because the SSE response headers are already flushed by the time
the block fires.
Non-streaming paths still let ModifyResponseException propagate to the
endpoint handler (which converts it into a 200). Streaming can't do
that, so keep the local synthesis: on the exception, rebind the
assembled response to a ModelResponse whose single choice carries the
block message as content and finish_reason=content_filter, and let the
downstream MockResponseIterator emit it as chunks. Same shape a
non-streaming block produces.
Adds a mapped-file regression test that mutation-kills the raise
behavior and locks in the synthetic-stream contract.
* fix(guardrails/bedrock): preserve upstream usage on streaming post_call block
Non-streaming post_call blocks report the upstream LLM call's real
token usage via ModifyResponseException.original_response, which the
endpoint handler unwraps through _blocked_response_usage. Streaming
post_call synthesizes its own ModelResponse locally (the exception
can't escape the SSE generator), and previously left .usage unset,
so the client saw accurate billing on non-streaming blocks and zero
on streaming blocks -- silent revenue leak.
Copy the assembled response's .usage onto the synthetic block
response before yielding. Pre-refactor code had the same gap
(create_guardrail_blocked_response never set usage); this is a net
improvement, not a regression fix.
* fix(proxy): capture logging_obj before post_call_failure_hook pops it in ModifyResponseException streaming path
post_call_failure_hook removes litellm_logging_obj from request_data before
iterating callbacks (it's not serialisable). The streaming branch of the
ModifyResponseException handler read it from _data after that call, so it
always received None and CustomStreamWrapper.__init__ crashed with
AttributeError: NoneType has no attribute model_call_details.
Capture it before the hook runs so the streaming path gets a valid object.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(proxy): add regression for streaming ModifyResponseException logging_obj capture
Covers the bug where logging_obj was read from request_data after
post_call_failure_hook had already popped it, causing CustomStreamWrapper
to crash with AttributeError.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(proxy): drive real chat_completion in ModifyResponseException streaming logging_obj regression
The original test inlined the fix pattern (capture before pop) in its
own body rather than calling the actual chat_completion handler in
proxy_server.py, so a revert of the fix left the test passing.
Confirmed via mutation check: reverting the two-line source fix and
re-running left the test green.
Rewrite the test to drive chat_completion directly:
- patch _read_request_body so chat_completion sees the seeded dict
- patch ProxyBaseLLMRequestProcessing.base_process_llm_request to
raise ModifyResponseException with the same request_data
- patch proxy_logging_obj so post_call_failure_hook mutates the dict
the way production does (pops litellm_logging_obj)
- intercept CustomStreamWrapper.__init__ and assert logging_obj is
the non-None object seeded in request_data
Mutation-verified: reverting the source fix now surfaces the exact
production crash inside CustomStreamWrapper's __init__
(AttributeError: NoneType has no attribute model_call_details) rather
than a silently-passing test.
Addresses Greptile P1 on PR #32665.
---------
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(guardrails/bedrock): honor disable_exception_on_block by raising ModifyResponseException
The Bedrock-specific GuardrailInterventionNormalStringError predates the
unified guardrails refactor and no proxy code path handles it, so a block
with the flag set surfaced as an uncaught Exception -> HTTP 500 in pre_call
mode and was silently discarded in during_call mode (model call proceeded
in the parallel asyncio.gather; the block hook's data["mock_response"]
mutation happened after route_request had already unpacked kwargs).
Convert the block to ModifyResponseException at the raise site inside
make_bedrock_api_request. That exception is the industry-standard proxy
contract already caught in proxy_server, anthropic_endpoints, response_api
_endpoints, and pass_through_endpoints; it turns into a 200 response with
finish_reason=content_filter and the block message as content, which is
exactly what the flag was documented to yield. Post-call blocks attach
the LLM response to original_response so the synthetic reply reports the
upstream call's real token usage instead of zero.
Deletes the now-orphaned GuardrailInterventionNormalStringError class and
the dead create_guardrail_blocked_response / mock_response plumbing in the
Bedrock hooks; updates the existing tests that had locked in the buggy
contract.
Resolves LIT-4186
* chore(guardrails/bedrock): drop dead str branch in _update_messages_with_updated_bedrock_guardrail_response
Follow-up to the disable_exception_on_block fix. That method used to
receive either a BedrockGuardrailResponse or a plain string (the block
message, when the flag was set). Now that a block always raises
ModifyResponseException before this method runs, the string branch is
unreachable; tighten the type to BedrockGuardrailResponse and delete
the guard.
* fix(guardrails/bedrock): streaming post_call block yields synthetic stream instead of surfacing as SSE 500
Regression from the LIT-4186 refactor: pre-refactor, the streaming
post_call iterator caught GuardrailInterventionNormalStringError locally
and replaced the assembled response with a synthetic content-filter
message, then re-emitted it as chunks via MockResponseIterator. After
the refactor the exception was re-raised as ModifyResponseException,
which async_streaming_data_generator serializes as a proxy 500 error
frame because the SSE response headers are already flushed by the time
the block fires.
Non-streaming paths still let ModifyResponseException propagate to the
endpoint handler (which converts it into a 200). Streaming can't do
that, so keep the local synthesis: on the exception, rebind the
assembled response to a ModelResponse whose single choice carries the
block message as content and finish_reason=content_filter, and let the
downstream MockResponseIterator emit it as chunks. Same shape a
non-streaming block produces.
Adds a mapped-file regression test that mutation-kills the raise
behavior and locks in the synthetic-stream contract.
* fix(guardrails/bedrock): preserve upstream usage on streaming post_call block
Non-streaming post_call blocks report the upstream LLM call's real
token usage via ModifyResponseException.original_response, which the
endpoint handler unwraps through _blocked_response_usage. Streaming
post_call synthesizes its own ModelResponse locally (the exception
can't escape the SSE generator), and previously left .usage unset,
so the client saw accurate billing on non-streaming blocks and zero
on streaming blocks -- silent revenue leak.
Copy the assembled response's .usage onto the synthetic block
response before yielding. Pre-refactor code had the same gap
(create_guardrail_blocked_response never set usage); this is a net
improvement, not a regression fix.
Two review findings on the passthrough modes.
The tool-call log records the upstream MCP server URL as mcp_server_resource,
which is persisted in spend-log metadata and sent to logging callbacks. A URL
carrying embedded userinfo or a secret query parameter would leak into logs, so
the value is now redacted to its bare resource identifier (scheme + host + path);
userinfo, query string, and fragment are stripped before it is logged.
The listing fan-out withholds the request-wide Authorization from a
true_passthrough / oauth_delegate server when another server in scope also
consumes it, so one bearer is not replayed across upstreams. The later
server.extra_headers copy loop did not honor that decision: a server listing
Authorization in extra_headers would re-copy the withheld bearer from raw_headers.
The withhold decision is now computed once and applied to both the forwarding
branch and the extra_headers loop.
Two correctness fixes for the client-forwarded token modes.
The preemptive-401 connect gate for true_passthrough and oauth_delegate
only inspected the request-wide Authorization, so a caller who bound the
upstream token via the per-server x-mcp-{alias}-authorization header (the
mandatory shape in a multi-server aggregate, where the request-wide
Authorization is withheld) was spuriously 401'd at connect even though
egress already honors that header. The gate now recognizes the per-server
header for both modes via a shared helper, mode-correctly: true_passthrough
treats any Authorization or the per-server header as the upstream token,
oauth_delegate keeps requiring a distinct x-litellm-api-key so a lone
Authorization consumed for admission is never mistaken for an upstream
token. The preemptive raise is also gated to single-server scopes so a
multi-server aggregate degrades gracefully (the listing absorbs a
per-server failure) instead of one missing token 401-ing the whole connect.
The browser-only Authorize flow was writing the upstream access and refresh
token to LiteLLM_MCPUserCredentials, contradicting the modes' persist-nothing
contract: the temp OAuth-relay server was cached with a hardcoded oauth2
auth_type, so needs_user_oauth_token was true and the token exchange stored
it. The create and edit forms now send the real auth_type for these modes,
so the temp server is not oauth2, needs_user_oauth_token is false, and the
exchange skips storage while still returning the token to the browser
session.
The server detail page's Tool Testing Playground gated its browser-held
token handling on the legacy PKCE-passthrough shape, so a
true_passthrough or oauth_delegate server listed tools unauthenticated
and surfaced 'Failed to fetch MCP tools' with no way to authorize. The
playground now treats both modes as browser-held-token servers: it
reads the sessionStorage token established by the create/edit
browser-only Authorize, forwards it via the x-mcp-{alias}-authorization
header, evicts it on a 401, and shows its own Authorize gate when the
token is absent.
That gate's flow uses the gateway's relayed authorize/register/token
endpoints with the real server id, which previously 400ed for anything
but oauth2. Those endpoints now also accept the client-forwarded token
modes (the minted token is upstream-audienced and browser-held; DCR
persistence stays off on this path), and registry builds run the same
RFC 9728/8414 endpoint discovery for these modes that oauth2 rows get,
since their rows never store an authorization_url.
true_passthrough and oauth_delegate persist no upstream credentials, so
the create/edit forms had no way to preview tools or configure the tool
allowlist: tools/list went upstream unauthenticated and came back 401.
This reuses the existing OAuth authorize machinery in browser-only mode
for those two auth types: the admin authorizes against the upstream
(DCR/PKCE, with optional client credentials for IdPs without dynamic
registration), the token lands in sessionStorage exactly like the
legacy PKCE-passthrough path, and the tools preview forwards it via the
per-server x-mcp-{alias}-authorization header, which the passthrough
resolver arm already accepts. Nothing is written to the server row or
the per-user credential store; the create payload keeps excluding
credentials for these auth types via AUTH_TYPES_REQUIRING_CREDENTIALS.
The tools preview endpoint now also extracts the Authorization header
for the two new auth types so the browser-held token reaches the
passthrough arm during create-time previews.
When an IdP denies SSO access it redirects back to /sso/callback with
error and error_description query params and no code param. The callback
previously fell through to the provider token exchange, which failed
with a generic "'code' parameter was not found in callback request"
400 that hides the real denial reason. Raise a 401 that surfaces the
IdP's error and description instead.
Ported from #26640 with conflicts resolved against current staging
* fix(guardrails): walk Responses-API text taxonomy in shared content helpers
Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently
drops all text on the /v1/responses path. AIM turns it into a loud 422 (
{"error":"No messages in the request"}); every other guardrail (Lakera v2,
Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret
detection) scans an empty payload and lets the request through unscanned.
Three defects, all in _content_utils.py:
1. _iter_text_parts_in_content recognised only part.type == "text", but the
Responses API uses input_text (request) and output_text (assistant).
2. _coerce_input_to_messages gated on "every item has a role key"; any
Responses input list containing a function_call or function_call_output
item failed the check and was wrapped as one opaque blob.
3. build_inspection_messages forwarded any role through, including a bare
tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze
reject with a schema error.
Fix walks the actual Responses item taxonomy (message, function_call,
function_call_output, bare content parts and strings), recognises
{text, input_text, output_text} everywhere, and coerces any role outside
{system, user, assistant} to user in the outbound inspection payload.
* style: ruff-format changed guardrail files
* test(guardrails): cover function_call_output string form; drop em-dash in new docstring
* fix(guardrails): map function_call_output straight to user role
Avoids ever materialising a schema-invalid bare tool message. The
downstream role-safety coercion in build_inspection_messages still
guards genuinely caller-supplied non-standard roles (developer,
function, custom values); add a regression test covering that path
so the coercion has real coverage after this simplification.
* test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages
* docs(test): soften AIM-specific claims in LIT-4294 test docstrings
Ryan's review flagged that several test docstrings assert AIM's
/fw/v1/analyze validates + rejects specific schema violations. That
behavior is customer-reported in the LIT-4294 writeup, not directly
verified by us. Rephrase to attribute the AIM 422 to the customer's
writeup and describe the underlying constraint as the OpenAI chat
schema; any downstream API that validates against that schema rejects
the same shape.
* refactor(guardrails): move unsupported-role coercion into AIM only
The generic coercion in build_inspection_messages collapsed any role
outside {system, user, assistant} to user for every caller of the
helper. Combined with the pre-existing apply_redacted_messages_back
write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400
on chat-completions tool-message masking into a silent semantic
corruption of the outbound request (role tool with tool_call_id got
rewritten to bare role user, dropping the assistant + tool_calls
sibling).
AIM specifically requires the coercion because its /fw/v1/analyze
validates the payload against the OpenAI chat schema; other guardrails
either do not validate roles or do their own reconstruction. Move the
coercion to AimGuardrail._build_aim_inspection_messages so the shared
helper keeps caller roles intact and no new cross-guardrail role
corruption is introduced. The pre-existing apply_redacted_messages_back
structural flatten remains as separate follow-up work.
function_call_output items still synthesise role user in the shared
helper because they have no natural role field, which is a different
concern from coercing a caller-supplied role.
* refactor(guardrails): preserve role fidelity in shared _content_utils
Shared inspection helpers should extract text and preserve semantic
role signals; role coercion for third-party schema safety stays inside
the guardrail that needs it (AIM).
Three shared-helper changes:
- Bare content-part dicts (input_text/output_text) with an explicit role
keep it; only role-less parts default to user.
- Responses message items already had their role preserved; the
behavior is now covered by an explicit test.
- function_call_output items default to role tool (semantic equivalent
of the chat-completions tool message shape) instead of role user, so
Responses and chat completions produce symmetric inspection payloads.
A caller-supplied role on the item is still preserved.
AIM's schema-safe coercion in _build_aim_inspection_messages already
handles the resulting role tool: it collapses to user before the POST
to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the
bare tool message (no tool_call_id can survive the flatten). Added a
regression test in test_aim.py covering that path.
An org admin updating a team budget from the Hub UI was rejected with 401, because the route gate only recognizes an org admin when the request body carries organization_id while the UI sends team_id. For /team/update, resolve the target team's organization_id from team_id before the gate runs, so an org admin of the team's own org clears the org-scoped branch without the client passing organization_id. Team admins and cross-org admins stay denied at the gate, and callers that already pass organization_id are unaffected, so the existing /team/update authorization matrix is unchanged
The connect gate resolved x-mcp-{alias}-authorization by matching the raw
lowercased alias/server_name/name only, but dashboard clients send
x-mcp-{sanitize_mcp_alias_for_header(alias)}-authorization, and egress resolves
those through lookup_mcp_server_auth_in_headers, which also tries the sanitized
alias. So a per-server token bound with a sanitized alias (e.g. alias 'pt-server'
arriving as header key 'pt_server') was forwarded at egress but still triggered a
preemptive 401 at connect. _client_has_per_server_auth_header now resolves through
the same lookup_mcp_server_auth_in_headers egress uses, so connect and egress
agree on which header names match.
The listing fan-out withholds the request-wide Authorization from a
true_passthrough / oauth_delegate server when another server in scope also
consumes it, so one bearer is not replayed across upstreams. The later
server.extra_headers copy loop did not honor that decision: a server listing
Authorization in extra_headers would re-copy the withheld bearer from
raw_headers. The withhold decision is now computed once and applied to both
the forwarding branch and the extra_headers loop.
The preemptive 401 for true_passthrough and oauth_delegate only inspected the
request-wide Authorization, so a caller who bound the upstream token via the
per-server x-mcp-{alias}-authorization header (the required shape in a
multi-server aggregate, where the request-wide Authorization is withheld) was
spuriously 401'd at initialize even though egress already honors that header.
The gate now recognizes the per-server header for both modes via a shared
helper, mode-correctly: true_passthrough treats any Authorization or the
per-server header as the upstream token, oauth_delegate keeps requiring a
distinct x-litellm-api-key so a lone Authorization consumed for admission is
never mistaken for an upstream token. The preemptive raise is also gated to
single-server scopes so a multi-server aggregate degrades gracefully instead
of one missing token 401-ing the whole connect.
For the client-forwarded token modes the gateway relays the caller's
Authorization to the upstream, so a request carrying more than one
Authorization header would make which token is forwarded ambiguous (the
ASGI header list collapses to last-wins) and could diverge from what
admission inspected. Multiple Authorization headers is malformed for
bearer auth anyway (RFC 9110: not a comma-combinable field), so the
ingress header converter now fails closed with a 400 instead of silently
keeping one. Applies to every MCP request, not just passthrough.
* fix(guardrails): forward grayswan scan id header
* test(guardrails): cover grayswan scan id forwarding
* fix(guardrails): prevent overwriting existing metadata headers when extracting scan id
* test(guardrails): cover header merging logic
* chore(guardrails): fix formatting
* test(guardrails): enforce case preservation
* chore(guardrails): corrected grayswan type annotations
* fix(guardrails): sanitized grayswan header metadata
* test(guardrails): covered grayswan logging headers
* fix(guardrails): guard grayswan header lookup against None and drop dead comment
- Fall back to {} when proxy_server_request is explicitly None so
request_data.get(...).get('headers') never raises AttributeError.
- Remove the commented-out user_api_key_auth pop; it was inert and
greptile called it out as ambiguous.
---------
Co-authored-by: Theodore Drzewinski <93957989+tediferJones@users.noreply.github.com>
On re-auth against a server with a persisted DCR client, register_client_with_server
short-circuits and returns a placeholder client_secret ("dummy") that the browser
echoes back to /token. exchange_token_with_server overrode the caller's client_id
with the persisted one but still fell back to the caller's secret when the server
had none stored, so a persisted public PKCE client (which has no secret) was paired
with the literal string "dummy" and the IdP rejected the exchange with 401 on every
re-authorization; the proxy surfaced that as a 500. First connects and brand-new
servers worked because a real DCR registration ran and no placeholder existed.
Resolve the secret from the server whenever the server's client_id wins, so a
secretless public client sends no client_secret at all
key_alias can become the secret name used by external secret manager
integrations (HashiCorp Vault, CyberArk Conjur) when store_virtual_keys is
enabled. Add raise_if_unsafe_secret_name, a shared validation check applied
unconditionally before a secret name reaches either integration or the
/key/generate, /key/update, and /key/regenerate API boundary, independent
of the existing enable_key_alias_format_validation opt-in flag.
Also hardens the Vault URL builder to percent-encode reserved characters
in secret_name (preserving "/" and "@"), and switches the Conjur policy
body to a real YAML serializer instead of raw string interpolation.
An MCP tool call that completes with CallToolResult.isError=true correctly
returns HTTP 200 per the MCP spec, but the shared post-call logging helper
always fired async_success_handler, so the standard logging payload carried
status=success and OTel (whose _parse_error only marks ERROR on
status=failure) showed green spans for failed tools.
The helper now checks the result after async_post_mcp_tool_call_hook runs
(guardrails may flip isError there) and routes error results to the failure
path: success gates are consumed so the @client wrapper cannot enqueue a
success log, failure_handler and async_failure_handler fire with a new
MCPToolResultError carrying the tool's first text content, and
post_call_failure_hook records the failure the same way raised exceptions
already do. Raised exceptions never reach the helper, so no double failure
logging. HTTP wire behavior is unchanged
Resolves LIT-4081
* fix(mcp): drop the cached per-user OAuth token when the credential row changes
The v2 authorization_code chain Cached(Refreshing(V2PerUserTokenStore)) caches a positive token
until its expires_at (or 300s without one), and CachedOAuthTokenStore.invalidate had no callers,
so a re-authorization or revocation wrote the DB while egress kept serving the replaced token
from the in-process cache until its TTL. LazyPerUserOAuthTokenStore now exposes invalidate,
MCPServerManager threads it to the write side, and the three credential write sites (the OAuth
callback, the Tools-tab persist endpoint, and the revoke endpoint) drop the cache entry after
the row changes. The v2 refresher's own persist stays untouched; RefreshingTokenStore already
feeds the rotated token back into the cache in the same fetch
* test(mcp): pin cache invalidation on the revoke already-gone branch
Greptile's review flagged that only the happy-path delete asserted the invalidate; a refactor
moving the call inside the try block would silently skip the cache drop when the row was
already deleted by a concurrent request while the cache still held the revoked token. The new
test fails on exactly that mutation
* test(mcp): cover invalidate on the redis-backed lazy store path
Codecov flagged the redis fast path of LazyPerUserOAuthTokenStore.invalidate as unexercised;
the existing invalidate tests only ran the no-redis chain. The new test builds the redis chain
via a fetch and asserts a subsequent invalidate reaches the same store instance without a
rebuild
Add a tag_rpm_limit field to virtual keys so each request tag gets its own independent RPM counter on the v3 rate limiter. A key configured with per-tag limits tracks each tag/group separately, and requests whose tag has no configured limit fall back to the key-level limit. Includes the dashboard UI to manage per-tag limits on key create and edit.
Resolves LIT-3147
* feat(ui): expose MCP max_concurrent_requests in server create and edit forms
The proxy has enforced a per-server outbound tool-call concurrency cap
(max_concurrent_requests) across every MCP egress path since #31641, and the
management API has accepted the field on create and update all along, but the
dashboard offered no way to set it. Add an optional Max Concurrent Requests
input to the MCP server create and edit forms; it applies to every auth type
and transport, so it renders unconditionally rather than gated on auth mode.
Clearing the field on edit sends null so the stored limit is unset.
Also rebuild the per-server semaphore when the configured limit changes.
Previously the semaphore was created once per server_id and never resized, so
an edited limit only took effect after a proxy restart even though the new
value was persisted and reloaded into the registry.
* feat(ui): mark MCP max concurrent requests field label as optional
* test(ui): stop OBO create-form tests from timing out on CI
The token-exchange payload test and the Entra scope-required test filled five
text fields with user.type, which dispatches a full keystroke sequence per
character; every input event runs the antd form onValuesChange handler and
re-renders the whole CreateMCPServer tree, roughly 120 renders per test. As
the form grew the two tests reached 8s and 18s locally, which crosses the 30s
vitest timeout on slower CI containers; ui_unit_tests failed twice this way.
Switch the plain text fields to fireEvent.change (one input event per field),
matching the existing stdio test pattern. Both tests assert form output, not
keystroke behavior, and now run in about 3s each.
