chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)

* feat(router): add LLM-based classifier option to complexity router (#32169)

* feat(router): add LLM-based classifier option to complexity router

Adds classifier_type: "heuristic" | "llm" to complexity_router_config.
When set to "llm", the router calls a configured model (e.g. a small
model like haiku) via structured output to pick the complexity tier,
falling back to the existing regex/keyword scorer on any error, empty
response, or unparseable output.

* feat(ui): add classifier_type option to complexity router UI, fix edit flow

Adds an "Advanced: Classification Method" section to ComplexityRouterConfig
with a heuristic/LLM toggle, revealing a classifier model picker and timeout
when LLM is selected.

Also fixes the auto router edit modal, which never rendered the complexity
router UI at all (it only handled the semantic router), and the "Edit Auto
Router" button visibility check, which was gated on auto_router_config and
never matched complexity router deployments.

* fix(router): attribute classifier calls to caller, raise default timeout

Forwards the original request's litellm_metadata into the classifier's
acompletion call. Without it, the proxy's cost-tracking gate sees no
user_api_key/team_id/user_id and silently drops spend logging and budget
accounting for every classifier call, letting an authenticated user rack
up unaccounted provider spend via repeated requests.

Also raises the default classifier timeout from 400ms to 3000ms (400ms
undershoots real LLM latency and would silently degrade to the heuristic
scorer on most requests) and corrects the module/class docstrings, which
still claimed zero external API calls after the llm classifier path was
added.

* fix(ci): resolve ruff strict-budget and frontend-lint failures

- Use PEP 585 generics (dict/tuple/list) in the new aclassify/_classify_with_llm
  signatures instead of typing.Dict/Tuple/List, and suppress BLE001 on the
  intentionally broad except in aclassify's fallback path with a reason.
- Fix prettier formatting in ComplexityRouterConfig.tsx.
- Regenerate eslint-metrics.json (was stale after the classifier UI changes).

* fix(ci): regenerate stale eslint-metrics.json

* fix(router): strip parent budget reservation from classifier metadata

The classifier's internal acompletion call previously forwarded the
parent request's full litellm_metadata, including its budget
reservation (user_api_key_budget_reservation / user_api_key_auth).
That reservation belongs to the routed completion the classifier is
deciding on, not to the classifier call itself, so it's now stripped
while key/team attribution fields are still forwarded for spend
logging.

* fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map (#32840)

* fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map

* test: use apac regional profile for cost-map fallback test since jp now has an entry

* fix(responses): preserve reasoning_tokens through chat->responses usage translation (#32837)

* fix(responses): preserve reasoning_tokens through chat->responses usage translation

Remove the unconditional else-branch that wrote reasoning_tokens=0 whenever
completion_tokens_details.reasoning_tokens was None or absent. Also change
OutputTokensDetails.reasoning_tokens from int=0 to Optional[int]=None so that
re-instantiation without explicit reasoning_tokens no longer silently zeroes out
the field, and remove the same hardcoded zero from the mock_responses_api_response
initializer.

* test(responses): update assertions to match Optional[int] reasoning_tokens default

* fix(responses): preserve explicit reasoning_tokens=0 in usage translation

Align the reasoning_tokens guard with the is-not-None guards used for
text_tokens and image_tokens: a provider-reported zero passes through
while an absent value stays omitted.

---------

Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>

* fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831)

* fix(bedrock): gate in-place system role messages on model support for Claude Invoke

* feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule

* fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param to accept dict (#32835)

* fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param

- BaseResponsesAPIStreamingIterator._maybe_raise_for_error_event inspects each
  chunk and raises litellm.APIError for type=error and type=response.failed events
  so callers see an exception instead of a benign stream chunk
- rate_limit* codes map to 429; client error codes (invalid_request_error,
  context_length_exceeded, etc.) map to 400; all other codes default to 500;
  raw integer codes are never used as-is as HTTP status codes
- ErrorEventError.param widened from Optional[str] to Optional[Union[str, Dict]]
  to prevent Pydantic ValidationError on dict-typed param payloads silently
  dropping error events before any type inspection

* test(responses-api): add streaming iterator error event tests to CI-covered path

* test(responses-api): cover response.failed, dict-error, null-error, and sync iterator paths

* test(responses-api): set completion_start_time on mock logging objects for internal staging _process_chunk

* fix(responses-api): map insufficient_quota to 429, derive failed-response log status from error code, and record failed-stream usage for spend accounting

insufficient_quota moves out of the 400 bucket; OpenAI returns HTTP 429 for it and the non-streaming exception mapping treats 429 as RateLimitError, so the in-stream mapping now agrees

_handle_logging_failed_response previously hardcoded APIError(status_code=500), so a rate-limited response.failed was logged to integrations as 500 while the caller saw 429; it now shares the same error-code-to-status mapping via _error_event_fields and _status_code_for_error_code

usage carried on a response.failed event is now stashed as combined_usage_object with its computed cost on the logging object before failure handlers run, reusing the mid-stream-interruption spend recovery path (_failure_handler_helper_fn, proxy post_call_failure_hook, _ProxyDBLogger), so failed streams count their billed tokens instead of logging zero cost

dedupe: TestMaybeRaiseForErrorEvent in tests/llm_responses_api_testing duplicated tests/test_litellm/responses/test_streaming_iterator_error_events.py, which is the canonical mirrored location and CI-covered via test-unit-responses-caching-types; the duplicate class is removed

* fix(responses-api): wrap retriable in-stream errors in MidStreamFallbackError and map error type field to status

Mirror chat streaming semantics from _handle_stream_fallback_error: 429 and
5xx in-stream error events now raise MidStreamFallbackError carrying the
mapped APIError so the router's FallbackResponsesStreamWrapper triggers
mid-stream fallback and cooldown; non-retriable 4xx still raise APIError
directly. Status mapping now reads both the OpenAI error type and code
fields, so type-classified client errors (e.g. invalid_request_error with
code invalid_prompt) map to 400 instead of falling through to 500.

* fix(responses-api): accumulate streamed output text so mid-stream fallback continues instead of restarting

MidStreamFallbackError was always raised with generated_content="", so the
router's stream_with_fallbacks treated every mid-stream error as pre-first-chunk
and retried with the original input, streaming duplicated content to clients
that had already received partial output. The iterators now accumulate
response.output_text.delta text (mirroring chat's response_uptil_now) and pass
it as generated_content, letting the router build a continuation input via
_build_responses_continuation_input.

* test(responses-api): pin in-stream token limit error to raised APIError

---------

Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>

* fix(prometheus): skip budget metric DB lookups when gauges are NoOpMetric (#32834)

adds a top-level guard in _increment_remaining_budget_metrics that returns early
when all four budget gauges are NoOpMetric (excluded from prometheus_metrics_config),
and per-entity guards in each _set_*_budget_metrics_after_api_request helper for
partial disabling. eliminates four async DB/cache round-trips per successful LLM
request when budget metrics are disabled.

Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>

* fix(anthropic): strip @version suffix in _model_map_lookup_candidates (#32833)

vertex_ai/claude-opus-4-8@default (and sibling @default models) were
misclassified as non-adaptive because _model_map_lookup_candidates only
stripped provider prefixes but never the @<suffix> portion. The lookup
produced candidates like ["vertex_ai/claude-opus-4-8@default",
"claude-opus-4-8@default"], neither of which exists in model_cost, so
_is_adaptive_thinking_model returned False. LiteLLM then sent
thinking.type=enabled to a @default Vertex AI endpoint that requires
thinking.type=adaptive, resulting in a 400.

_strip_version_suffix now removes @<suffix> from each candidate,
adding the bare model name (e.g. "claude-opus-4-8") to the lookup
chain. Also adds supports_adaptive_thinking: true to the three
@default model_cost entries that were missing it as belt-and-suspenders.

Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>

* fix(datadog): split log batches proactively under intake payload limits (#32860)

* fix(datadog): split log batches proactively under intake payload limits

* fix(datadog): size intake chunks with exact wire serialization

* fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support (#32867)

* fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support

AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface
(thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed
model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5)
get the effort translated to a legacy thinking budget_tokens. Models with no
reasoning support have thinking/effort dropped under drop_params. And because
adaptive thinking carries no budget while the legacy form must satisfy Anthropic's
max_tokens > budget_tokens rule, the translated budget is capped below max_tokens,
dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass
through untouched.

This matters because clients like Claude Code speak native Anthropic /v1/messages
and send the adaptive interface unconditionally, regardless of the routed model.
The native passthrough previously only capability-gated the OpenAI-style
reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so
a pre-4.6 model rejected it with "This model does not support the effort parameter"
and the request failed. Claude Code already gets drop_params auto-set, so its
requests now succeed.

* test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests

Addresses review feedback on the max_tokens-too-small branch. Previously a
thinking-capable model whose max_tokens could not fit the minimum thinking budget
had thinking silently dropped regardless of drop_params, while a residual
output_config field in the same call still raised when drop_params was off. Gate
both consistently on drop_params: raise a clear error (naming max_tokens for the
undersized case) when drop_params is off, drop otherwise. Claude Code gets
drop_params auto-set, so it still succeeds.

Adds tests for the undersized-max_tokens raise, the residual output_config raise,
and the no-adaptive-interface passthrough on a non-adaptive model.

* fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts

The previous raise-when-not-drop_params behavior broke existing bedrock and vertex
messages tests: those providers already silently strip unsupported output_config
for pre-4.6 models (issue #22797) with no drop_params required, and the shared
parent transform raising pre-empted that. It also conflicted with the goal of
keeping requests working rather than failing them.

Make the reshape silent: translate effort to legacy thinking for thinking-capable
models, drop thinking for non-reasoning models, and remove only the consumed effort
key from output_config, leaving any residual (e.g. format) for provider subclasses
(bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the
review note about inconsistent drop_params handling by making every path uniform.

Updates the tests to assert the silent behavior and residual output_config
preservation.

* fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5)

Greptile caught a real bug: the early-return guard treated supports_output_config
as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises
supports_output_config (it accepts output_config.effort) but is not adaptive, so it
rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this
model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking
block raw, reproducing the exact failure the fix is meant to prevent.

thinking:{type:adaptive} and output_config.effort are independent capabilities.
Only early-return for adaptive-thinking models. For a model that supports
output_config.effort but is not adaptive, keep the native effort and drop only the
unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code
payload now returns 200 instead of 400.

Adds regression tests for Opus 4.5 with and without adaptive thinking.

* fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models

Claude Opus 4.5 advertises supports_output_config but not adaptive thinking,
so the early-return guard forwarded thinking.type=adaptive raw and Anthropic
rejected it. The guard now only skips true adaptive models; effort-only
requests on effort-capable models still pass through untouched. The
_map_reasoning_effort call is wrapped to surface unrecognized effort values
as a clean 400, matching _translate_reasoning_effort_to_anthropic

* fix(anthropic): fall back to legacy thinking when effort level unsupported

Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code
defaults to xhigh on newer models, so preserving that level raw gets rejected
by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model
and fall through to the budget translation for unsupported levels

* fix(anthropic): keep effort-only requests untouched for provider normalization

The xhigh fall-through consumed effort-only requests on effort-capable
models, breaking bedrock invoke's own normalization which clamps xhigh to
the model's ceiling after the base transform runs
(test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict
the fall-through to requests that carry adaptive thinking; effort-only
requests pass through so provider subclasses keep owning level clamping

---------

Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>

* test(models): assert capability fields on regional Azure gpt-5.6 entries (#32875)

* feat(auto_router): keyword tier overrides and semantic keyword matching for the complexity router

Add deterministic keyword-to-tier overrides and optional embedding-based
(semantic) keyword matching to the complexity router, and surface both in the
Add Auto Router UI behind a Router Type selector: "Auto-Router v2 [Recommended]"
(complexity tiers + keyword overrides + semantic matching, the default) and
"Semantic Router [to be deprecated]" (the existing utterance-based router,
unchanged). Keyword-to-tier overrides resolve to the highest tier matched
rather than the first keyword matched, so match order no longer affects the
routing decision.

Backend:
- config: KeywordTierRule model plus keyword_tier_rules, semantic_keyword_matching,
  embedding_model, and match_threshold on ComplexityRouterConfig, with a validator
  requiring an embedding model and rules when semantic matching is on
- complexity_router: evaluate keyword rules before scoring; lexical matches escalate
  to the most-severe matched tier (order-independent), and semantic mode reuses
  LiteLLMRouterEncoder + SemanticRouter to match paraphrases by cosine similarity,
  falling back to the scorer when nothing matches
- model management: clear complexity_routers on cache reload so config edits take effect

Frontend:
- Add Auto Router tab restores the Router Type radio (Auto-Router v2 recommended
  by default, Semantic Router still available) and sends keyword_tier_rules plus
  the semantic settings on the recommended path, instead of flattening keywords
  into custom_technical_keywords
- client-side guard blocks submit when semantic matching is enabled without an
  embedding model or without any keyword tier rules, mirroring the backend validator
- moved the "How Classification Works" explainer below Custom Technical Keywords
  and above Keyword Tier Overrides
- remove the Test Connection action from the recommended flow, which can't build a
  valid pre-save payload for a router (leaves a TODO for a JSON preview / config
  test follow-up)

Tests cover lexical escalation, semantic matching via the real library with injected
embeddings, the semantic config guard, config validation, the reload-clear
regression, and the frontend payload builder

* fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882)

Exact cost-map hits resolve before fallback-generalization rules, so the
mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the
bedrock-anthropic-claude-mid-conversation-system rule and hoisted
mid-conversation system messages, invalidating the prompt cache.

* feat(team): let org admins reach PATCH /team/{team_id} like POST /team/update

Wire the coarse route gate so PATCH /team/{team_id} is reachable by exactly the roles that can call POST /team/update: proxy admins, org admins of the team's own organization, and JWT admins. Regular internal users and view-only proxy admins stay blocked, matching the existing endpoint

Because the team id lives in the path rather than the body, the org-context resolver now also reads it from path_team_id for the bare /team/{team_id} route, so an org admin's organization is resolved and injected the same way it already is for POST /team/update. /team/{team_id} is added to management_routes rather than the role-agnostic self_managed_routes; the latter would have opened POST /team/new to any authenticated user through the shared /team/{team_id} path pattern

* feat(ui): root the gateway breadcrumb in the AI Gateway selector

The AI Gateway select (ViewSwitcher) now sits at the root of the DashboardHeader breadcrumb instead of on the right, so the top bar reads [AI Gateway select] > Page to match the redesign. It keeps the same dropdown, including the Chat / Chat UI options.

When no plugins are registered and Chat UI is disabled there is nothing to switch between, so the breadcrumb falls back to the static section crumb rather than rendering a dangling leading separator

* fix(complexity_router): build semantic route index once under concurrent cold-start

Concurrent first requests each hit asyncio.to_thread to build the SemanticRouter
index, firing duplicate embedding calls for the static route utterances. Guard the
lazy build with a per-router asyncio.Lock (double-checked) so the index is
constructed exactly once regardless of how many callers race in cold.

Adds a regression test asserting ten simultaneous cold-start requests build the
index the same number of times as a single request, and reworks the fake embedding
router to count builds by how often a route utterance is embedded (robust to which
embedding path the library uses) while still recording sync-call thread ids for the
off-event-loop assertion.

* feat(ui): always show the gateway selector with a discoverable Chat entry

The AI Gateway selector now always renders at the breadcrumb root, even with no plugins and Chat UI disabled, so the Chat feature stays discoverable. The Chat entry is always listed: clickable when enabled, and disabled with an "Admins can enable in Settings" hint when it is off.

Since the selector is now unconditional, the useViewSwitcherVisible hook and the section-crumb fallback added in the previous commit are removed

* fix(proxy): guard delete_model router eviction on auto_router/ prefix

delete_model popped the auto_routers/complexity_routers registries by the deleted
deployment's model_name without checking it was actually an auto_router/* deployment.
Deleting a regular DB model that merely shares a name with a config-defined router
therefore evicted that router, which add_deployment never restores, leaving it
unroutable until a proxy restart. This is the same cross-tenant DoS clear_cache was
hardened against; mirror its auto_router/ prefix guard here.

Extracts _deployment_name_and_model to read model_name and litellm_params.model from
the deployment (delete_deployment returns the raw model_list dict at runtime despite
its Deployment annotation), and adds a regression test asserting a same-named config
router survives deletion of an unrelated regular model.

* refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds

* feat(fallback-generalizations): widen adaptive-thinking gate to any claude family at major 5+

* fix(fallback-generalizations): tolerate legacy remote rule schema and keep register_model cache-pricing inheritance

* fix(fallback-generalizations): let exact cost-map entries beat capability rules across lookup-candidate ladders

* fix(team): bound json merge patch recursion depth

apply_json_merge_patch recurses into nested objects, which the repo's recursive_detector code-quality check flags because unbounded recursion over caller-supplied JSON has caused CPU/stack issues before. Cap the recursion at a depth far above any realistic team-metadata shape and reject deeper patches with a ValueError so a pathologically nested body fails closed instead of overflowing the stack, then register the function in the detector's ignore list alongside the other depth-bounded JSON walkers

* fix(auth): tolerate request objects without path_params in common_checks

The PATCH /team/{team_id} org-context wiring reads request.path_params to
resolve the team id from the path. A real Starlette Request always exposes
path_params, but common_checks is exercised with lightweight request doubles
that don't, which raised AttributeError. Read it defensively so a missing or
null path_params falls back to no path team id, matching the "not a bare team
route" outcome; real requests are unaffected

* fix(mcp): re-register DCR client when proxy origin no longer matches its registered redirect_uri

A dynamically registered (RFC 7591) OAuth client persisted onto the MCP server row is bound to the redirect_uri it was first registered with, but that binding was never recorded. After the proxy's public origin changed, every authorize paired the reused client with the new callback and the IdP rejected it permanently.

The DCR persist now records redirect_uris alongside the client identity. The admin register path treats a positive mismatch between the recording and the current callback as stale and re-registers a replacement client; rows without a recording (pre-existing installs and admin-configured clients) are grandfathered so upgrades never re-mint client_ids or orphan refresh tokens. The persist also writes client_secret and token_endpoint_auth_method explicitly as None when absent so the credential blob merge cannot pair a re-registered public client with the previous client's secret. Public register routes and non-admin callers keep existing behavior.

Closes #32473

* fix(mcp): emit one operator warning per DCR re-registration event

The stale-redirect path logged three warnings for a single re-registration: the staleness probe plus the reuse skip in both register_client_with_server and the persist race guard. The reuse-skip message is a mechanical consequence of the probe's decision, so it now logs at debug; the actionable warning that names both bindings and the re-authentication impact is emitted once by _persisted_dcr_redirect_uri_is_stale

* ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI

* refactor(ui): use TanStack Pacer debounce for the team keys search

Replace lodash/debounce in TeamVirtualKeysTable with useDebouncedValue from
@tanstack/react-pacer, matching the sibling VirtualKeysTable and
PaginatedKeyAliasSelect which already debounce their key-alias search that way.
Pacer is already a dependency, so this drops the odd-one-out lodash usage and
keeps the search-debounce pattern consistent across the key tables.

* fix(fallback-generalizations): cover bare Claude majors in baseline and routing, require claude- prefix in adaptive gate

* fix(mcp): strip scheme default port from get_request_base_url netloc

* feat(ui): typed openapi-fetch foundation (fetchClient) + first typed caller (useCustomers) (#29884)

* feat(ui): add the typed openapi-fetch client (fetchClient) as the dashboard fetch foundation

Introduces fetchClient (openapi-fetch) bound to schema.d.ts, used inside ordinary TanStack Query hooks so path/query/body types come from the proxy's OpenAPI spec. A small runtime registry feeds the client the base URL and auth header name (registered by networking) and the session token (published by AuthContext), so call sites carry no token plumbing; auth-header injection and ApiError mapping live in openapi-fetch middleware reusing deriveErrorMessage/ApiError from client.ts, and non-2xx maps to a thrown ApiError so query functions just read .data.

The base URL default resolves from NEXT_PUBLIC_BASE_URL so a request still targets the right origin if it fires before networking registers its getter. AuthContext clears accessToken alongside the token on logout so no query fires unauthenticated after the session ends.

Foundation only; callers migrate one at a time, each fully typed, in follow-up changes.

* feat(ui): migrate useCustomers to the typed fetchClient

Converts useCustomers from allEndUsersCall to fetchClient.GET("/customer/list"); the response is typed as LiteLLM_EndUserTable[] from the schema, so the hand-written Customer/CustomersResponse types are deleted. They were also inaccurate (allowed_model_region was string but is "eu"|"us", and a budget_id the table has no field for). No cast; the schema type flows to the one consumer. First caller on the new pattern.

* fix(ui): route typed-client errors through the session-expiry handler

The typed fetchClient middleware threw ApiError without invoking the
handleError side effect that the legacy createApiClient wires via
onError, so a migrated caller hitting an expired key no longer triggered
the auto-logout. Add an error-handler seam to runtime.ts, register
handleError from networking.tsx alongside the base-url/header getters,
and call it in the middleware before throwing so both clients behave the
same. Regression test asserts the handler fires with the derived message
on non-2xx and stays silent on success

* fix(ui): point the customers EndUser type at CustomerResponse

The /customer/list response model was renamed to CustomerResponse on
staging; the merged branch still aliased EndUser to LiteLLM_EndUserTable,
so the exported type and its test mock had drifted from what the schema
actually returns. CustomerResponse is also the accurate shape (it types
allowed_model_region as 'eu' | 'us' and carries budget_id)

* chore(ui): refresh eslint-metrics baseline after staging merge

The recorded baseline predated the litellm_internal_staging merge, so its
no-explicit-any and no-large-inline-object-arg counts were higher than the
merged tree actually has. Regenerate via npm run lint:metrics so the gate
reflects current reality

* refactor(ui): source the typed client token from the session cookie, not AuthContext

The typed client read its bearer from a runtime value that AuthContext pushed
via setAuthToken, but migrated hooks gate enabled on useAuthorized, which
decodes the cookie directly. Two independent derivations of the same cookie with
different timing: on first load the query fires (useAuthorized sees the token)
before AuthContext's async effect publishes it, so the first request goes out
unauthenticated and only succeeds on a React Query retry.

Make the token a registered getter like the base-url and header-name getters,
reading the same cookie useAuthorized decodes, so the client's token and the
gate can't diverge. Revert the AuthContext changes entirely; nothing is pushed
from React state anymore.

* test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857)

* test(e2e): cover Langfuse logging.yaml P0 logs_spend cells

Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat
traffic and assert calculatedTotalCost matches StandardLogging response_cost
and proxy spend. Also assert tool calls and applied guardrails land on the
trace. Missing env or proxy is a hard failure, never a skip

* test(e2e): use langfuse_otel callback for Langfuse spend coverage

Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to
Langfuse) instead of the classic langfuse SDK. Match generations named
litellm_request by prompt marker and user_api_key_alias

* test(e2e): require Langfuse spend assert; drop AGENTS.md

Guardrail path no longer soft-gates logs_spend. Non-stream responses must
return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md

* test(e2e): fail when Langfuse spend is missing on guardrail path

Always run logs_spend assertions for tool_permission; require positive
x-litellm-response-cost on non-stream and positive /spend/logs spend

* test(e2e): do not fall back to unmatched spend log rows

poll_proxy_spend_for_key returns None when response_id or positive-spend
filters match nothing, instead of silently using rows[0]

* fix(complexity_router): use max aggregation for semantic keyword route scoring

SemanticRouter defaults to mean aggregation across a route's utterances. Since
each tier's route holds one utterance per configured keyword, a real semantic
match on one keyword was averaged together with the tier's other, unrelated
keywords and dragged below match_threshold — e.g. a MEDIUM tier with keywords
[beep, boop, new york] never fired for a genuine "new york" paraphrase, because
mean(sim_to_beep, sim_to_boop, sim_to_new_york) landed well under the threshold
even though sim_to_new_york alone cleared it. Pass aggregation="max" so a tier
matches if the query is close enough to any one of its keywords, not the
average of all of them.

Verified against live Voyage embeddings: raw cosine similarity for "new york"
vs a paraphrase was 0.54 (above a 0.5 threshold), but the route scored 0.28
under mean aggregation and never matched; max aggregation fixes it.

Adds a regression test with a tier holding one matching and two unrelated
keywords, asserting the tier still fires; fails without aggregation="max".

* refactor(auth): resolve PATCH team org-context from the route template

Replace the request.path_params read (and its defensive getattr guard) with
the route template. A real Starlette request always exposes path_params, but
common_checks runs on lightweight request doubles that don't, so reading it
directly forced a getattr workaround that only existed to tolerate those
doubles.

Instead, match the route template (/team/{team_id}) to identify the RESTful
update route and take the team id from the last path segment. This drops the
path_params dependency entirely, and because the template distinguishes the
PATCH route from its single-segment siblings (/team/new, /team/list, ...), it
also avoids a spurious team lookup those routes would otherwise trigger if we
matched the resolved path shape alone.

* chore(ui): remove eslint-metrics.json lint-count snapshot

The eslint-metrics.json snapshot duplicated the violation counts already
enforced by eslint-budgets.json. Keeping it current added a CI drift check,
a pre-commit regenerate-and-flag step, and a standalone npm run lint:metrics
script, none of which caught anything the budget gate did not, yet all of
which failed noisily whenever the snapshot went stale. This drops the file
and that machinery while leaving eslint-budgets.json as the actual ratchet
gate

* fix(complexity_router): preserve user_api_key_auth in sub-call metadata

Removing user_api_key_auth entirely from classifier/embedding sub-call
metadata (as _BUDGET_RESERVATION_METADATA_KEYS previously did) prevented
_filter_deployments_by_model_access_groups from scoping those sub-calls to
the caller's authorized access groups. An access-group-scoped caller could
therefore reach embedding/classifier deployments outside their group.

Only strip user_api_key_budget_reservation, which is the actual budget-
reservation state that must not reach sub-calls. user_api_key_auth is now
kept so access-group filtering works correctly for both the embedding path
and the LLM classifier path.

* test(e2e): drop vertex from pipecat tool smoke (#32925)

Exclude vertex_ai from pipecat tool smoke; raw-ws tool_call_round_trip
remains the Vertex source of truth. Also remove the Playwright key models
dropdown suite so stage is not blocked by that UI harness

* fix(complexity_router): sanitize budget reservation inside forwarded user_api_key_auth

* fix(complexity_router): review hardening - blank keywords, router registry eviction, edit-modal controls

- config: KeywordTierRule now strips and drops blank/whitespace keywords (a stray
  "" makes _keyword_matches match every prompt, silently forcing that tier for all
  traffic); still requires at least one real keyword to remain
- frontend build_complexity_router_config: trim keywords and drop rules left empty so
  an unfilled "Add keyword rule" row no longer ships a rule the backend rejects with a
  400 in the heuristic (non-semantic) flow, where the client-side semantic guard doesn't run
- proxy clear_cache / delete_model: the auto_router/ prefix also covers quality_router/
  and adaptive_router/, so pop the model_name from all four router registries (no-op
  where absent) instead of only auto/complexity; otherwise a DB quality_router's stale
  entry made reload raise "already exists" and abort, and adaptive left a leak
- frontend ComplexityRouterConfig: only render the Keyword Tier Overrides and Semantic
  keyword matching sections when their change handlers are provided, so the edit-auto-
  router modal (which omits them) no longer shows interactive-but-dead controls

* fix(anthropic): thread real provider through capability probes instead of pinning anthropic

* docs(anthropic): note the two provider params' roles in _map_reasoning_effort

* fix(anthropic): override custom_llm_provider in provider config subclasses so capability probes use the right namespace

* feat(mcp): admit dcr_bridge oauth_delegate clients via a single envelope bearer

* fix(mcp): admit dcr_bridge envelopes under the live key, not a frozen identity

The bridge envelope sealed only user_id/server_id, and admission fabricated a
UserAPIKeyAuth(user_id=...) with no object_permission, team_id, org_id, or key
identity. Downstream MCP permission checks read the missing restrictions as
unrestricted, so a caller holding a valid envelope for a restricted key could
reach tools and servers that key was never granted, and a revoked key kept
working until the envelope expired.

Bind the hashed authorizing key into the envelope identity and reload the live
UserAPIKeyAuth by it at admission via get_key_object, failing closed with a 401
when the key is missing, blocked, or expired. Authorization is resolved fresh
per request instead of frozen at mint time, so current key/team/org and tool
restrictions plus revocation are enforced.

* fix(mcp): enforce team block and alias-priority token injection on bridge admission

Two follow-ups on the envelope admission arm flagged in review.

Team revocation bypass: _reload_admitted_key checked only the key's own
blocked/expires, so blocking a key's team left every envelope minted under it
live until expiry. Reload the team and reject a blocked team, mirroring
common_checks, so a team block revokes its envelopes immediately.

Caller-overridable upstream token: egress resolves the per-server auth header
alias-first, but injection keyed under server_name, so for a server with a
distinct alias a caller-forwarded x-mcp-{alias}-authorization sat at the
higher-priority slot and paired the admitted identity with an attacker's
upstream credential. Inject under alias-first so the sealed token owns the slot
egress resolves.

* fix(mcp): route bridge admission through the centralized policy gate and mirror the SCIM owner check

* fix(mcp): import assert_never from typing_extensions for Python 3.10

* fix(mcp): surface real status from bridge admission policy gate instead of flattening to 401

Over-budget rendered 401 (should be 429), model-access and other typed
failures collapsed to 401, and a transient DB outage was masked as an auth
error. Mirror UserAPIKeyAuthExceptionHandler: budget maps to 429, a sub-check's
own HTTPException/ProxyException keeps its status, a DB outage is a retryable
503, and only a genuinely unresolvable failure stays the fail-closed 401.

* fix(rate-limit-v3): populate x-ratelimit-* remaining/limit values in standard_logging_object for streaming (LIT-4333) (#32711)

Streaming requests return from common_request_processing before
async_post_call_success_hook runs, so response._hidden_params.additional_headers
never gets the v3 x-ratelimit-{descriptor_key}-{remaining|limit}-{rate_limit_type}
entries. Prometheus / logging callbacks that read those values from
standard_logging_object.hidden_params.additional_headers then see nothing;
combined with the pre-existing gap that Prometheus reads from that same slot
(LIT-2577 / PR #28816), per-key remaining RPM/TPM cannot be monitored for
streaming traffic at all.

Fix in three parts:

- Stash the pre-call RateLimitResponse in the metadata channels the async
  success-logging callback inherits, alongside the existing top-level entry
  the non-streaming path reads.
- Add async_logging_hook to the v3 handler. It fires in a distinct earlier
  loop inside async_success_handler (all callbacks' async_logging_hook
  complete before any async_log_success_event starts), so mirroring the
  pre-call snapshot into standard_logging_object.hidden_params.additional_headers
  and response._hidden_params.additional_headers here guarantees every
  downstream success callback sees the values regardless of registration
  order. Non-streaming keeps the existing async_post_call_success_hook write
  and this hook re-populates the same values idempotently.
- Extract the shared `_merge_ratelimit_statuses_into_additional_headers`
  helper the non-streaming path already had inlined so both callsites emit
  the identical key shape.

* fix(proxy): skip None model_name in clear_cache router eviction set

* fix(mcp): map a DB outage during bridge key reload to a retryable 503

get_key_object's raw transport error propagated uncaught out of
_reload_admitted_key as an opaque 500; classify it via the shared
_raise_503_if_db_unavailable helper (also used by the live-policy gate) so a
database outage is a retryable 503, while a key-not-found ProxyException stays
the fail-closed 401.

* test: remove live OpenAI fine-tuning job-creation test blocked by platform wind-down (#32933)

OpenAI is winding down self-serve fine-tuning and the org can no longer
create fine-tuning jobs (403 training_not_available; the CI key surfaces
it as a 500 server_error), so test_create_fine_tune_jobs_async fails on
every batches_testing run since 2026-07-11 and reruns never clear it.
The request contract stays covered by the mocked create/list/cancel/
retrieve tests in the same file, and the deleted test's unique
standard_logging_object assertions now run inside
test_mock_openai_create_fine_tune_job.

* refactor(anthropic): consolidate the provider fallback into a _resolved_provider property

* feat: add lite auth print-token for Claude Code apiKeyHelper support (#32846)

* feat: add silent CLI token refresh for apiKeyHelper support

lite auth print-token prints a valid proxy credential for use as Claude
Code's apiKeyHelper, transparently refreshing it first if the cached JWT
is stale. This unblocks MDM-managed apiKeyHelper deployments (managed
via `lite auth print-token`) that need silent mid-session credential
rotation without restarting the client.

Refresh capability is backed by a virtual key minted with an empty model
list and cli_refresh metadata, kept strictly separate from the actual
(short-lived, real-model-scoped) call credential -- so a leak of the
credential that flows through every LLM request and subprocess env var
can't also self-renew. The refresh flow is single-use: /sso/cli/refresh
mints a fresh JWT + refresh token pair and blocks the presented refresh
token immediately, so a replay can't mint a second pair from it.

Server: /sso/cli/refresh (rotate) and /sso/cli/logout (revoke) endpoints.
lite login now also stores a refresh token; lite logout revokes it
server-side instead of only clearing the local file.

* fix: allow non-admin users to hit CLI refresh routes; resolve apiKeyHelper base_url from token.json

Found via a live end-to-end test against a real proxy + real Claude Code
session: /sso/cli/refresh and /sso/cli/logout were unreachable for any
non-proxy-admin caller, since Depends(user_api_key_auth) pulls in a
route-RBAC gate that 403s any route not on an explicit allowlist. That
made the feature unusable for actual end users, who authenticate as
internal_user. Add both routes to internal_user_routes; the handlers
already do their own fine-grained check (metadata.cli_refresh) same as
/key/block does today.

Also: `lite auth print-token` required an explicit --base-url/
LITELLM_PROXY_URL matching the stored token's origin, defaulting to
localhost:4000 otherwise. But apiKeyHelper is configured bare (no
flags), so this always mismatched a real deployment. Track whether
--base-url was explicitly passed (via click's ParameterSource) and, if
not, resolve the server from token.json directly instead of the CLI
default.

* test: mock refresh-token minting in test_cli_poll_key_tolerates_missing_user_row

Landed on litellm_internal_staging after this branch's refresh-key minting
change; needs the same mock as the other cli_poll_key tests since minting
now runs unconditionally whenever a JWT is generated.

* fix(ci): update test_cli_auth.py for refresh_token contract, regenerate schema.d.ts

_poll_for_authentication now always includes "refresh_token" in its
returned dict, and _handle_team_selection_during_polling returns a dict
instead of a bare JWT string -- test_cli_auth.py predates this branch's
refresh-token work and still asserted the old shapes.

schema.d.ts regenerated via `npm run gen:api` to pick up the new
/sso/cli/refresh and /sso/cli/logout routes (plus unrelated drift from
other PRs merged since it was last generated).

* fix(ci): apply CI's own schema.d.ts diff (enterprise routes I can't generate locally)

Local `npm run gen:api` only sees OSS routes -- this machine's
litellm_enterprise editable install points at a now-deleted temp
directory, so it silently drops enterprise-only routes from the spec.
Applied the exact diff CI's own generation produced instead of
re-running the generator locally.

* fix: close refresh-token race, fail closed on DB down, fix logout base_url

Addresses Greptile review findings on the CLI refresh-token PR:

- cli_refresh_token minted a new JWT + refresh token BEFORE blocking the
  presented one. Two concurrent requests bearing the same refresh token
  could both pass auth and both mint fresh pairs, yielding four live
  credentials from one consumed token. Now the presented token is
  consumed atomically first via update_many (only succeeding if it flips
  blocked from False/None to True); the loser gets count=0 and is
  rejected before anything is minted.
- When prisma_client is None, refresh silently returned a new JWT
  without ever being able to mark the presented token consumed, leaving
  it valid indefinitely. Now fails closed with a 500 instead.
- `lite logout` sent its revocation POST to ctx.obj["base_url"], which
  defaults to localhost:4000 when --base-url isn't passed -- the same
  bug print_token had before the base_url_explicit fix, just missed
  here. Now resolves the same way: trust the stored token's origin
  unless the caller explicitly overrode --base-url.

* fix(ci): satisfy ruff format and narrow token_data type in logout

* fix(security): never trust refresh-token metadata for authorization

Addresses a real privilege-escalation path Veria flagged: cli_refresh_token
read team_id, team_alias, and max_budget straight off the presented
token's own metadata and used them to authorize the new JWT. Since any
authenticated user can self-mint a virtual key with arbitrary metadata
via the ordinary /key/generate endpoint, a self-forged key with
{"cli_refresh": true, "team_id": "<any-team>", "max_budget": 999999999}
would sail through _require_cli_refresh_token's only check
(metadata.cli_refresh == True) and get a JWT scoped to a team the
caller never belonged to, with a budget it never had -- full
cross-team / budget bypass, and a removed team member could keep
refreshing team-scoped sessions indefinitely.

Metadata's team_id is now treated as an untrusted UX hint only: honored
solely if the CALLER (identified by the authenticated key's own
user_id, not client input) is a current member per a fresh
get_user_object lookup. team_alias and max_budget are never read back
from metadata at all -- team_alias comes from a live get_team_object
lookup and max_budget is recomputed with the exact same capping logic
the initial SSO login poll uses. _mint_cli_refresh_token no longer
accepts or stores team_alias/max_budget, only the team_id hint.

Added regression tests proving: a forged/stale team_id is dropped
(falls back to no team, not silently honored), and a forged max_budget
in metadata never reaches the issued JWT.

* fix(ci): catch HTTPException specifically instead of bare Exception (BLE001)

* fix: un-consume refresh token if minting the replacement fails

Greptile flagged a real reliability gap: cli_refresh_token blocks the
presented token atomically, then does several more DB calls before
returning a replacement (user lookup, team lookup, JWT mint, new
refresh-key mint). Since this endpoint exists specifically for fully
unattended apiKeyHelper operation, a single transient failure in that
window (DB hiccup, etc.) permanently stranded the user: their old
token was already dead and no new one was issued, with no recovery
path short of a full interactive browser re-login.

Wrap that window in try/except; on any failure, best-effort revert the
consumed token back to usable (blocked=False) before re-raising, so a
retry can succeed. Standard compensating-action pattern since
generate_key_helper_fn doesn't take an injectable transaction, so
wrapping the whole thing in a real DB transaction isn't practical here.

* fix(security): refresh key had unrestricted model access, not none

Critical bug: _mint_cli_refresh_token used models=[] intending "no LLM
access", but that's backwards in this codebase. Per
_check_model_access_helper: `len(filtered_models) == 0 and len(models)
== 0` -> all_model_access = True. An empty models list on a key with no
team_id means UNRESTRICTED access to every model, not zero access. The
CLI refresh token -- meant to be usable for nothing but silently
exchanging itself for a new JWT -- was actually a fully unrestricted
API key for its entire 90-day lifetime, completely undermining the
whole point of keeping it separate from the short-lived call
credential.

Fixed with two independent layers: allowed_routes hard-restricts the
key to exactly /sso/cli/refresh and /sso/cli/logout (the real enforced
boundary, checked in the shared user_api_key_auth dependency for every
route); models is set to an unmatchable sentinel string as
defense-in-depth in case any code path only consults the models field.

Added an end-to-end regression test that exercises the actual
model-access-control function against a key shaped like the minted
refresh token, rather than only asserting on what arguments were passed
to the key-generation call -- the latter kind of test is exactly what
let the original bug ship, since asserting `models == []` is equally
consistent with "no access" and "unrestricted access" without checking
what the access-control code actually does with that shape.

Also: the compensating-rollback added for reliability un-blocked a
consumed refresh token even when the underlying user no longer exists.
That's a permanent, intentional rejection, not a transient failure --
un-blocking it would let a stale refresh token become valid again for a
different account if the user_id is ever reused/re-registered. Moved
the user-existence check outside the rollback-on-failure block so it
stays permanently blocked.

* refactor: rotate CLI refresh tokens via regenerate_key_fn instead of hand-rolled consume/rollback

The refresh token is already a plain litellm virtual key, so rotation can
delegate to the same atomic DB update /key/regenerate uses instead of a
bespoke update_many + compensating-rollback dance. This makes silent CLI
refresh an Enterprise feature, same as regular key regeneration.

* refactor: replace CLI stateless JWT + refresh-key pair with one self-rotating virtual key

The CLI previously minted two credentials on login: a stateless self-signed
JWT for LLM calls, and a separate DB-backed refresh-only key (scoped away
from ever calling an LLM) just to authorize minting a new JWT. Collapse
this into a single real virtual key, used directly as the LLM bearer token
and re-presented to /sso/cli/refresh to rotate its own secret in place.

This also means the CLI session key now shows up in the Admin UI's Keys
page and can be revoked/regenerated like any other key, rather than being
an invisible, unmanageable stateless token.

* refactor: drop silent CLI refresh, key just expires and requires re-login

/sso/cli/refresh only ever benefited Enterprise deployments (regenerate_key_fn's
gate), while everyone else already fell through to "re-run lite login" on
failure. Cut the endpoint, the rotation logic, and the client-side refresh
path entirely; print-token now just prints the cached key until it hits its
LITELLM_CLI_JWT_EXPIRATION_HOURS duration, then fails fast telling the user
to log in again. Session key itself is unaffected: still a real, revocable
virtual key visible in the Keys UI, `lite logout` still revokes it directly.

* fix(ci): regenerate schema.d.ts after removing /sso/cli/refresh route

* revert: go back to stateless JWT, keep only lite auth print-token

The virtual-key redesign (revocable, Keys-UI-visible credential) wasn't
needed just to support print-token, and cost real server-side surface
(a mint path, a logout-revoke endpoint, migrated tests/docs) for a property
this repo doesn't need yet. Reverting cli_poll_key/_types.py/schema.d.ts
back to the original stateless-JWT design; the only durable addition from
this whole effort is `lite auth print-token` (reads the cached credential,
prints it while fresh, fails with a clear message once it's past
LITELLM_CLI_JWT_EXPIRATION_HOURS) plus the base_url_explicit plumbing it
needs. `lite logout` goes back to clearing the local file only, since a
stateless JWT can't be revoked server-side.

* refactor: move CLI token freshness check to cli_token_utils, drop unnecessary renames

Addresses review: the freshness check is a pure token-shape/timestamp
util, not command logic, so it belongs alongside the other SDK-level
CLI token helpers (load_cli_token, get_litellm_gateway_api_key) rather
than in commands/auth.py. Also reverted a few incidental jwt_token/
session_key variable and string renames that weren't load-bearing.

* fix(mcp): run the route gate on bridge admission so allowed_routes are enforced

The envelope arm reloaded the identity and ran _run_centralized_common_checks
but skipped RouteChecks.should_call_route, which the standard pipeline runs
between the builder and common_checks. Because the centralized checks treat MCP
as an inference route and never re-check allowed_routes, a key barred from MCP
routes could mint an envelope at the token endpoint (not itself an MCP route)
and replay it against MCP. Run the route gate before admitting, and clear the
request-scoped budget_reservation, matching the wrapper's sequence; a disallowed
route now surfaces the gate's own 403.

* fix(proxy): reserve budget for tiered pricing

Ensure tier-only models reserve their estimated request cost so concurrent requests cannot bypass exhausted budgets.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(proxy): bill tier-only deployments instead of $0

Route cost calculation to the deployment's router_model_id entry when it carries tiered_pricing but no flat per-token rate, so models like dashscope/qwen3.7-plus are billed via their tier table rather than the pricing-stripped shared alias.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(ui): convert activity metrics charts to shadcn/recharts (#32726)

* refactor(ui): convert activity metrics charts to shadcn/recharts

Swap the seven tremor AreaChart/BarChart sites in activity_metrics.tsx to
the shared shadcn/recharts wrappers and switch CustomLegend/CustomTooltip
to the ported versions in shared/charts. Chart props, colors, formatters,
and legend behavior are unchanged; tests now assert on real recharts SVG
output instead of tremor mocks.

* fix(ui): restore tremor No data placeholder for empty AreaChart data

* test(ui): scope activity metrics chart assertions to card titles instead of render order

* feat(ui): extend topnav border across the sidebar header (#32920)

Pin the sidebar header to the same 56px height as the dashboard topnav and
give it a matching bottom border, so the two borders sit flush and read as one
continuous line. Revert to auto height when the rail is collapsed so the
stacked logo and toggle are not clipped.

* fix(cost): coerce string tiered-pricing costs and share tier helper

YAML-parsed tier costs can arrive as strings (e.g. "4e-07"), which broke
arithmetic in the graduated tiered-pricing calculation. Coerce per-token
costs to float in both the in-range and remaining-tokens paths.

Move the tiered-cost helper out of the Dashscope module into a
provider-neutral home so the proxy budget reservation no longer depends on
a provider-specific module.

Co-authored-by: Cursor <cursoragent@cursor.com>

* docs(anthropic): clarify the Opus 4.5 branch in adaptive-effort translation

Add an inline comment explaining that the effort-capable non-adaptive branch in
_translate_adaptive_effort_for_non_adaptive_model exists for models like Claude Opus
4.5 that accept output_config.effort but reject adaptive thinking, and why effort-only
requests pass through while adaptive requests with an unsupported effort level fall
through to the legacy translation.

* fix(anthropic): translate raw adaptive thinking for chat completions on pre-4.6 models

Clients that pass thinking={"type": "adaptive"} directly (not via the
reasoning_effort alias) on the /chat/completions interface had it forwarded
unmodified to pre-4.6 Anthropic models, which reject the shape. Mirrors the
translation already applied on the native /v1/messages passthrough (#32867):
translate to legacy thinking={type: enabled, budget_tokens}, capped below
max_tokens, dropping thinking when max_tokens can't fit even the minimum
budget. Hoists the shared budget-capping helper onto AnthropicConfig so both
paths use one implementation.

* fix(proxy): reserve tiered budget all-or-nothing across all deployments

Alibaba Model Studio (Dashscope) tiered pricing is all-or-nothing: the tier is
selected by a request's total input tokens and every token, input and output, is
billed at that one tier's rate. The reservation path used graduated slicing and,
worse, picked the output tier from the output-token count, so a long-context
request with a large output allowance reserved far less than the provider charges
and could slip past a depleted budget. Select the tier from input tokens and apply
its rates to all input and output tokens.

Reservation also read tiered pricing from only the first deployment in a model
group. A caller could hit an alias whose cheaper deployment was listed first and
exceed the budget once routed to a costlier sibling. Estimate against every
eligible deployment's pricing and reserve the maximum.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(complexity_router): log the cause of each routing decision

The complexity router's info log didn't say what drove a routing decision.
Literal and semantic keyword matches logged an identical "keyword rule fired"
line (no way to tell which mechanism fired), and the scorer's line carried no
consistent marker tying it to the same question.

Emit one greppable line per decision naming the cause: literal_keyword_match,
semantic_keyword_match, or complexity_scorer. The hook already knows which ran
(the config's semantic_keyword_matching flag distinguishes lexical from
semantic; the override-vs-scorer branch distinguishes keyword match from
scorer), so this is label-only: no behavior change, no new types, no added
latency.

Adds regression tests asserting each decision path logs its cause; they fail if
a label is swapped or the cause= marker is dropped.

* feat(ui): add redesigned sidebar account menu (#32931)

* feat(ui): add redesigned sidebar account menu

Introduce SidebarAccountMenu, a sidebar-only account/logout menu built on
shadcn Popover/Switch/Badge/Separator/Button, and wire it into leftnav in
place of the shared UserDropdown. The panel has a LiteLLM header with the
bouncing moon and a clickable version tag, Tier/Role/Email/User ID rows
with copy actions, the five display toggles, and Logout.

UserDropdown is left untouched so the control-plane / chat navbar keeps
its existing menu. The version tag links to the same release notes page
as the navbar tag, and the bouncing icon reuses the existing header
animation gated by the Hide Bouncing Icon toggle.

* test(ui): point account-menu e2e specs at the migrated sidebar menu

The sidebar account menu moved from an antd Dropdown to a Base UI popover
(SidebarAccountMenu), so the login, logout, proxy-logout-url, and internal
user identity specs were still waiting on antd-era locators
(.ant-dropdown, the popupRender wrapper class, the user-dropdown-panel test
id, and a menuitem-role Logout). Point them at the new panel test id
(sidebar-account-menu-panel) and the button-role Logout instead. The logout
behavior is unchanged since both menus call the same useLogout handler.

* fix(bedrock-converse): translate adaptive thinking for pre-4.6 models

Follow-up to #32867 (native /v1/messages) and the /chat/completions
commit earlier on this branch, extending the same adaptive-thinking
translation to the Bedrock Converse path.

Clients like Claude Code send thinking={type: "adaptive"} on every
request. When routed via Bedrock Converse to pre-4.6 models
(claude-haiku-4-5, claude-sonnet-4-5), this was forwarded as-is and
rejected by the model. Mirrors the translation already applied on the
/chat/completions and /v1/messages paths: map to legacy
thinking={type: enabled, budget_tokens}, capped below max_tokens.

Also fixes the missing custom_llm_provider arg in the chat completions
path's call to AnthropicConfig._map_reasoning_effort.

* fix(mcp): run proxy-wide pre-DB gates on bridge envelope admission

The envelope arm bypasses user_api_key_auth, so it never ran
pre_db_read_auth_checks (request-size and body-safety limits, the IP allowlist,
and the general_settings route allowlist) that the normal MCP admission path
runs before any key lookup. A caller blocked by IP or a disallowed proxy route
could be admitted through an envelope where the same principal on the normal
path is rejected. Run those gates before the envelope crypto, mirroring the
pipeline's pre-DB ordering; a blocked IP or route surfaces its own 403.

* fix(anthropic): pass resolved provider to adaptive-thinking check

The rebase onto staging changed _is_adaptive_thinking_model to require
custom_llm_provider (no default), so the one-arg call in the raw adaptive
thinking branch raised TypeError at runtime for any /chat/completions
caller sending thinking={type: adaptive}. Use self._resolved_provider,
matching the reasoning_effort branch just below. Caught by Greptile.

* test(bedrock-converse): cover adaptive-thinking drop when max_tokens too small

Adds the regression test for the warning-drop branch in the Converse
adaptive-thinking translation, mirroring the chat completions path's
test_raw_adaptive_thinking_dropped_when_max_tokens_too_small.

* fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712)

* fix(guardrails): filter Add-Guardrail mode dropdown per provider

The GET /guardrails/ui/add_guardrail_settings endpoint returned every
GuardrailEventHooks value in one flat supported_modes list, so the Admin
UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving
Content Filter or Tool Permission with pre_mcp_call then failed with a
400 because those guardrails' server-side supported_event_hooks list
excludes it.

Expose each guardrail's supported hooks as a get_supported_event_hooks
classmethod on CustomGuardrail (mirrors the existing get_config_model
pattern) and have the endpoint iterate guardrail_class_registry to build
a supported_modes_by_provider map. The UI Mode dropdown filters by that
map when the selected provider is known and falls back to the global
list otherwise. __init__ now sources its own supported_event_hooks list
from the classmethod so the two sides can't drift.

Also register BedrockGuardrail, ToolPermissionGuardrail, lakera,
lakera_v2, and presidio in guardrail_class_registry so they participate
in the map (they were previously only in guardrail_initializer_registry
and had no class-registry entry).

Behavior change: guardrails that previously had no supported_event_hooks
declared (aim, javelin, azure/text_moderation, cato_networks,
crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx,
prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai,
lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now
validate the configured mode at instantiation. Existing configs where
the mode was silently a no-op will fail at proxy startup with a clear
validation error rather than running as a broken guardrail.

Resolves LIT-4226

* fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form

Address Greptile P1 (startup break) and P2 (edit form UX):

LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported
event_hook, unchanged behavior for the guardrails validated pre-PR).
Setting it to false logs a warning and continues, giving deployments an
opt-out while they fix configs that now surface as errors instead of
silently no-op'ing. Regression test covers both modes.

Edit form now surfaces the currently-saved mode even when it is not in
the filtered per-provider list, so a legacy row (e.g. content_filter
saved with pre_mcp_call before this fix) no longer disappears from the
dropdown; the option renders with a 'not supported by <provider>' note
so the user knows to pick another.

* fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint

Audited every get_supported_event_hooks classmethod against the hooks
each guardrail's own tests exercise and its handler methods. Five were
too narrow and their tests caught it in CI: rubrik gains pre_call,
presidio gains during_call and pre_mcp_call, prompt_security, onyx and
qualifire gain during_call. The remaining classes match either their
original __init__ declarations or their exercised modes exactly.

Cursor review fixes: the Add form now drops selected modes the new
provider does not support when the user switches providers, so a
pre_mcp_call selection cannot ride along into a provider that rejects
it at save; the edit form handles list-shaped stored modes instead of
treating mode as always a string.

Extracted shared toModeArray and getSupportedModesForProvider helpers
into guardrail_info_helpers so both forms use one implementation, typed
the remaining any usages in both forms, removed nested ternaries, and
committed the ratcheted-down eslint metrics and pruned suppressions

* fix(proxy): reserve tiered output at the higher reasoning rate

Some tiered Dashscope models price reasoning output above standard output
(output_cost_per_reasoning_token > output_cost_per_token). The reservation charged
all output at the standard rate, so a reasoning-heavy request reserved too little
and concurrent calls could exceed the budget before reconciliation. The reasoning
share is unknown before the request runs, so reserve every output token at the
higher of the two configured rates, for both tiered and flat pricing.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(ui): convert user agent and per-user usage charts to shadcn/recharts (#32725)

* refactor(ui): convert user agent and per-user usage charts to shadcn/recharts

Swap the tremor BarChart import for the shared shadcn/recharts wrapper in
user_agent_activity.tsx (DAU/WAU/MAU charts) and per_user_usage.tsx (usage
distribution histogram). All chart props are unchanged; the wrapper exposes
the same tremor prop surface with matching defaults.

Extend user_agent_activity.test.tsx and add per_user_usage.test.tsx with
parity assertions on the real recharts SVG output: bar series per category,
stacked x positions, resolved fill colors, axis bucket labels, legend text,
and value formatter output on axis ticks. Remove the dead ResizeObserver
polyfill in user_agent_activity.test.tsx now that the scoped global mock in
tests/setupTests.ts renders charts, which also lowers the no-explicit-any
metric by one.

* test(ui): harden bar x-position parsing against recharts path format

* feat(ui): working Test Connection for the complexity auto router

The consolidated auto-router tab dropped the Test Connection button because
the shared prepareModelAddRequest helper returns an empty array for an auto
router (it has no model_mappings), so the caller crashed destructuring
result[0].litellmParamsObj. That is the crash in #31590 and the open PR
#31794. #31794 only silenced the crash by pointing the test at
auto_router/complexity_router, which is not a provider model, so the
/health/test_connection health check (a real litellm.ahealth_check
completion) would still error.

Bring the button back and make it meaningful: an auto router dispatches to
saved model groups, so Test Connection now probes those directly. It builds
a deduped target list from the configured tiers (tiers sharing a model group
collapse to one probe) plus the embedding model when semantic keyword
matching is on, then runs a live /health/test_connection against each and
shows per-target pass/fail. This never touches prepareModelAddRequest, so the
original destructure crash cannot recur.

Scope is the recommended complexity router only; the to-be-deprecated
semantic router is untouched. No backend changes.

Supersedes #31794. Resolves #31590.

* refactor(ui): extract a shared CopyButton and fix the sidebar copy confirmation (#32945)

* fix(ui): show sidebar copy confirmation only on a successful write

The sidebar account menu's copy button switched to the checkmark
synchronously, before the clipboard write settled, so it confirmed a
copy that never happened when navigator.clipboard was undefined on
non-secure origins or when writeText rejected. The handler now guards
navigator.clipboard, awaits the write, and flips to the checkmark only
on success

Also updates the header accent emoji in the same menu

* refactor(ui): extract a shared CopyButton for the sidebar account menu

The copy-icon-to-checkmark pattern was hand-rolled in several places,
including the sidebar account menu whose private copy button held the
false-confirmation bug. Extract a single canonical CopyButton into
components/shared, built on the Button primitive with a guarded and
awaited clipboard write so the checkmark appears only on a real
success, and have SidebarAccountMenu consume it

The success and failure-mode coverage now lives in the shared
component's own test; the sidebar test keeps one case asserting the
email row is wired to it

* fix(ui): probe auto-router tiers via real proxy routing, not /health/test_connection

Live testing showed the first cut was broken: /health/test_connection merges
{...configParams, ...requestParams}, so passing the public model_group name as
the request model overrode the resolved provider model and every tier failed
with "LLM Provider NOT provided". The frontend only has the public group name,
not the underlying litellm_params, so it cannot build the request that endpoint
needs.

Switch to testing each model group the way production actually routes it: send a
minimal request to /v1/chat/completions (or /v1/embeddings for the embedding
model) by public group name through the shared apiClient. The router resolves
the group, credentials, and provider itself, so a green row means the tier is
genuinely reachable. Verified live: voyage embedding returns 200, a tier with a
bad key returns the real provider auth error.

Also address Greptile feedback: rows now update progressively as each probe
settles instead of all at once, and TIER_ORDER is derived through a
`satisfies Record<keyof ComplexityTiers, null>` guard so adding a tier without
listing it is a compile error.

* fix(completion): forward aws credential kwargs into litellm_params so the responses bridge keeps WIF auth

Chat-completions requests to responses-only Bedrock Mantle models are
bridged to the Responses API, but completion() forwarded only
aws_bedrock_project_id into get_litellm_params, so aws_role_name,
aws_web_identity_token, aws_session_name and the other SigV4 credential
kwargs never reached sign_request and botocore fell back to the default
credential chain ("Bedrock Mantle auth failed: no Bearer token and no
usable AWS credentials"). Forward the whole AWS credential kwarg family,
extracted from the OPTIONAL_KWARGS_KEYS set get_litellm_params already
supports.

* refactor(ui): convert entity usage and usage page charts to shadcn/recharts (#32729)

* refactor(ui): convert entity usage and usage page charts to shadcn/recharts

Swap the tremor BarChart/DonutChart render sites in EntityUsage,
SpendByProvider, TopKeyView, TopModelView, KeyModelUsageView and
UsagePageView to the shared shadcn/recharts wrappers. Convert the two
sole-chart Daily Spend cards and the KeyModelUsageView card to the
shadcn Card primitives.

Close the donut parity gap with strictly additive optional DonutChart
props: showLabel/label render a center total (tremor showed
valueFormatter(sum) by default) and startAngle/endAngle forward to the
Pie so both provider donuts keep tremor's clockwise-from-12 layout.
Defaults preserve the previous wrapper behavior.

DailyData and two site-local row types move from interface to type
alias so they satisfy the wrappers' Record<string, unknown> constraint;
interfaces lack implicit index signatures.

Tests now assert on real recharts output: bar/sector counts, cyan
fills, axis labels, donut center totals, and the TopKeyView bar-click
drill-down into the key info modal. The dead tremor chart mocks in
UsagePageView.test.tsx are removed and lint metrics/suppressions are
regenerated for the dropped tremor imports.

* fix(ui): compute donut center label only when shown and assert Model Usage renders as a card title

* refactor(e2e): bucket rate limits, budgets, and spend tracking under quota_management

* fix(e2e): name the route the spend_calculate registry cell actually exercises

* refactor(e2e): move budgets and spend_tracking suites under quota_management

* test(e2e): cover key rpm/tpm rate limiting, window reset, and pacing headers

* test(e2e): assert the tpm block at its exact token crossing instead of a call-count heuristic

* test(e2e): fail the rpm reset test when the limiter resets early

* test(e2e): name the tpm window deadline's latency margin

* refactor(e2e): model the tpm spend loop's two outcomes as values

* test(e2e): source the ratelimit suite's model from E2E_CHEAP_ANTHROPIC_MODEL

* feat(guardrails): add pre_mcp_call support to Content Filter (#32936)

* feat(guardrails): add pre_mcp_call support to Content Filter

* test(guardrails): cover canonical MCP key gate under pre_mcp_call mode

* fix(guardrails): scan MCP arguments per value and gate mixed-mode scans by call type

* fix(guardrails): cap MCP argument scan depth and register the walker with the recursion detector

* test(guardrails): update LIT-4226 UI settings tests for content filter pre_mcp_call support

* fix(guardrails): use builtin generics in MCP scan annotations to satisfy strict-rule budget

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix(bedrock): allow bedrock-mantle:CreateInference in the web identity session policy

* fix(ui): drop max_tokens from the auto-router connection probe

max_tokens=1 makes reasoning models (o1/o3/...) return a 400 "max_tokens
reached" because reasoning tokens count against the cap, so a reachable
reasoning tier showed a false failure in Test Connection. Live-verified: o3
400s with the cap and succeeds without it.

Extract the request shape into a pure buildModelGroupTestRequest and cover it
with a test asserting the chat body carries no max_tokens (or
max_completion_tokens), so this regression is caught in unit tests instead of
only against a live reasoning model.

* test(main): assert the responses bridge forwards static aws keys as well as web identity params

* bump: litellm-proxy-extras 0.4.75 -> 0.4.76 (#32957)

* feat(proxy): add expires filter to GET /key/list (#32953)

* feat(proxy): add expires filter to GET /key/list

Add an opt-in expires query param to GET /key/list so callers can fetch
only expired or only active keys without paginating every page and
filtering client-side. 'expired' matches keys whose expires is in the
past (NULL expires excluded); 'active' matches keys that never expire or
expire in the future. Omitting the param preserves existing behavior for
every caller. An unrecognized value returns HTTP 400 rather than silently
returning all keys.

The filter is pushed to the database via the existing Prisma where
builder so callers avoid pulling the full key table into application
memory.

Resolves LIT-3387

* refactor(proxy): declare VALID_EXPIRES_FILTER_VALUES before its first use

* refactor(ui): colocate the usage view, keeping the shared usage components (#32952)

Split for the usage (UsagePage) segment. Most of the folder is the usage page's
own view, but four pieces are reused elsewhere and stay in @/components/UsagePage:
TopKeyView (old-usage), KeyModelUsageView and value_formatters (activity_metrics),
and the shared types (activity_metrics, chartUtils). The other 21 files move into
usage/_components, preserving the folder structure.

The external consumers import only the retained files, so they are untouched. The
moved files' imports of the retained files become @/components/UsagePage paths,
other escaping relative imports are absolutized, and lint suppressions are re-keyed
for moved files only. No behavior change.

* chore: update Next.js build artifacts (2026-07-11 23:35 UTC, node v20.20.2) (#32960)

* test(e2e): cover model-aware mid-conversation system handling on Bedrock Invoke /v1/messages

* docs(github): add QA runbook section to the PR template

* docs(github): scope the QA runbook to tests/e2e edits and add example checklists

* docs(github): shape QA runbook examples as node id plus behavior bullets

* refactor(ui): convert projects page chart to shadcn/recharts (#32722)

* fix(xecguard): use StandardLoggingGuardrailInformation in logging hook (#32911)

XecGuard's async_logging_hook wrote a bare dict to
standard_logging_object["guardrail_information"] while the typed
contract is Optional[List[StandardLoggingGuardrailInformation]].
Readers that iterated the field walked dict keys, raised on
info.get, or silently dropped the entry from guardrail usage
tracking and spend-log writes

Construct the typed entry and append it to the existing list or
create a new one, matching the shared helper pattern. Record the
configured guardrail name instead of a hardcoded "xecguard" and
pass the GuardrailEventHooks enum for guardrail_mode

* feat(ui): adopt openapi-react-query ($api) and convert useCustomers (#32949)

* feat(ui): adopt openapi-react-query and convert useCustomers to $api

Add openapi-react-query and expose $api = createQueryClient(fetchClient)
alongside fetchClient. Rewrite useCustomers as
$api.useQuery("get", "/customer/list", {}, { enabled, select }), which
derives the query key from method + path (dropping the hand-written
createQueryKeys entry and the manual key) and forwards the request signal
for cancellation. The response type still flows from schema.d.ts as
CustomerResponse[]. Tests assert the path, the admin/token enabled gate,
and the empty-body select fallback.

* test(ui): read the last render's options in useCustomers helper

The lastCallOptions helper was named for the last call but read
mock.calls[0]. Harmless while each test renders once, but it would
silently assert against first-render options if a test ever re-renders.
Read the final call instead.

* refactor(ui): colocate the mcp-servers view, keeping the shared mcp_tools surface (#32968)

* refactor(ui): colocate the usage view, keeping the shared usage components

Split for the usage (UsagePage) segment. Most of the folder is the usage page's
own view, but four pieces are reused elsewhere and stay in @/components/UsagePage:
TopKeyView (old-usage), KeyModelUsageView and value_formatters (activity_metrics),
and the shared types (activity_metrics, chartUtils). The other 21 files move into
usage/_components, preserving the folder structure.

The external consumers import only the retained files, so they are untouched. The
moved files' imports of the retained files become @/components/UsagePage paths,
other escaping relative imports are absolutized, and lint suppressions are re-keyed
for moved files only. No behavior change.

* refactor(ui): colocate the mcp-servers view, keeping the shared mcp_tools surface

* docs(github): add Final Attestation and per-test sanity-check step to QA runbook

* refactor(ui): convert endpoint usage charts to shadcn/recharts (#32723)

* refactor(ui): convert endpoint usage charts to shadcn/recharts

Adds a LineChart wrapper to the shared charts kit, mirroring the
BarChart/AreaChart composition with connectNulls and curveType props,
and converts EndpointUsageBarChart and EndpointUsageLineChart from
tremor to the shared wrappers. Both endpoint chart tests now assert on
real recharts SVG output instead of tremor mocks.

* refactor(ui): drop unused endpointData prop from EndpointUsageLineChart

* fix(ui): point endpoint chart test type imports at the UsagePage types alias after colocation move

* fix(auto_router): filter embedding models out of tier selects, require all tiers, add inline validation

The Add Auto Router complexity tab let chat models fill the embedding-model
slot (and vice versa) since neither dropdown filtered on ModelGroup.mode, and
submit only required at least one of the four tiers instead of all four. Adds
getMissingTiersError alongside the existing getSemanticConfigError, and
highlights unfilled tier/embedding selects inline once a submit attempt fails.

* fix(model-cost-map): anchor the bedrock-claude-ids routing rule to the start of the id

* fix(proxy-auth): deny provider-wildcard access inferred through an unrecognized model namespace

* fix(auto_router): reset inline validation errors when switching router type

* fix(auto_router): flag name field and tier fields together on empty submit

Clicking Add Auto Router with the name empty returned early with only a
toast, so blank tier selects never got their inline error state. The
empty-name branch now sets showValidationErrors and triggers antd
validation on the name field, so every unfilled mandatory field is
flagged at once. Adds a regression test for the tab component.

* feat(mcp): mint gateway-bound envelope at the token endpoint for dcr_bridge oauth_delegate

* feat(mcp): seal the authorizing key hash in the dcr_bridge envelope

The mint bound only user_id/server_id into the envelope, which gave admission
no way to reload the caller's key and enforce its current restrictions. Seal the
hashed authorizing key instead (a one-way digest, not a usable credential), so
admission reloads the live UserAPIKeyAuth by it and the key's team/org/tool
permissions and revocation apply per request.

Extract the token endpoint's key resolution into a shared _resolve_active_litellm_key
so the per-user token store (user_id) and the bridge mint (key hash) derive from one
active-key-gated path, and fail the mint closed with invalid_request when no active
key accompanies the request.

* fix(mcp): return 502 not KeyError when a bridge upstream response lacks access_token

The eager access_token = token_response["access_token"] extraction ran before
the dcr_bridge branch, so a missing upstream access_token raised an unhandled
KeyError and _bridge_grant_from_token_response's nil guard (which maps to a clean
502) was dead code. Move the extraction onto the non-bridge result path so the
bridge branch reaches its 502 guard.

* fix(mcp): let a keyless-user active key mint a bridge envelope

_resolve_active_litellm_key gated on _active_key_user_id, which returns None both
for blocked/expired keys AND for valid keys with no user_id, so a team-scoped or
service-account key was wrongly rejected with invalid_request at bridge token
exchange. Split the active-state gate (_key_is_active: blocked/expiry only) from
the user_id extraction; the mint seals the key hash, not the user, and admission
already handles a keyless-user key. The per-user token store still gets no user
for such a key, as there is none to key a stored credential by.

* style(mcp): use X | None annotations on the touched key-resolution helpers

The keyless-user fix moved these signatures, so their pre-existing Optional[...]
annotations counted against the diff and tripped the UP045 strict-budget gate.
Modernize the four touched return annotations to the X | None form the gate
wants; runtime behavior is unchanged.

* fix(mcp): coerce numeric expires_in and make the active-key check total

Two correctness gaps in the bridge mint. _bridge_grant_from_token_response only
accepted an int expires_in, dropping a float (3600.0) or numeric-string ('3600')
lifetime to None so the envelope fell back to its 1h cap and could outlive a
shorter-lived upstream token; coerce it to a positive int (bool excluded). And
_key_is_active called datetime.fromisoformat on the str|datetime expires outside
the resolver's try, so a malformed stored expiry raised an unhandled 500 instead
of the fail-closed invalid_request; it now fails closed (inactive) on an
unparseable expiry. Regression tests cover int/float/string/bool coercion, the
short-float TTL, and the malformed-expiry fail-closed path.

* fix(mcp): harden the bridge token mint (multi-lens review pass)

Findings from a full adversarial review of the mint path across security,
correctness, error-handling, concurrency, and OAuth-protocol dimensions.

- expires_in coercion is now total: int(float(...)) can raise OverflowError on
  Infinity / a giant numeric string, which escaped the ValueError/TypeError catch
  and 500'd the token endpoint. Unified to catch OverflowError too.
- Resolve the litellm identity BEFORE exchanging the single-use upstream code, so
  a missing or transiently-unresolvable identity fails closed with invalid_request
  without burning the code (the mint re-resolves via a cache hit).
- The no-identity failure is now an RFC 6749 5.2-shaped invalid_request
  (JSONResponse, top-level error, no-store) instead of a detail-wrapped
  HTTPException, matching the BYOK OAuth endpoint.
- EnvelopeTooLarge (upstream token too big to seal) surfaces a 502, not a 500.
- The upstream refresh_token is no longer sealed into the envelope: the edge
  never consumes it, so it was dead weight embedding a long-lived upstream
  credential in the client bearer and enlarging the envelope; refresh is a
  follow-up (a dedicated refresh-envelope).

Security review found no exploitable defect (forgery, cross-server/user replay,
leakage, confused-deputy all closed). Regression tests cover the OverflowError,
the code-not-burned path, the RFC-shaped error, the 502, and the dropped refresh.

* fix(mcp): close the burn-before-check gate for both grants and validate master_key first

Follow-up to the pre-exchange identity gate, which I had only added to the
authorization_code branch and which left the master_key check inside the mint
(after the upstream exchange) - so the very burn-then-fail pattern it was meant to
prevent still applied to refresh_token grants and to a misconfigured gateway.

- Hoist a single pre-exchange gate above the upstream call that covers BOTH grant
  types: it fails closed (invalid_request) on an unresolvable litellm identity and
  500s on an unset master_key BEFORE the single-use code or refresh token is
  exchanged/rotated, so a bad key or a misconfigured gateway never burns the
  upstream credential.
- Report expires_in from the envelope JWT's own second-truncated exp (rounding the
  elapsed portion up) instead of the raw expires_at - now delta, so the client is
  never told the bearer is valid past the ~1s point admission already expires it.

Regression tests assert the upstream exchange is never called on the no-identity
refresh grant and the master_key-unset path, and that the reported expires_in does
not overstate the JWT exp.

* refactor(mcp): make the bridge delegate mint a phased failures-as-values pipeline

The dcr_bridge oauth_delegate token mint validated its preconditions in two
places: a pre-exchange guard inside exchange_token_with_server (master_key set,
resolvable litellm identity) and an authoritative re-check inside the post-exchange
_mint_bridge_delegate_token_response. Keeping the two in step by hand is what kept
producing the same class of finding: a precondition guarded on one grant branch but
not the other, master_key checked after the exchange on one path, identity resolved
twice, and each failure raising an ad-hoc HTTPException with its own status and body
shape.

Model the mint as three phases whose failures are values. _prepare_bridge_mint runs
before the exchange, checks every precondition once (master_key, then identity), and
returns either a frozen _BridgeMintReady carrying the resolved key hash and the
master-key-derived envelope keys, or a _BridgeMintError literal. Because every
precondition lives in prepare, and prepare runs before the upstream POST, no failure
can burn the single-use code or rotate a refresh token, for either grant type, by
construction rather than by a guard we have to remember to keep in sync.
_finish_bridge_mint runs after the exchange and has no preconditions left that can
fail; its only failure values are properties of the upstream response itself (no
usable access_token, or a token too large to seal). One mapper,
_bridge_mint_error_response, turns each _BridgeMintError into an RFC 6749 section
5.2-shaped body with a status truthful about where the failure is (400 for the
caller, 500 for gateway config, 502 for the upstream), with an exhaustive match plus
assert_never so a new failure mode cannot be added without a matching status.

Behavior is unchanged for the client. Every failure that previously raised now
returns the same status as an OAuth error body, which is the correct token-endpoint
contract; the three tests that asserted a raised HTTPException now assert the
returned response. _exchange_for_bridge_server additionally asserts the identity
resolver is awaited exactly once for a bridge server and never for a non-bridge one.

* fix(mcp): let the bridge envelope report expires_in 0 at the jwt exp boundary

_finish_bridge_mint floored the reported expires_in at 1. Admission expires the
envelope against the JWT's second-truncated exp, so when the mint lands in the same
second that exp falls on (a sub-second upstream lifetime, for instance), the true
remaining life is 0 and reporting 1 tells the client the bearer lives one second past
the point admission already rejects it. Floor at 0 instead so the reported lifetime
never overstates the exp; the value still cannot go negative.

The regression pins the boundary directly: minting at now=100.25 with a 1s upstream
token seals exp=101, and the reported expires_in is max(0, 101 - ceil(100.25)) = 0.
Under the old floor of 1 it reads 1, so the test fails on that mutation.

Also drops the unused mcp_server parameter from _prepare_bridge_mint; identity and
key derivation there never referenced the server.

* refactor(mcp): make bridge-mint resolvers return tagged unions so status is truthful by construction

Three findings landed together, all one defect: a resolution step crushed several distinct outcomes
into a single None or a silent default, so the mint's error mapper could not tell them apart and
assigned the wrong status. Identity resolution mapped a database outage to the same None as a missing
credential, which the mint reported as 400 invalid_request, blaming the caller for a gateway outage
while admission statuses the same outage 503/500. Lifetime coercion mapped an explicit non-positive
expires_in to the same None as an absent one, so an upstream token the IdP reports as already dead was
sealed into an hour-long envelope. And the refresh_token grant was run through the upstream exchange
(which can rotate the client's upstream refresh credential) and its result then discarded, even though
a bridge server seals no refresh_token and the client never holds one to present.

Rather than add a mapping branch per finding, the fix changes the return types so a wrong status is not
representable. Each resolution step now returns a precise tagged value instead of None: identity
resolution returns a _ResolvedKey or one of no_active_key / unavailable / unresolvable, classified the
same way admission's _reload_admitted_key classifies the same conditions; upstream-lifetime
classification returns a positive number of seconds, "unspecified" (absent or unparseable, which the
envelope caps), or "expired" (a parseable non-positive value, an already-dead token); and upstream-grant
validation returns a typed grant or one of no_access_token / expired_lifetime. Thin exhaustive mappers
(match plus assert_never) lift each vocabulary into one bridge-mint taxonomy of eight named failures,
and a single _bridge_mint_error_response gives each its truthful RFC 6749 §5.2 status: 400 for the
caller's missing credential or an unsupported grant, 503 for a transient auth-DB outage, 500 for a
gateway that cannot resolve identity or is not configured, and 502 for an upstream response with no
usable token, an already-expired lifetime, or a token too large to seal. Adding a failure mode now
requires a new literal and a match arm the type checker forces, so the class of wrong-status bug cannot
recur silently.

The refresh_token grant is rejected in _prepare_bridge_mint before the exchange with
unsupported_grant_type, so it can never rotate or consume the client's upstream refresh credential;
renewal is re-running authorization_code, as the sealed refresh_token=None already intends. An absent or
unparseable expires_in still mints a capped envelope (the by-design behaviour for an upstream that omits
the field); only an explicitly-dead lifetime is rejected.

Tests cover the resolver's three failure classes (including a real connection-error outage and a missing
prisma_client), the mint statuses for each (503 before the upstream exchange, 500, 502 on an expired
upstream lifetime, and a capped mint on an unknown one), and the refresh-grant rejection before any
exchange. The three findings are mutation-checked: reverting each fix turns its regression test red.

* fix(mcp): treat a positive sub-second upstream lifetime as alive, not expired

_classify_upstream_lifetime decided "expired" from int(float(expires_in)), which truncates toward
zero, so a positive fractional lifetime in (0, 1) became 0 and was misread as already elapsed. That
rejected the mint with 502 in _finish_bridge_mint after the single-use upstream code had already been
consumed, even though the upstream reported a positive remaining lifetime.

Decide expired on the parsed numeric value rather than its truncated int, so only a genuinely
non-positive value is expired. The envelope works in whole seconds and cannot represent a sub-second
lifetime, so a positive value that truncates to 0 clamps up to the 1s floor instead of being rejected.
Values >= 1 still truncate toward zero so the envelope never claims more life than the upstream stated,
and NaN / Infinity / oversized input still read as unparseable ("unspecified").

Regression covers the classifier (0.5 and 0.001 clamp to 1, 1.9 truncates to 1, -0.5 stays expired) and
the mint (a 0.5s upstream lifetime mints a 200 envelope rather than a 502); reverting to the
truncate-then-check reddens both.

* fix(auto_router): inline error for missing LLM classifier model

Selecting the LLM classifier without picking a model only surfaced a
toast on submit; the classifier model select now gets the same red
outline and helper text as the tier and embedding selects once a submit
attempt has failed.

* build(dev-env): add make bootstrap and unprovisioned-checkout preflight to pre-commit

* feat(router): random-pick multi-model complexity tiers (#32967)

* feat(router): random-pick multi-model complexity tiers

Tier pools already make sense without adaptive; stop pinning lists to
index 0 and shuffle within the classified tier instead.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(ci): format complexity router config

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(ci): use PEP 585 types for tier pools

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(xecguard): sanitize scan result before recording it for logging (#32935)

* chore: keep it brief

* chore: keep it brief

* docs(readme): point developer-mode setup at make bootstrap

* chore: keep it concise

* feat(router): add Router(plugins=[...]) routing-plugin pipeline (#32972)

* feat(router): add Router(plugins=[...]) routing-plugin pipeline

Runs a sequence of user-supplied plugins before the routing decision is
made. Each plugin reads/mutates a RoutingContext (messages, candidate
models, metadata, signals); the narrowed candidate list is enforced when
picking a deployment, raising rather than silently falling back if a
plugin narrows to zero candidates.

Prototype for the routing-plugin pipeline discussed in #32168.

* fix(router): use ruff-modern typing, add raw/structured messages to RoutingContext

- Use dict/list/X|None instead of Dict/List/Optional in new code, staying
  within the ruff strict-rule budget ratchet
- Extract the guardrail-translation message normalization ComplexityRouter
  already had into a shared resolve_structured_messages() helper
  (litellm_core_utils/prompt_templates/factory.py), reused by
  ComplexityRouter and the new routing-plugin pipeline instead of
  duplicating it
- RoutingContext now exposes both raw_messages (as received) and
  structured_messages (normalized across chat completions / Anthropic
  messages / Responses API), mirroring CustomGuardrail.apply_guardrail's
  pattern, per review feedback on #32972
- Add direct unit tests for _run_routing_plugins and
  _filter_by_routing_plugin_candidates (router_code_coverage gate requires
  every router.py function be called by name somewhere in tests/)

* fix(test): rename to test_router_routing_plugins.py

router_code_coverage.py's AST scanner only inspects test files whose
filename contains the substring "router" -- test_routing_plugins.py
doesn't match (routing != router), so it silently skipped this file
and flagged _run_routing_plugins/_filter_by_routing_plugin_candidates
as untested despite the direct unit tests added for them.

* fix(router): fail closed when plugins are configured but the resolved
routing path can't run them

Router.completion() (and other sync entry points) resolves deployments
via the synchronous get_available_deployment(), which never runs
async_pre_routing_hook and therefore never runs the routing-plugin
pipeline. async_get_available_deployment() itself falls back to that
same synchronous method for routing strategies without an async-native
selector (e.g. legacy "usage-based-routing" v1). Both paths would let a
policy plugin (e.g. a deny-all rule) be silently bypassed.

Raise instead of silently proceeding when self.routing_plugins is
configured and the sync path is reached, since applying the pipeline to
every selector path is a larger change out of scope for this PR.

Per review: https://github.com/BerriAI/litellm/pull/32972/changes/BASE..bdfb583c2c6f8df10004fb249e11629d41ce71fa#r3565373303

* feat(router): soft-floor adaptive mode for complexity router (#32947)

* feat(router): soft-floor adaptive mode for complexity router

Let complexity_router_config.adaptive=true Thompson-sample across the
union of tier pools with a tier-distance penalty, and wire the existing
adaptive post-call bandit so mis-tiered requests can still recover.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(router): reattach adaptive hooks for hybrid complexity

Finalize was wiping every AdaptiveRouterPostCallHook and only
re-registering standalone auto_router/adaptive_router deployments,
so complexity adaptive=true never received bandit updates.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(router): drop unnecessary hybrid docstrings

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(router): attribute adaptive feedback

Credit user reactions to the model that produced the previous response while keeping current-response signals on the serving model

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(router): tune hybrid cold defaults

Use the cost-weighted policy that beat equal-pool complexity in the full bakeoff, and make the committed harness compare identical tier pools

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(router): preserve hybrid cold quality floor

Sample only unobserved models in the classified tier until feedback exists, then apply adaptive scoring without mis-penalizing models shared across tiers

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(router): bound feedback context cache

Cap retained session feedback so unique session IDs cannot exhaust router memory

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(router): preserve exhaustion signals

Include tool-result exhaustion in adaptive feedback and clear strict lint regressions blocking CI

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(router): remove stale owner cache

Remove obsolete attribution state, tighten the embedded router type, and keep the test diff focused on adaptive behavior

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(router): centralize hook cleanup

Use the callback manager to discover and remove adaptive hooks across every registered callback list

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(responses): continue MCP gateway tool turns from the final response and surface failures

When a /responses request uses a hosted MCP tool (server_url: litellm_proxy/<label>)
with store=true and the model calls a tool, the gateway auto-executes the tool and
streams one logical response stitched from several upstream responses: an interim
response whose only output is the function_call, then the post-tool answer

B1 (correctness): every streamed event was pinned to the first round's response id,
i.e. the interim response that carries the function_call but no tool output. The
client then continued the next turn from that dangling response and the provider
rejected it with "No tool output found for function call <id>", which on the
streaming path surfaced as a silent empty completion. The fix adopts each
auto-execute round's own response id (the cached id is reset when a follow-up round
starts) so the client continues from the final round, whose stored input chain
includes the function_call_output

B2 (robustness): initial and follow-up call failures were swallowed; the stream
emitted the mcp_list_tools discovery events and then closed with HTTP 200 and no
output and no error. The fix stashes the failure, makes the initial call eagerly in
aresponses_api_with_mcp so a pre-stream failure re-raises as a real 4xx before any
SSE bytes are written, and emits a terminal error event when a follow-up call fails
mid-stream

Adds regression tests covering continuation exposing the final round's response id
rather than the interim tool-call id, a follow-up failure emitting a terminal error
event, and an initial-call failure being stashed for eager re-raise

* ci(ui): report only error-level knip findings in CI (#32971)

* feat(batches): track cost for unmanaged Bedrock batches, generalize the flag (#32315)

* feat(batches): track cost for unmanaged Bedrock batches, generalize the flag

CheckBatchCost skipped Bedrock batches whose unified_object_id is a raw
model-invocation-job ARN, the same root cause previously fixed for
unmanaged Vertex batches. Bedrock batches embed the model name in their
s3:// input file name instead (litellm-bedrock-files-{model}-{uuid}.jsonl),
so the same routing mechanism now derives the model from that layout and
matches it to a configured bedrock deployment.

track_unmanaged_vertex_batch_cost is renamed to track_unmanaged_batch_cost
since two providers now share this mechanism.

* fix(batches): parse Bedrock batch output and price with deployment model name

Bedrock model-invocation-job results use modelOutput/error rows and short
internal model ids that are not in the cost map, so unmanaged batch cost
tracking logged tokens but $0 spend. Use deployment model name for pricing
and add regression tests.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(guardrails): walk custom_tool_call_output items in _content_utils (#32969)

* fix(guardrails): walk custom_tool_call_output items in _content_utils

* Change _OUTPUT_ITEM_TYPES to Frozenset type

* fix(guardrails): use builtin frozenset generic for _OUTPUT_ITEM_TYPES annotation

Frozenset is not a defined name (typing exports FrozenSet, the builtin is
frozenset), so module import raised NameError and broke every proxy test
suite. The builtin generic is valid on the supported python floor (3.10)
and keeps the UP006 ruff-strict budget at its ceiling, which the typing
alias would exceed

* fix: show and allow editing team model aliases after team creation (#33047)

* refactor(ui): rename OldTeams component file to Teams

* fix: show and allow editing team model aliases after team creation

* fix(ui): mark team model_aliases as nullable to match the prisma schema

* fix(ci): bump pillow to 12.3.0 to resolve osv-scan CVEs (#33093)

* fix(proxy): track unauthenticated pass-through requests in spend logs (#32410)

Pass-through endpoints configured with auth=false reach the cost-tracking callback with no key/user/team/end-user, so _should_track_cost_callback returned False and the spend-log write was skipped, leaving the request out of request/usage logs. Track pass-through call types even when unauthenticated so the SpendLog row is still written.

Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(lasso): send source.type=litellm for Used By attribution (#33090)

Co-authored-by: Or Gershoni <org@lasso.security>

* feat(mcp): generalize the bridge envelope identity to a key_hash or user_id subject

The scripted two-header client mints under a virtual key it presents at the token
endpoint (key_hash), but the interactive DCR client authenticates via SSO at the
bridged authorize, which yields a user, not a key. Make EnvelopeIdentity a
discriminated subject (subject_type key_hash | user_id) with key_hash_identity /
user_identity constructors, and dispatch admission on it: a key_hash reloads the
key, a user_id reloads the user and admits them as themselves (user-level budget
and SCIM enforced via the same centralized gate; no team bound, since a user
belongs to many teams or none). The interactive producer that mints a user_id
envelope lands in the follow-up commit.

* feat(mcp): interactive SSO sign-in for dcr_bridge oauth_delegate DCR clients

Completes the oauth_delegate bridge for real DCR clients (Claude Code, Claude
Desktop), which send no litellm key and cannot use the scripted two-header path.
On the short-circuit bridge arm the gateway now captures the SSO-authenticated
litellm user from the browser session at /authorize and seals it into the OAuth
state; at /callback it seals that user plus the upstream code into a gateway
authorization code the client echoes back; at /token it recovers the user,
exchanges the real upstream code, and mints a user-subject envelope. The user
identity captured in the browser thus rides to the back-channel token call with
nothing stored server-side, and admission opens the envelope under that user. The
scripted key_hash path is unchanged (raw upstream code, key from the request);
without a session the browser is sent through login first.

* fix(mcp): classify the user-subject reload's errors like the key path (503 outage, 401 missing)

_reload_admitted_user mirrored only part of _reload_admitted_key's error contract: it caught
ProxyException and HTTPException but had no arm for anything else, so a transient DB outage surfaced as
an opaque 500 instead of the retryable 503 the key path guarantees, and a missing user surfaced as a 500
too. The missing-user case is the subtle one: get_user_object raises a bare Exception for a deleted user
(not a ProxyException like get_key_object does for a missing key), so the ProxyException/HTTPException
clause never caught it and the user_object-is-None branch it was supposed to hit is unreachable on the
production path.

Add the same except-Exception arm the key path uses, with the one deliberate difference the differing
get_user_object contract requires: a database-service-unavailable error still raises the retryable 503,
while a missing user or any other non-outage resolution failure fails closed as a 401 rather than
propagating as a 500. The regression tests now drive the real behavior (get_user_object raising) rather
than a None return that never happens in production, and cover both the 503 outage and the 401
missing-user paths.

* fix(mcp): admit a user-subject envelope with the user's own MCP object permission

_reload_admitted_user returned a bare UserAPIKeyAuth(user_id=...), so the shared
get_allowed_mcp_servers found no key/team/object-permission grants and an interactive SSO client could
admit successfully yet see zero tools on a normal (allow_all_keys=False) server. The key path returns
the full key record whose object permission drives that computation; the user path dropped it.

Resolve the user's own MCP object permission and put it on the returned auth, so the same
get_allowed_mcp_servers the key path uses grants the user their litellm-granted servers and access
groups. This reuses get_object_permission (the id-to-grants resolver keys and teams already use) and
does not duplicate any permission logic; get_user_object does not load object_permission, so it is
resolved from the user's object_permission_id the same way the key and team paths do.

Only the user's own object permission is bound. A UserAPIKeyAuth carries a single team_id while a user
may belong to many teams, so team-inherited MCP grants for a user are a follow-up: they need a
many-teams union get_allowed_mcp_servers does not do off one auth object, and faking one here would be
the kind of half-measure that spawns more bugs. Tests cover the user's object permission riding onto the
admitted auth, and the existing admit/SCIM/missing-user/503 cases still hold.

* fix(ui): respect litellm_key_header_name in BYOK credential save and workflow runs fetches (#33103)

* refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant (#33040)

* feat(ui): rebuild the Virtual Keys table on the shared DataTable (#32991)

* feat(ui): rebuild the Virtual Keys table on the shared DataTable

Replaces the hand-rolled Tremor table and bespoke toolbar/pagination on the admin
Virtual Keys page with the shared DataTable: server-side sort, paginate, and
filter, a sticky scrolling body, a search plus column-visibility plus filters
toolbar, a right-side filter drawer, and a rows-per-page footer. A page header
with the existing key icon carries the Create New Key action.

Adds reusable, shadcn-default building blocks for the tables migrating onto the
DataTable next: shared IdentityCell, ModelsCell, and SpendBudgetCell in
shared/table_cells, plus a shared PageHeader. The models cell reveals overflow in
a hover tooltip and the spend/budget cell uses the Meter primitive.

All data and domain logic is preserved, including the useKeys query, team and org
alias resolution, the user popover, and the KeyInfoView detail swap. The rich
async Team/Org/Alias filters move into the drawer, and the toolbar search maps to
the key-alias substring search. Status now also reflects key expiry alongside
blocked and SCIM-blocked.

The VirtualKeysTable tests are updated to the new markup and extended with focused
coverage for each new shared cell

* fix(ui): address Virtual Keys redesign review feedback

Fold the status badge into the clickable Key cell and drop the separate Status
column so a key's alias, secret, and status read as one unit. The Key cell is
now the single click target that opens the key detail; the whole-row click is
removed

Migrate the filter drawer off AntD to shadcn. A new Combobox composed from
Popover and Input backs the Team, Organization, and Key Alias filters, keeping
search and the alias infinite-scroll

Show $0.00 for zero spend instead of a hyphen, and extend the shared DataTable
with badge, chips, and meter skeleton shapes so the loading state matches the
loaded cells (status pill, model chips, spend meter) rather than uniform bars

Fix key sorting: the Key column sent its column id "key" as sort_by, which
/key/list rejects with 400. It now sorts by the backend field key_alias

* fix(ui): use the shadcn base combobox and refine the keys filters and skeletons

Replace the hand-rolled filter combobox with the supported shadcn Base UI combobox
(ui/combobox, added via the CLI and reused through a small SearchSelect wrapper).
Its vended input-group and textarea deps are written for React 19 (plain functions
with ref-as-prop); this app is on React 18, where those subcomponents drop the refs
Base UI passes for focus and anchoring, so InputGroupInput, InputGroupButton, and
ComboboxTrigger are adapted to forwardRef. Those ui/ files now diverge from the
registry, and a future shadcn add would overwrite the adaptation until the app moves
to React 19. Adds class-variance-authority, which input-group needs

Give loading skeletons a per-column renderSkeleton escape hatch on the shared
DataTable and mirror the Key cell exactly (alias line, secret, status pill), so
skeleton rows match the real rows instead of being shorter and simpler

Resolve the automated review: the toolbar search and the drawer Key Alias filter
both mapped to the key-alias query, so the search silently overrode the drawer value
while its chip stayed visible. Consolidate to a single alias search in the toolbar
(placeholder now "Search by key alias…") and drop the redundant drawer field. Re-add
coverage for the Created By column's alias-over-email display

Refine the Team and Organization filters: they match on name and id, so the labels
read "Team" and "Organization" rather than "... ID", each option shows the name with
the id on a muted second line instead of "name (id)", and the active-filter chip
shows the friendly name

* chore(ui): drop duplicate class-variance-authority, use the repo cva package in input-group

* fix(mcp): relay upstream OAuth token and DCR rejections instead of a generic 500

An upstream token endpoint rejection (e.g. Google requiring client_secret even for PKCE web clients) escaped exchange_token_with_server as a raw httpx.HTTPStatusError, which the global exception handler turned into an opaque 500 Internal server error in the create-flow UI. The RFC 6749 section 5.2 error body the IdP sent (error, error_description, error_uri) is now relayed with the upstream's own 400/401 status; rejections outside the section 5.2 contract map to 502 so a broken upstream is not misattributed to the caller. The same relay covers the non-bridge DCR registration arm, and a 200 token response without a usable access_token now answers 502 instead of a KeyError 500. The catch wraps the post call itself because litellm's AsyncHTTPHandler raises MaskedHTTPStatusError at call time, which also made the pre-existing bridge-relay status check unreachable in production. The dashboard's token exchange error message now composes error and error_description so the form shows the IdP's reason

* refactor(mcp): drop bridge relay status check made unreachable by the unified relay

The try/except around the registration post now relays every upstream 4xx/5xx for both arms, so the bridge_relay status_code check could never fire; removing it addresses the Greptile P2 dead-code finding

* fix(mcp): classify get_user_object's wrapped DB outage across the exception chain

get_user_object catches every DB failure in a broad except and re-raises a bare ValueError (litellm/proxy/auth/auth_checks.py), so a real outage and a missing user look identical and the original error survives only as __context__. The dcr_bridge admission path keyed its 503-vs-401 decision on the exception type, so a transient outage during a user-subject reload surfaced as a 401 rather than a retryable 503, and the regression test injected a raw ConnectionError, a shape get_user_object never produces, so it passed on a fiction

Add PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain, which walks __cause__/__context__ (bounded and cycle-safe) the PEP 3134 way, and route _raise_503_if_db_unavailable through it. Move the user's object_permission resolution inside the single classified try so an outage there is a 503 too, never an opaque 500. Pin get_user_object's wrapping with a contract test that drives the real function, and drive the reload tests with that same faithful shape so a chain-blind regression fails them

* fix: redact async complete streaming response for custom callbacks (#33106)

* fix response not being redacted for custom callbacks with streaming enabled

* reduce code duplication

* add unit test

* fix: resolve lint violations in adopted redaction fix

* fix: scope streaming response redaction to the opted-out custom logger

---------

Co-authored-by: Moritz Müller <moritz.mueller2@tu-dresden.de>

* build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1 (#33041)

* refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant

* build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1

* fix(ui): address Virtual Keys redesign review nits (#33112)

* fix(ui): address Virtual Keys redesign review nits

Restore sorting by budget on the merged Spend / Budget column. The column now
uses a new DataTableMultiSortHeader whose chevron opens a menu offering Spend
and Budget in both directions plus Reset, so the progress-bar cell stays merged
while the sort field becomes an explicit choice. Sorting is server-side, so the
chosen field id (spend or max_budget, both accepted by /key/list) flows straight
through as sort_by

Fill the DataTable to its container width when column resizing is on. The table
width was pinned to the sum of column widths, so hiding columns left an empty
gutter on the right. It now keeps that width as a minimum and stretches to 100%
on underflow while still scrolling on overflow, which also covers the same gap
in TeamVirtualKeysTable since both share the component

Drop the dark background box behind the page-header icon so the Virtual Keys
header reads like the Teams header, and pull the 4-line inline filter lambda in
SearchSelect out into a named matchesQuery helper

Extends the DataTable and VirtualKeysTable tests to cover the new multi-field
sort menu (field id maps to sort_by, active indicator, reset) and the
fill-to-container width

* fix(ui): emphasize the active field in the Spend / Budget sort header

The merged Spend / Budget header always read "Spend / Budget" regardless of
which field drove the sort, so after picking Budget descending there was no way
to tell what was sorted without reopening the menu. The header now builds its
label from the sort fields and emphasizes whichever one is active (bold,
full-strength text) while muting the other, so the sorted column reads at a
glance alongside the direction chevron. Drops the now-redundant title prop since
the label is derived from the fields

* fix(ui): remove w-full so the keys page content stops overflowing by 32px

The virtual keys content wrapper used "w-full mx-4", which sets the width to
100% of the parent and then adds 16px of horizontal margin on each side, so its
margin-box came to 100% + 32px and overflowed the scrollable main region by
exactly 32px. That surfaced as a horizontal scrollbar along the bottom of the
whole content area, under the pagination. A block div is already full-width, so
dropping w-full lets mx-4 inset it correctly with no overflow

* fix(ui): darken the clickable Key cell on hover so it reads as clickable

The Key cell was the click target that opens the key detail, but hovering only
faded the chevron in with no change to the cell itself, so there was no cue that
the area was clickable. Give the cell a subtle muted background and a pointer
cursor on hover. The button spans the full cell (a negative inline margin plus a
matching width offset so the hover fill reaches both cell edges while the title
stays aligned with the other columns)

* fix(openai/responses): clamp max_output_tokens below API minimum (#33098)

* fix(openai/responses): clamp max_output_tokens below API minimum

Claude Code sends a max_tokens=1 warmup probe when running /model, which
the Anthropic Messages -> Responses adapter forwards as max_output_tokens=1.
OpenAI's Responses API rejects values below 16, so the probe failed with a
400. Clamp anything below the minimum up to 16 in map_openai_params so all
Responses API entrypoints (direct, chat->responses, anthropic->responses)
are covered.

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* refactor(openai/responses): extract _enforce_min_max_output_tokens helper

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(prometheus): read v3 rate limiter remaining values for per-key model gauges (#33119)

* fix(ui): drop w-full from page-content wrappers to remove 32px horizontal overflow (#33118)

Several dashboard pages wrap their content in a div styled w-full mx-4, so the
element's width is 100% of the scrollable main while mx-4 adds 16px of margin on
each side. That makes the margin-box 100% + 32px wide, which overflows main by
exactly 32px. Because main uses overflow-y-auto its overflow-x computes to auto,
so the overflow surfaces as a horizontal scrollbar along the bottom of the whole
content area under the pagination

The wrapped block is already full width without w-full, so removing that one
token keeps the layout and drops the overflow to 0. This is the same fix already
applied to the Virtual Keys page in #33112, extended to the remaining pages that
share the wrapper: Models + Endpoints, Tag Management, Organizations, Vector
Stores, AI Hub, and Logging & Alerts

* refactor(ui): migrate straightforward value debounces to react-pacer (#33042)

* refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant

* build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1

* refactor(ui): migrate straightforward value debounces to react-pacer

* feat(mcp): client-held refresh envelope for the dcr_bridge oauth_delegate flow

A dcr_bridge oauth_delegate access envelope is capped at one hour, and until now the mode had no refresh
at all: when the envelope expired the client had to re-run the interactive authorization_code flow. This
adds a second client-held credential, the refresh envelope, so the client renews on a back channel and
only re-authenticates when the refresh envelope expires or the upstream refresh token dies.

The refresh envelope is a distinct llm_refresh_ credential that seals only the upstream refresh token
(never the access token) bound to the same litellm identity and MCP server as the access envelope, under
the same master-key-derived keys, with nothing stored server-side. Both envelopes now carry a signed
kind claim ("access" or "refresh") that open() requires to match, so a refresh envelope can never open as
an access credential even if its wire prefix is swapped (the prefix is not signed; the claim is). A
refresh envelope presented at the MCP tool-call edge is not an access envelope, so admission fails it
closed the same way it already fails any non-access bearer.

At the token endpoint the authorization_code mint now returns a refresh envelope alongside the access
envelope whenever the upstream returned a refresh token, and the refresh_token grant is supported for
bridge servers: the client presents its refresh envelope, the endpoint opens it, re-validates the sealed
litellm key so a revoked key cannot keep refreshing, unwraps the real upstream refresh token, exchanges
it with the upstream IdP, and returns a fresh access envelope. Because the endpoint re-seals a refresh
envelope only when the upstream returns a new refresh token, the design mirrors the upstream's own
rotation policy rather than reinventing it: with a rotating upstream the client rotates and reuse is
detected upstream; with a non-rotating upstream the original refresh envelope stands until its bounded
14-day TTL. Both preconditions and the unwrap run before the exchange, so a rejected refresh never
consumes or rotates an upstream token.

The pure envelope and credential layers stay side-effect free: mint/open share one signing, size, and
kind gate across both envelope kinds, and every failure is a value. Tests cover the refresh round-trip,
the kind-claim and server-id bindings, the revoked-key gate, upstream rotation carried through, the
unwrap sending the real upstream token upstream, and edge rejection of a refresh envelope; the three
security bindings are mutation-checked. Limitation documented in the PR: gateway-enforced refresh
rotation with reuse detection would require server-side state, which this zero-custody mode omits by
design, so the refresh envelope inherits the upstream's rotation posture plus gateway identity binding
and a bounded TTL.

* fix(mcp): reject a refresh envelope explicitly at the tool-call edge

The live proof showed a refresh envelope presented at the MCP tool-call edge was rejected, but through
the generic oauth2 arm ("expected a virtual key starting with sk-") rather than the bridge arm, because
the admission routing gate is_bridge_envelope_shaped matched only the access prefix. The rejection was
already fail-closed and never forwarded anything upstream, but the path was imprecise and the unit test
modelled a route the real router did not take.

Match either envelope kind in is_bridge_envelope_shaped so the bridge arm engages for a refresh envelope
too, and have resolve_bridge_envelope return BridgeEnvelopeInvalid for it: a refresh envelope is a valid
gateway credential but only ever presented back to the token endpoint, never usable to authenticate a
tool call. Admission now fails it closed with the bridge arm's own 401 ("Invalid or expired
credential"), live-verified, with the upstream never touched. is_bridge_envelope_shaped has a single
caller (the admission routing gate), so the change is contained.

* fix(mcp): SecretStr the unwrapped refresh token, drop the dead request arg, fail closed on a missing user

Three review findings on the refresh path, addressed at the root:

_BridgeRefreshReady.upstream_refresh_token was a plain str, the one credential in the envelope/bridge
layer that escaped the SecretStr discipline every other one follows (RefreshCredential.refresh_token,
UpstreamTokenGrant.access_token, EnvelopeKeys.signing_key). A repr or a traceback capturing a local
_BridgeRefreshReady would have logged the raw upstream refresh token. It is now a SecretStr, carried as
the SecretStr open_bridge_refresh_envelope already returns and unwrapped only at the point the exchange
builds the upstream request body.

_prepare_bridge_refresh took a request it never read; on the refresh path identity comes entirely from
the sealed envelope, not the HTTP request, so the parameter was dead and misleadingly implied it read
from the request the way the authorization_code prepare does. Removed, and the caller updated.

_reload_active_user_by_id misclassified a missing user as unresolvable (500). This is the same root
cause as the admission user-reload fix: get_user_object raises a bare Exception for a deleted user
rather than a ProxyException, so its except-Exception arm must fail closed to no_active_key (which the
refresh path maps to invalid_grant) for anything that is not a database-service-unavailable outage,
rather than treating a missing user as an opaque gateway fault. Regression tests cover the missing-user
and DB-outage classifications directly.

* fix(mcp): make the dcr_bridge refresh path fail correctly on outages, dead tokens, and revoked owners

Four fixes to the refresh_token grant for dcr_bridge oauth_delegate, surfaced by an adversarial pass over the exchange path

Route the user-subject re-validation's outage check through the chain-aware classifier, so a transient DB outage (which get_user_object wraps in a bare ValueError) reports as unavailable (a retryable 503) rather than collapsing to no_active_key and an invalid_grant, matching how admission now handles the same wrapper

When the upstream reports its own refresh token as already elapsed (refresh_expires_in non-positive), do not seal it into a full-TTL refresh envelope; return no refresh so the exchange degrades to an access-only response, mirroring how the access grant refuses an already-elapsed access token instead of capping it

When the upstream rejects the sealed refresh token with 400 invalid_grant (revoked or expired at the IdP), return an RFC 6749 invalid_grant response so the OAuth client re-runs authorization_code, rather than surfacing the opaque upstream error it cannot act on

Gate key-subject renewal on the owner's SCIM state, mirroring admission's _reject_if_admitted_owner_scim_deactivated, so an offboarded user cannot keep refreshing a still-active key; the check fails open on a missing owner or a DB blip so a key that outlives its owner record does not get wrongly revoked

Each fix has a mutation-checked regression test

* test(proxy): add regression tests for management_endpoints edge cases (#32976)

Mutation testing surfaced branches in cost_tracking_settings and common_utils that the suite executed but never asserted on. Pin those behaviors with targeted tests: the returned (model, provider) from _resolve_model_for_cost_lookup for deployments carrying a custom_llm_provider and for deployments missing the litellm_params / model_info keys, plus the exact error-response bodies, the caller-identity lookup arguments, and the member and guard branches in common_utils.

* fix(auto-router): correct Responses API tool_choice shape and propagate alias litellm_params (#32974)

* fix(anthropic-messages): send bare-string tool_choice to Responses API, propagate router-alias litellm_params

The Anthropic /v1/messages -> Responses API adapter always wrapped
tool_choice in an object ({"type": "auto"}, {"type": "required"}), but
the Responses API's tool_choice schema for these cases is a bare
string ("auto"/"required"/"none"). Sending the object shape to an
OpenAI-compatible backend (e.g. vLLM) fails Pydantic validation with a
400. The "none" case also fell through to "auto" instead of mapping to
"none".

Separately, litellm_params configured directly on a router-alias
deployment (auto_router/complexity_router, adaptive_router,
quality_router, or semantic auto_router) - e.g.
cache_control_injection_points, drop_params - were silently dropped
for every request through that alias. async_pre_routing_hook swaps
`model` from the alias name to the selected tier/route's model before
the deployment lookup runs, so the outbound call only ever merged in
the tier deployment's own litellm_params, never the alias's. Register
non-routing-config litellm_params from the alias deployment and apply
them to the request whenever a pre-routing hook substitutes the model.

* fix: satisfy ruff-strict-budget UP006 and router coverage checker

Use builtin dict[...] generics instead of typing.Dict for the two new
annotations introduced in the previous commit, since they pushed
UP006 over the codebase ceiling in ruff-strict-budget.json. Add a
direct unit test for _register_pre_routing_alias_overrides so the
text-based router_code_coverage.py checker sees it exercised by name.

* fix(router): replace alias-param denylist with a tight allowlist

_PRE_ROUTING_ALIAS_RESERVED_PARAMS excluded router-init-only keys from
the alias's litellm_params before forwarding the rest as request
kwargs, but GenericLiteLLMParams also holds deployment-management
fields (tpm, rpm, weight, tags, max_budget, budget_duration,
use_in_pass_through, litellm_credential_name, ...) on the same object.
Any of those left off the denylist would get silently forwarded as if
they were request kwargs.

Replace the denylist with a tight allowlist of exactly the two
request-shaping params this feature exists for - drop_params and
cache_control_injection_points - so unrelated management fields never
reach the outbound call regardless of what else GenericLiteLLMParams
grows to hold.

* fix(router): re-register adaptive-alias overrides on set_model_list reload

set_model_list() unconditionally clears pre_routing_alias_overrides on
every call (e.g. /config/reload), but _finalize_adaptive_router_if_configured()
skips rebuilding an AdaptiveRouter whose model_name already exists in
self.adaptive_routers - so _register_pre_routing_alias_overrides() never
ran again for an auto_router/adaptive_router alias after a reload,
silently dropping its drop_params/cache_control_injection_points.

Build the Deployment unconditionally and re-register its overrides even
on the skip-existing-router path; only the (expensive) AdaptiveRouter
construction itself stays skipped.

* style: ruff format after merging litellm_internal_staging

* fix(router): drop the alias-param allowlist, exclude only model

Per review discussion: instead of a router.py-local allowlist of exactly
which litellm_params an alias (auto_router/complexity_router,
adaptive_router, quality_router, semantic auto_router) can forward to
the request it routes, _register_pre_routing_alias_overrides now
forwards everything except `model` (the alias marker itself, e.g.
auto_router/complexity_router, never a real provider model).

Router-init-only fields (complexity_router_config,
complexity_router_default_model, auto_router_config,
auto_router_config_path, auto_router_default_model,
auto_router_embedding_model, adaptive_router_config,
adaptive_router_default_model, quality_router_config,
quality_router_default_model) now flow into request_kwargs unfiltered
too. That's safe because litellm.completion()/acompletion() already
strips anything in litellm.types.utils.all_litellm_params before
building the provider request - added these 10 keys there, alongside
the deployment-management fields (tpm, rpm, weight, ...) already listed.
Verified live: without that addition, complexity_router_config lands in
extra_body and ships raw to the provider; with it, it's stripped.

This moves the "which fields aren't real LLM params" list from a
router.py-local allowlist to the single existing global list every
completion() call already depends on, instead of maintaining two.

* refactor(router): look up alias litellm_params on demand instead of caching them

_register_pre_routing_alias_overrides cached each alias's litellm_params
into self.pre_routing_alias_overrides at deployment-init time, which
required keeping that cache in sync with set_model_list() reloads - the
exact bug the previous adaptive-router-reload fix was patching around
(AdaptiveRouter survives a reload, but the cache didn't always get
refreshed to match).

Delete the cache and the registration method entirely. async_pre_routing_hook
now looks up the alias's own litellm_params directly from self.model_list
via self.model_name_to_deployment_indices at request time, the same
model_list that's already correctly rebuilt on every set_model_list()
call. No second piece of state to invalidate, so the reload staleness
bug class isn't possible anymore, and it's less code than before.

* fix(mcp): keep out-of-contract upstream error bodies out of client responses

The token and DCR relays serve unauthenticated OAuth clients, so only the RFC 6749/7591 error fields may cross the trust boundary. A rejection body outside those contracts (HTML error page, proxy banner, stack trace) is now logged server-side, bounded, and the client response names only the upstream status. Addresses the Veria information-exposure finding

* fix(mcp): re-request the sealed scope on a bridge refresh when the client omits it

The refresh envelope seals the upstream scope as the scope to re-request (RefreshCredential), but _prepare_bridge_refresh dropped it, unwrapping only the refresh token, and the exchange added scope to the upstream request only from the client's HTTP form. A DCR/MCP client typically omits scope on refresh, so the sealed scope was never sent and a stricter upstream could narrow or drop the renewed token's scope

Thread the sealed scope through _BridgeRefreshReady.upstream_scope and fall back to it when the client sends none; a client-supplied scope still wins, which RFC 6749 section 6 bounds to the original grant. The regression test drives a refresh where the client omits scope and asserts the upstream POST carries the sealed scope, mutation-checked against both the drop and the fallback

* fix(ui): render the sidebar scrollbar with shadcn ScrollArea (#33124)

* fix(ui): render the sidebar scrollbar with shadcn ScrollArea

The sidebar navigation scrolled through a native overflow-y-auto container, so the browser drew its default scrollbar. It now scrolls through the shadcn ScrollArea primitive so the thumb matches the rest of the dashboard

Switching to ScrollArea surfaced a latent styling gap. The Base UI scroll-area, tabs, and separator primitives rely on data-horizontal and data-vertical Tailwind variants that resolve to [data-orientation="horizontal"] and [data-orientation="vertical"], and those variants ship in shadcn's shared stylesheet. The project never imported it, so the classes matched nothing and the scrollbar collapsed to zero width. This adds shadcn as a devDependency and imports shadcn/tailwind.css, which also repairs the vertical tabs and separator styling. See shadcn-ui/ui#9196 for the upstream tracking issue

* refactor(ui): inline the Base UI data-* variants, drop the shadcn dep

The earlier fix imported shadcn/tailwind.css through the shadcn devDependency, which pulled 219 packages and tied the CSS build to shadcn's package exports (an open Turbopack-breaking bug, shadcn-ui/ui#10931). shadcn's model is that we own the components, so the custom variants those components depend on belong in our own stylesheet rather than a runtime dependency. This inlines the nine data-* custom variants and the no-scrollbar utility that the Base UI primitives reference into globals.css, and removes the shadcn package.

* refactor(ui): migrate callback debounce sites to react-pacer with regression tests (#33043)

* refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant

* build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1

* refactor(ui): migrate straightforward value debounces to react-pacer

* refactor(ui): migrate callback debounce sites to react-pacer with regression tests

* chore(ui): restore trailing newline in eslint-suppressions.json

* test(ui): mock all pacer debounce hooks in VirtualKeysTable test

* fix(ui): update merged debounce tests for OldTeams to Teams rename

* fix(mcp): carry the requested scope forward when the upstream omits it on a bridge refresh

The prior fix sent the sealed scope on a refresh, but the re-minted refresh envelope re-seals scope from the upstream response, and RFC 6749 section 5.1 lets an upstream omit scope when it is unchanged. So after one refresh whose response omitted scope, the new envelope sealed scope=None and every subsequent refresh dropped it, letting a stricter upstream narrow the renewed token

When the upstream omits scope on a bridge refresh, seal the scope we requested (which RFC 6749 section 5.1 defines as the granted scope when omitted) into the renewed access and refresh envelopes, so the scope survives the whole refresh chain. The regression test refreshes against an upstream that omits scope, asserts the new refresh envelope still carries it, and refreshes again off that envelope to prove the chain does not lose it, mutation-checked

* refactor(mcp): classify upstream OAuth faults once and derive status, code, and prose from the value

Replaces the accreted relay helpers with a faults package (types, classify, render_oauth): every
upstream token/DCR rejection is classified into exactly one fault value and the response status,
wire error code, and prose are all derived from that value, so a caller-fault code can never ship
on a server-fault status (the bugbot finding on invalid_grant over a 500). Classification takes the
credential source into account: invalid_client and friends against the server's stored credentials
are the operator's fault and render as 502 server_error with gateway-authored prose while the IdP's
prose stays in server logs; the same codes against caller-supplied credentials relay on the status
the code implies. Classifiers are total, so an unreadable rejection body (lying content-encoding,
unconsumed stream) yields the same 502 fault instead of resurrecting the opaque 500 (the second
bugbot finding); DCR rejections normalize to 400 per RFC 7591 regardless of the upstream's status

* style(mcp): unquote annotations and use PEP 604 unions in the faults package

* fix(mcp): detect upstream invalid_grant by the RFC 6749 error field, not a body substring

The bridge refresh path decided whether an upstream token-endpoint rejection was invalid_grant by substring-matching the raw response body, so a rejection whose actual error is something else but whose error_description merely contains the string invalid_grant would false-match, map to invalid_grant, and trigger a needless authorization_code re-run

Parse the RFC 6749 section 5.2 error object and compare the error field. A non-JSON body, or an error that is not invalid_grant, now propagates as the upstream error rather than being reinterpreted. The regression test drives an invalid_client rejection whose description contains the string invalid_grant and asserts it is not mapped, mutation-checked against the substring match

* fix(mcp): keep upstream self-blame codes and gateway capability gaps off the caller

Extends the fault matrix per review: server_error and temporarily_unavailable are codes by which the
upstream blames itself, so they classify as a new UpstreamReportedFault arm rendering 502/503 with a
matching wire code instead of a 400 that blames the caller; invalid_target is a gateway capability
gap (RFC 8707 resource indicators, LIT-4339) and is gateway-blamed regardless of whose credentials
were presented; the DCR classifier shares the same blame assignment. The gateway-fault arm is renamed
GatewayRejected since it now covers capability gaps as well as stored-credential rejections

* chore: add CODEOWNERS for ui and proxy UI build artifacts (#33131)

* feat(ui): rebuild the Teams table on the shared DataTable (#33128)

* feat(ui): rebuild the Teams table on the shared DataTable

The Your Teams tab moves off the Ant Design table onto the shared DataTable that the Virtual Keys page uses, following the new dashboard design. It gains server-side sort, pagination and filtering, a toolbar with a filter drawer and a columns menu, and a per-row actions menu

Sorting is wired only to the columns /v2/team/list can actually order by (team_alias, created_at); Spend / Budget and Updated stay unsorted because the endpoint silently ignores those fields. The design's "Created by" column is dropped since the team object has no such field, and the drawer's "Has keys" filter is dropped for the same reason. The Resources cell shows members, models and keys as colored pills, and the actions menu keeps the existing Edit, Copy team ID and Delete behaviors, with Edit and Delete gated to Admin

The teams grid gets its own unit tests in TeamsPage/TeamsTable.test.tsx. Teams.tsx keeps the create-team modal, delete modal, detail view and tabs, now refreshing the list through React Query invalidation instead of a manual refetch

* fix(ui): match Teams loading skeletons to the rendered row height

The default twoLine and chips skeleton shapes rendered the Team and Resources cells shorter than the loaded row (a real row measures 55px, the old skeleton ~49px), so the loading state looked visibly squat. Give the Team column a custom renderSkeleton that mirrors the two-line IdentityCell (measured 54px) and the Resources column one that mirrors the pills, and mark the hidden Rate Limits column as two-line so it matches when shown

* fix(ui): keep team admins' Members tab by deriving is_team_admin from the selected team

The redesign computed is_team_admin from useTeam(selectedTeamId), but that hook returns teamInfoCall's nested { team_info: { members_with_roles } } shape, so the top-level members_with_roles read was always undefined and is_team_admin was always false. For a non-proxy-admin team admin that hid the Members, Member Permissions and Settings tabs in the team detail view, which broke the team-admin add/remove member e2e tests. Pass the Team object up from the table instead (/v2/team/list returns it with a top-level members_with_roles), matching the pre-redesign behavior; proxy admins were unaffected because is_proxy_admin already granted access

Also point the Delete-a-team e2e at the new kebab: open the row actions menu, then click Delete team, rather than clicking the old inline delete icon

* fix(keys): persist key_type so the UI shows correct key scope instead of "All Proxy Models" (#33115)

* fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models'

key_type is not persisted on a key (the proxy maps it to allowed_routes and
drops it), so the keys tables only inspected the models list and rendered
'All Proxy Models' for any key with an empty models array, including SCIM,
Management and Read-only keys that cannot call a single model.

Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a
scope tooltip for those recognized scopes; unrestricted, AI-API and custom
keys keep the existing model-list rendering.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(ui): move key_scope helper to components root

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(keys): persist key_type on virtual keys so the UI reads scope directly

Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy,
and proxy-extras schemas plus an additive migration) and stop dropping the
value in handle_key_type, so management/read_only/llm_api/default keys store
their type alongside the derived allowed_routes. Surface it on the key read
and create response models. The dashboard now prefers the persisted key_type
for the no-inference buckets and keeps the allowed_routes derivation as the
fallback for keys created before the column existed (key_type null), so no
backfill is required.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(keys): add key_type column to LiteLLM_DeletedVerificationToken

The deleted-token archive model inherits key_type from the verification
token, so regenerate/delete flows write key_type into
LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration)
so the archive insert does not fail with FieldNotFoundError.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN)

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(router): opt-in session affinity for complexity router (#33126)

* feat(router): opt-in session affinity for complexity router

Complexity router reclassified every turn, which could flip the routed
model group mid-session and break provider-side prompt caching. Add a
session_affinity config flag: when a session_id is resolvable, pin the
model chosen on the first turn and reuse it for the rest of the session,
skipping reclassification. Pinned turns still stamp the adaptive
bandit's chosen-model metadata so reward feedback keeps working when
adaptive=True.

* fix(router): refresh session-affinity TTL on hit, scope pin by API key

Two issues from review: the TTL was only set on the first classification,
so an active session outliving session_affinity_ttl_seconds silently lost
its pin instead of refreshing as documented. And the cache key was scoped
only by session_id, which is client-supplied and unauthenticated, so two
different callers reusing the same session_id could poison each other's
routing pin. Refresh the TTL on every cache hit, and namespace the cache
key by the proxy-derived API key hash when available.

* feat(prometheus): expose video duration and image count consumption metrics (#33138)

* test(e2e): otel trace completeness on /chat/completions (#33132)

* test(e2e): OTEL trace completeness on /chat/completions against a local Jaeger destination

Adds the logging-suite infrastructure for LIT-3787 trace-completeness coverage:
a jaeger service in the compose stack as the OTEL v2 destination (arize_phoenix
preset pointed at it via PHOENIX_COLLECTOR_HTTP_ENDPOINT, so gen-AI spans export
through a preset-owned provider - the code path where trace splits happen), a
typed Jaeger query read-back client, and the first test: one successful
non-streaming /chat/completions call exports ONE complete trace (root SERVER
span + auth/db/cost children + gen-AI CLIENT span, no dangling parents).

* test(e2e): harden the otel trace read-back per review

Jaeger reads now query server-side by the litellm.call_id span tag instead of
paging recent traces and filtering client-side; the compose stack's background
jobs alone can push a request trace past the page. A failed query hard-fails
instead of reading as an empty result, the settle predicate now also waits for
the prefix-matched db span the assertion demands, parent-chain walking follows
CHILD_OF references only, the zero-trace and split-trace failures get distinct
messages, jaeger gets a healthcheck so the depends_on condition is accurate,
and the chat docstring names the route the code actually asserts

* test(e2e): author the chat trace test docstring

* Update logging section in CLAUDE.md

Removed mention of OTEL trace-tree completeness from logging integration section.

* fix(sso): paginate through all pages when fetching service principal group assignments (#33149)

get_group_ids_from_service_principal only read the first page of the
Graph API appRoleAssignedTo response, so tenants with more than 100
groups assigned to the enterprise application silently lost group
memberships during SSO login. Loop over @odata.nextLink with the same
MAX_GRAPH_API_PAGES cap that get_user_groups_from_graph_api already
uses, and warn when the cap is hit.

Ported from #32792 by @saisurya237 so CI can run.

Fixes #32790

Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com>

* test(e2e): otel trace completeness on /v1/messages (#33133)

* test(e2e): OTEL trace completeness on /v1/messages

Extends the LIT-3787 trace-completeness suite to the Anthropic-native route:
one successful non-streaming /v1/messages call must land at the destination as
ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT
span, no dangling parents). Adds the raw /v1/messages sender to the logging
suite client.

* test(e2e): reuse the shared AnthropicMessagesBody per review

Drops the duplicate /v1/messages request model in favor of the one models.py
already provides (budget_client uses the same one), passes max_tokens at the
call site to match the sibling chat test, notes in the docstring why the
gen-AI span is named chat on this surface, and adopts the hardened read-back
signature

* test(e2e): author the messages trace test docstring

* test(e2e): declare the messages surface on the covers marker

* test(e2e): otel trace completeness on /v1/responses (#33134)

* test(e2e): OTEL trace completeness on /v1/responses

Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API
route: one successful non-streaming /v1/responses call must land at the
destination as ONE connected trace. Adds the raw /v1/responses sender, a
CHEAP_OPENAI_MODEL config constant, and registers responses in the otel
registry cell's exercised_on.

* test(e2e): author the responses trace test docstring

* test(e2e): declare the responses and chat surfaces on the covers markers

* feat(ui): add adaptive routing settings to Auto-Router v2 (#33146)

* refactor(mcp): extract the dcr_bridge token flow into bridge_token_flow.py

discoverable_endpoints.py had grown to 2695 lines mixing FastAPI route handlers with the dcr_bridge token-flow logic, against the no-monster-files convention. This moves the bridge token flow (the litellm-key/user resolution, the SCIM revalidation gate, and the mint/refresh envelope logic with their types and error mappers) into a dedicated bridge_token_flow.py, leaving the route handlers and the shared exchange_token_with_server orchestrator in discoverable_endpoints.py importing from it

Pure relocation, zero behavior change. The moved code is byte-verbatim except one type annotation quoted as a forward reference (_BridgeAuthorizationCode is used only for typing and imported under TYPE_CHECKING to avoid a cycle), and the new module imports nothing from discoverable_endpoints at runtime. 275 tests pass unchanged; the test patch targets for moved internals were repointed to the new module and verified to still apply

* bump: litellm-enterprise 0.1.49 -> 0.1.50, litellm-proxy-extras 0.4.76 -> 0.4.77, litellm 1.93.0 -> 1.94.0 (#33229)

* chore(deps): pin httplib2 and setuptools transitive floors (#33233)

Raise the constraint floors for two transitive dependencies so resolution moves them to their latest maintenance releases: httplib2 0.31.2 -> 0.32.0 and setuptools 82.0.1 -> 83.0.0. Both are pulled in only by optional integrations (Google API client, grpc tooling, lunary observability, the nvidia-riva extra), all lower-bound only, so the floors stay inside every requirer's allowed range and a default install is unaffected

* feat(ui): left-anchor the Create Key and Create Team CTAs (#33248)

Move the Create New Key and Create Team buttons out of the page header's
right-side action slot. On Teams the button now sits in the tab bar's left
slot, separated from the three tabs by a vertical rule, so the CTA and tabs
read as one left-anchored cluster. On Keys, which has no tabs, the button
anchors left on its own row beneath the title.

* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models (#33244)

* fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models

* test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping

* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models

Narrow the fix to the temperature reconciliation; the reasoning_effort
budget cap is reverted because the live translation grid relies on
budget_tokens >= max_tokens to reject unsupported effort tiers
(xhigh/max) on budget-mode models, so capping turned those 400s into
200s.

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook (#33136)

* fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook

* fix(guardrails): keep request-body dispatch predicate unchanged

* fix(guardrails): fail closed when proxy extras are missing at deployment hook

* fix(proxy)!: enforce user budget on team keys (read-time + reservation) with UI opt-out (#32005)

* fix: enforce user budget on team keys

User budget was skipped when the key belonged to a team, letting
users exceed their personal budget by going through a team key.

Remove the team_object guard in _user_max_budget_check so user
budgets are always enforced. Add skip_user_budget_on_team_key
general_settings flag to opt back into the old behavior.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: update test to expect user budget enforcement on team keys

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): enforce user budget on team keys in reservation path and expose skip flag in UI

Extends the read-time fix so the optimistic budget reservation also reserves the user spend counter for team-scoped keys, register skip_user_budget_on_team_key in ConfigGeneralSettings so /config/field/update accepts it, and surface it as a Boolean toggle on the Admin UI General Settings table via allowed_args in /config/list.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: assert budget_exceeded ProxyException in personal budget test

Tighten the broad pytest.raises(Exception) so the test only passes when
the auth flow rejects with a budget_exceeded ProxyException, and switch
the new ConfigGeneralSettings field to Optional[bool] to match the
surrounding annotation style

* fix: revert to bool | None to stay under UP045 strict budget

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>

* fix(e2e): bound spend-log snapshots to a /spend/logs/v2 window (#33265)

The rate-limited batch spend test snapshotted unattributed rows via the
unpaginated /spend/logs whole-table read, which grows with the environment
(58MB on stage) and OOMKilled the e2e runner at its 512Mi limit on every
scheduled run. Gateway.spend_logs_window pages /spend/logs/v2 over an
explicit date window instead, and SpendLogsParams now rejects a filterless
read so the whole-table call cannot come back

* refactor: make the code easier to read

* feat(pricing): add gemini-omni-flash-preview with video output token pricing

* fix(gemini): map video response modality instead of MODALITY_UNSPECIFIED

* fix(anthropic): use native output capability (#33235)

* fix(anthropic): route native structured output

Use model capability metadata so new native structured-output models do not require transformation allowlist changes.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(anthropic): pass provider to capability

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(anthropic): cover dotted model IDs

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(anthropic): handle remote capability lag

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(ci): retry setup-uv installs to survive transient manifest fetch failures

* docs(e2e): add cache_hit to the naming grammar assertion vocabulary

* refactor(e2e): share anthropic cache-control shapes in endpoints_client

* fix(proxy): never log raw virtual keys in key insertion debug output (#33268)

* fix(proxy): never log raw virtual keys in key insertion debug output

* fix(proxy): tolerate None token in insert_data debug log redaction

* fix(auth): scope the JWT enterprise gate to actual JWTs (#33296)

With enable_jwt_auth enabled but no enterprise license (premium_user
False), the JWT premium check fired on every request before the token
was inspected, so the master key, sk- virtual keys, and the encrypted
CLI/UI SSO session token that `lite login` issues all 401'd with "JWT
Auth is an enterprise only feature" and were never decoded. That broke
`lite login`, `lite claude`, and the proxy master key on any deployment
that turned JWT auth on without a license.

Move the premium check inside the is_jwt branch so it gates only real
JWTs. Non-JWT credentials fall through to their own auth paths
regardless of license; actual JWTs still require premium, so the
enterprise gate is unchanged for the feature it protects.

* test(e2e): scope virtual keys to the deployment under test

* fix(s3): sanitize slashes in response-id-derived object key file name (#33271)

* refactor(ui): migrate guardrails table onto shared DataTable (#33303)

* feat(ui): migrate guardrails table onto shared DataTable

Move the guardrails list onto the shared DataTable + cell library as the
proof-of-concept for the simple-tables design migration, following the Teams
reference pattern.

Split the table into a thin container (guardrail_table.tsx) and column defs
(guardrailTableColumns.tsx): client-side sort defaulting to created_at desc, a
search + refresh toolbar, IdCell / DateCell / StatusBadge cells, real provider
logos, a rich empty state, and skeleton loading rows. Row actions move into a
per-row overflow menu; deletion stays disabled for config-file guardrails, now
surfaced as a disabled menu item instead of a greyed trash icon. Detail view
and the delete modal remain owned by GuardrailsPanel.

Restyle the "Add New Guardrail" control to the shared Button + dropdown menu.

Update the regression tests for the menu-based actions and drop the now-stale
eslint suppression entry that the rewrite eliminated.

* fix(ui): match guardrails table to the design

Address design-review feedback on the guardrails migration:

- Drop the search + refresh toolbar. The original table had neither and the
  SimpleTable design has no toolbar; the container now just renders the sorted
  table and its empty state.
- Give the Guardrail ID cell the design's hover affordance by rendering it with
  the shared IdentityCell (monospace, chevron on hover) instead of the blue
  IdCell pill.
- Stop pinning the actions column. Pinning added a sticky divider that the
  design and the Teams table don't have; it is now a plain right-aligned menu
  column, matching Teams.

* fix(ui): match loading skeleton row height to loaded rows

The compact skeleton row did not carry the h-8 height that real compact
rows get, so loading rows rendered shorter than loaded ones and the table
height jumped when data arrived. Mirror the same size-based height on the
skeleton row in the shared DataTable so every compact table loads at a
stable height

* test(ui): drop stale onGuardrailUpdated from guardrails table baseProps

The prop was removed from GuardrailTableProps when the toolbar went away;
the test baseProps still listed it. Harmless at the call site since it is
spread rather than an object literal, but dead and worth removing

* fix(ui): remove dead edit_guardrail_form after guardrails migration

The guardrails table migration dropped the last import of EditGuardrailForm,
which knip flags as an unused file. The form was already unreachable before
the migration: the table wired a delete button only, and nothing ever called
handleEditClick to open the modal, so the import was the sole thing keeping
the file referenced. Delete it and prune its now-stale eslint suppression
entry. Guardrail editing is unchanged and lives in the detail view
(GuardrailInfoView)

* feat(guardrails): streaming text transformation in generic_guardrail_api (#33110)

* feat(guardrails): support streaming text transformation in generic_guardrail_api

* chore(guardrails): address PR review feedback

* fix(guardrails): fail closed on tool-call and prefix-rewrite leaks in streaming transform

* fix(guardrails): address Bugbot review on streaming transform correctness

* fix(guardrails): coerce holdback in handler for in-process guardrails

* fix(guardrails): harden streaming transform (holdback coercion, tool-call passthrough, n>1 finish_reason)

* test(guardrails): targeted _mode_matches coverage for all guardrail_mode shapes

* fix(guardrails): inspect streamed tool calls and harden incremental_diff edge cases

* test: move ComplianceChecker mode tests to the compliance PR

* fix(guardrails): strip content from tool-call passthrough so streamed text can't bypass the transform

* fix(guardrails): four correctness fixes for incremental_diff streaming path

Four bug fixes on top of the OSS PR's incremental_diff streaming text
transformation, all inside the incremental_diff code paths only. No
existing block_only, non-streaming, or pre_call behavior is touched.

Fix #1 — Mixed content+tool_call finish_reason ordering
  _tool_call_passthrough_chunk now takes an optional finish_reason_per_choice
  map. For a choice carrying both delta.content and delta.tool_calls,
  finish_reason is stripped from the passthrough and recorded on the map so
  the final synthetic text chunk delivers it. Without this, SSE-compliant
  clients stopping at finish_reason drop the guardrailed text — defeating
  the redaction the whole feature exists for. (Greptile P1 twice, Veria.)

Fix #2 — Choice index sort in _process_streaming_transform
  indices/texts_to_check were derived from dict insertion order. For n>1
  streams where choice 1 emits before choice 0, guardrail-returned texts
  aligned to the input order mapped back to the wrong choice indices on
  write-back — wrong text goes to wrong choice. Sort raw_by_index.keys()
  up front so realignment is deterministic. (Bugbot Medium.)

Fix #3 — Cross-chunk pre-tool-call text flush
  With default streaming_sampling_rate=5, text chunks followed by a pure
  tool-call chunk carrying finish_reason='tool_calls' would emit the
  passthrough with finish_reason before any transformed text delta had
  fired. Same failure mode as fix #1 but cross-chunk. Now we flush any
  accumulated text via _round(is_final=False) BEFORE yielding the
  tool-call passthrough. (Greptile P1.)

Fix #4 — Terminator chunk for deferred finish_reason on empty mutated_text
  _build_transform_chunk returned None early when mutated_text_per_choice
  was empty. If a mixed content+tool_call chunk had deferred its
  finish_reason (via fix #1) and the guardrail then suppressed the text
  (empty return), the deferred finish_reason was never delivered. Now on
  is_final=True with empty mutated_text_per_choice, we emit a terminator
  carrying finish_reason per choice from finish_reason_per_choice.
  (Bugbot High.)

Also normalized Optional[X] → X | None across the OSS PR's added surface
via ruff UP045 autofix to keep the strict-rule gate within budget. Pure
mechanical typing style change, no semantic effect.

Regression tests for all four fixes:
- test_mixed_chunk_finish_reason_arrives_after_transformed_text (#1)
- test_text_flush_precedes_tool_call_passthrough (#3)
- test_final_finish_reason_flushed_when_guardrail_suppresses_text (#4)
- test_transform_sends_texts_sorted_by_choice_index (#2)

All fixes reachable only when streaming_transform_mode == 'incremental_diff'
is configured (via _run_incremental_transform_stream) or when a
StreamTransformSink is present (via _process_streaming_transform). Verified
scope-clean: no changes to block_only, non-streaming, pre_call, moderation,
or sibling guardrails.

---------

Co-authored-by: Marton Schneider <marton@schneider.co.nl>

* test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548)

* test(claude_code): move the Claude Code compatibility matrix under tests/e2e

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(claude_code): drop the CircleCI compat PR gate; the matrix runs in the scheduled e2e suite instead

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci: restore the upload-coverage job dropped by mistake with the compat gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(e2e/claude_code): print rate-limit summary on failed compat runs and fix stale run_daily.sh header comments

* test(claude_code): assert fine-grained tool streaming via input_json_delta instead of an event-count floor

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>

---------

Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
Co-authored-by: tin-berri <tin@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Shivam Rawat <shivam@berri.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Thibault Serot <thibault@linktr.ee>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Or Gershoni <org@lasso.security>
Co-authored-by: Moritz Müller <moritz.mueller2@tu-dresden.de>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Marton Schneider <marton@schneider.co.nl>
Co-authored-by: mateo <mateo@berri.ai>

.github/CODEOWNERS

@@ -0,0 +1,2 @@
+/ui/ @yuneng-jiang @ryan-crabbe-berri
+/litellm/proxy/_experimental/out/ @yuneng-jiang @ryan-crabbe-berri

.github/actions/detect-backend-changes/action.yml

@@ -0,0 +1,48 @@
+name: "Detect backend-relevant changes"
+description: >-
+  Classify the pull request's changed files with .circleci/scripts/classify_changes.sh
+  and expose decision=run|skip. decision=skip means only ui/**, **.md or **.mdx files
+  changed, so callers can short-circuit expensive steps while the job still completes
+  successfully and satisfies its required status check. The decision defaults to run for
+  any non pull_request event or whenever the changed set cannot be resolved, so tests are
+  never skipped when the classification is uncertain.
+
+outputs:
+  decision:
+    description: "run when backend-relevant files changed, otherwise skip"
+    value: ${{ steps.classify.outputs.decision }}
+
+runs:
+  using: composite
+  steps:
+    - id: classify
+      shell: bash
+      env:
+        BASE_SHA: ${{ github.event.pull_request.base.sha }}
+      run: |
+        set -uo pipefail
+        if [ -z "${BASE_SHA:-}" ]; then
+          echo "detect-backend-changes: not a pull_request event; running job"
+          echo "decision=run" >> "${GITHUB_OUTPUT}"
+          exit 0
+        fi
+        if ! git fetch --no-tags --depth=1 origin "${BASE_SHA}" >/dev/null 2>&1; then
+          echo "detect-backend-changes: could not fetch base ${BASE_SHA}; running job"
+          echo "decision=run" >> "${GITHUB_OUTPUT}"
+          exit 0
+        fi
+        changed="$(git diff --name-only "${BASE_SHA}" HEAD 2>/dev/null)" || {
+          echo "detect-backend-changes: git diff failed; running job"
+          echo "decision=run" >> "${GITHUB_OUTPUT}"
+          exit 0
+        }
+        if [ -z "${changed}" ]; then
+          echo "detect-backend-changes: no changed files vs ${BASE_SHA}; skipping job"
+          echo "decision=skip" >> "${GITHUB_OUTPUT}"
+          exit 0
+        fi
+        echo "detect-backend-changes: changed files vs ${BASE_SHA}:"
+        printf '%s\n' "${changed}" | sed 's/^/  /'
+        decision="$(printf '%s\n' "${changed}" | bash .circleci/scripts/classify_changes.sh backend)" || decision="run"
+        echo "detect-backend-changes: decision=${decision}"
+        echo "decision=${decision}" >> "${GITHUB_OUTPUT}"

.github/actions/setup-uv-with-retries/action.yml

@@ -0,0 +1,47 @@
+name: "Set up uv with retries"
+description: >-
+  Install uv via astral-sh/setup-uv, retrying on transient failures. Even with
+  an exact pinned version, the action resolves the artifact URL by fetching
+  https://raw.githubusercontent.com/astral-sh/versions/main/v1/uv.ndjson in a
+  single request with no retry, timeout, or fallback, so one connection-level
+  network error ("fetch failed") fails the whole job before any test runs.
+  Retrying the full step covers the manifest fetch and the binary download.
+
+inputs:
+  version:
+    description: "uv version to install"
+    required: true
+
+runs:
+  using: composite
+  steps:
+    - name: Set up uv (attempt 1)
+      id: attempt-1
+      continue-on-error: true
+      uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
+      with:
+        version: ${{ inputs.version }}
+
+    - name: Wait before attempt 2
+      if: steps.attempt-1.outcome == 'failure'
+      shell: bash
+      run: sleep 15
+
+    - name: Set up uv (attempt 2)
+      id: attempt-2
+      if: steps.attempt-1.outcome == 'failure'
+      continue-on-error: true
+      uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
+      with:
+        version: ${{ inputs.version }}
+
+    - name: Wait before attempt 3
+      if: steps.attempt-2.outcome == 'failure'
+      shell: bash
+      run: sleep 30
+
+    - name: Set up uv (attempt 3)
+      if: steps.attempt-2.outcome == 'failure'
+      uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
+      with:
+        version: ${{ inputs.version }}

.github/pull_request_template.md

@@ -41,3 +41,27 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
 ✅ Test
 
 ## Changes
+
+## QA runbook
+
+<!-- Only needed when your PR edits tests/e2e; delete this section otherwise
+
+For each e2e test you added or changed, list the manual steps a reviewer can follow to reproduce it by hand against a live proxy, mapping 1:1 to what the test asserts: one top-level bullet per test giving its pytest node id followed by what it proves in plain words, then a nested "- [ ]" checklist where each item is a concrete action (route, request body, expected response) and the final item is the sanity-check step shown in the examples. Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for full examples
+
+Example checklists:
+
+- tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_rpm_limit_blocks_over_limit - a key allowed 2 requests a minute serves exactly 2 and refuses the 3rd
+  - [ ] Generate a limited key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"rpm_limit": 2}'
+  - [ ] Send three /v1/chat/completions requests with that key inside one minute
+  - [ ] Expect the first two to return 200 and the third to return 429 naming the rpm limit
+  - [ ] Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
+
+- tests/e2e/management/test_management_e2e.py::TestModelRoutes::test_model_create_appears_in_ui - a deployment created through the API shows up on the Admin UI models page
+  - [ ] POST /model/new with the master key, a bedrock model, and aws_region_name (needs STORE_MODEL_IN_DB=True and AWS credentials)
+  - [ ] Open http://localhost:4000/ui/?page=models and expect a deployment row showing the returned model id
+  - [ ] Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
+-->
+
+### Final Attestation
+
+- [ ] The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

.github/workflows/_test-unit-base.yml

@@ -45,19 +45,25 @@ jobs:
     name: Run tests
     runs-on: ubuntu-latest
     timeout-minutes: ${{ inputs.timeout-minutes }}
+    outputs:
+      decision: ${{ steps.changes.outputs.decision }}
 
     steps:
       - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
         with:
           persist-credentials: false
 
+      - name: Detect backend-relevant changes
+        id: changes
+        uses: ./.github/actions/detect-backend-changes
+
       - name: Set up Python
         uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
         with:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 
@@ -72,16 +78,19 @@ jobs:
             ${{ runner.os }}-uv-
 
       - name: Install dependencies
+        if: steps.changes.outputs.decision != 'skip'
         run: |
           .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
 
       - name: Generate Prisma client
+        if: steps.changes.outputs.decision != 'skip'
         env:
           PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
         run: |
           uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
 
       - name: Run tests
+        if: steps.changes.outputs.decision != 'skip'
         env:
           TEST_PATH: ${{ inputs.test-path }}
           MAX_FAILURES: ${{ inputs.max-failures }}
@@ -114,7 +123,7 @@ jobs:
           fi
 
       - name: Save coverage report
-        if: always()
+        if: always() && steps.changes.outputs.decision != 'skip'
         uses: actions/upload-artifact@4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1 # v4.6.1
         with:
           name: coverage-${{ inputs.artifact-name }}-${{ github.run_id }}-${{ github.run_attempt }}
@@ -124,7 +133,7 @@ jobs:
   upload-coverage:
     name: Upload coverage to Codecov
     needs: run
-    if: always()
+    if: always() && needs.run.outputs.decision != 'skip'
     runs-on: ubuntu-latest
     permissions:
       contents: read

.github/workflows/auto_update_price_and_context_window.yml

@@ -18,7 +18,7 @@ jobs:
         with:
           persist-credentials: false
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
       - name: Update JSON Data

.github/workflows/check-ui-api-types.yml

@@ -31,7 +31,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 

.github/workflows/codspeed.yml

@@ -37,7 +37,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 

.github/workflows/guard-main-branch.yml

@@ -31,12 +31,12 @@ jobs:
           echo "PR head repo: $HEAD_REPO"
           echo "PR head branch: $HEAD_REF"
           if [ "$HEAD_REPO" != "$BASE_REPO" ]; then
-            echo "::error::PRs to main must originate from the canonical repository ($BASE_REPO), not a fork ($HEAD_REPO). External contributors should open PRs against the 'litellm_oss_staging' branch instead."
+            echo "::error::PRs to main must originate from the canonical repository ($BASE_REPO), not a fork ($HEAD_REPO). External contributors should open PRs against the current daily OSS branch (named litellm_oss_daily_YYYY_MM_DD; a fresh one is cut each weekday, so target the most recent) instead."
             exit 1
           fi
           if [ "$HEAD_REF" = "litellm_internal_staging" ] || [[ "$HEAD_REF" == litellm_hotfix_?* ]]; then
             echo "Allowed source branch."
             exit 0
           fi
-          echo "::error::PRs to main must originate from 'litellm_internal_staging' or a 'litellm_hotfix_*' branch. Got: '$HEAD_REF'. If this is a contribution, retarget the PR against 'litellm_oss_staging' instead."
+          echo "::error::PRs to main must originate from 'litellm_internal_staging' or a 'litellm_hotfix_*' branch. Got: '$HEAD_REF'. If this is a contribution, retarget the PR against the current daily OSS branch (named litellm_oss_daily_YYYY_MM_DD; a fresh one is cut each weekday, so target the most recent) instead."
           exit 1

.github/workflows/mutation-test.yml

@@ -39,7 +39,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 

.github/workflows/oss_daily_guardrails.yml

@@ -35,7 +35,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 

.github/workflows/test-code-quality.yml

@@ -38,7 +38,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 

.github/workflows/test-linting.yml

@@ -33,7 +33,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 
@@ -48,7 +48,7 @@ jobs:
 
       - name: Install dependencies
         run: |
-          uv sync --frozen --group proxy-dev
+          uv sync --frozen --group proxy-dev --group e2e-dev
 
       # basedpyright resolves Prisma's generated client (litellm/proxy/schema.prisma)
       # only after `prisma generate` writes prisma/client.py et al. Without this the
@@ -107,6 +107,16 @@ jobs:
         run: |
           (uv run --no-sync basedpyright --outputjson || true) | uv run --no-sync python scripts/type_check_gate.py --base "$BASE_SHA"
 
+      - name: Check tests/e2e basedpyright (zero errors)
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+        run: |
+          if git diff --name-only --diff-filter=ACMRD "$BASE_SHA"...HEAD -- 'tests/e2e/**/*.py' | grep -q .; then
+            uv run --no-sync basedpyright tests/e2e
+          else
+            echo "No changed tests/e2e Python files; skipping."
+          fi
+
       - name: Check for circular imports
         run: |
           cd litellm
@@ -162,7 +172,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 

.github/workflows/test-litellm-ui-build.yml

@@ -36,79 +36,3 @@ jobs:
 
       - name: Build
         run: npm run build
-
-  frontend-lint:
-    runs-on: ubuntu-latest
-    timeout-minutes: 8
-    defaults:
-      run:
-        working-directory: ui/litellm-dashboard
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
-        with:
-          fetch-depth: 0
-          persist-credentials: false
-
-      - name: Collect changed files
-        id: changed
-        env:
-          BASE_SHA: ${{ github.event.pull_request.base.sha }}
-        run: |
-          : > "$RUNNER_TEMP/prettier_files.txt"
-          : > "$RUNNER_TEMP/eslint_files.txt"
-          while IFS= read -r f; do
-            [ -f "$f" ] || continue
-            case "$f" in
-              *.js | *.jsx | *.ts | *.tsx | *.mjs | *.cjs)
-                printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt"
-                printf '%s\n' "$f" >> "$RUNNER_TEMP/eslint_files.txt" ;;
-              *.json | *.css | *.scss | *.md | *.mdx | *.yml | *.yaml | *.html)
-                printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt" ;;
-            esac
-          done < <(git diff --name-only --diff-filter=ACMR --relative "$BASE_SHA"...HEAD -- .)
-          if [ -s "$RUNNER_TEMP/prettier_files.txt" ] || [ -s "$RUNNER_TEMP/eslint_files.txt" ]; then
-            echo "has_files=true" >> "$GITHUB_OUTPUT"
-          else
-            echo "has_files=false" >> "$GITHUB_OUTPUT"
-            echo "No lintable UI files changed in this PR; nothing to check."
-          fi
-
-      - name: Setup Node.js
-        if: steps.changed.outputs.has_files == 'true'
-        uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
-        with:
-          node-version: "20"
-          cache: "npm"
-          cache-dependency-path: ui/litellm-dashboard/package-lock.json
-
-      - name: Install dependencies
-        if: steps.changed.outputs.has_files == 'true'
-        run: npm ci
-
-      - name: Lint changed files (prettier + eslint)
-        if: steps.changed.outputs.has_files == 'true'
-        run: |
-          prettier_files=()
-          eslint_files=()
-          while IFS= read -r f; do prettier_files+=("$f"); done < "$RUNNER_TEMP/prettier_files.txt"
-          while IFS= read -r f; do eslint_files+=("$f"); done < "$RUNNER_TEMP/eslint_files.txt"
-          status=0
-          if [ ${#prettier_files[@]} -gt 0 ]; then
-            echo "::group::Prettier (${#prettier_files[@]} files)"
-            npx prettier --check "${prettier_files[@]}" || { status=1; echo "::error::Unformatted files. Fix with: npm run format"; }
-            echo "::endgroup::"
-          fi
-          if [ ${#eslint_files[@]} -gt 0 ]; then
-            echo "::group::ESLint (${#eslint_files[@]} files)"
-            npx eslint --no-warn-ignored --pass-on-unpruned-suppressions "${eslint_files[@]}" || status=1
-            echo "::endgroup::"
-          fi
-          exit $status
-
-      - name: Check lint budgets
-        if: ${{ !cancelled() && steps.changed.outputs.has_files == 'true' }}
-        run: |
-          npx eslint . -f json -o "$RUNNER_TEMP/lint-report.json" || true
-          node scripts/check-lint-budgets.mjs "$RUNNER_TEMP/lint-report.json" eslint-budgets.json --check eslint-metrics.json

.github/workflows/test-litellm-ui-lint.yml

@@ -0,0 +1,92 @@
+name: UI Lint
+permissions:
+  contents: read
+
+on:
+  pull_request:
+    branches:
+      - main
+      - litellm_internal_staging
+      - litellm_oss_staging
+      - "litellm_**"
+
+jobs:
+  frontend-lint:
+    runs-on: ubuntu-latest
+    timeout-minutes: 8
+    defaults:
+      run:
+        working-directory: ui/litellm-dashboard
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: Collect changed files
+        id: changed
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+        run: |
+          : > "$RUNNER_TEMP/prettier_files.txt"
+          : > "$RUNNER_TEMP/eslint_files.txt"
+          while IFS= read -r f; do
+            [ -f "$f" ] || continue
+            case "$f" in
+              *.js | *.jsx | *.ts | *.tsx | *.mjs | *.cjs)
+                printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt"
+                printf '%s\n' "$f" >> "$RUNNER_TEMP/eslint_files.txt" ;;
+              *.json | *.css | *.scss | *.md | *.mdx | *.yml | *.yaml | *.html)
+                printf '%s\n' "$f" >> "$RUNNER_TEMP/prettier_files.txt" ;;
+            esac
+          done < <(git diff --name-only --diff-filter=ACMR --relative "$BASE_SHA"...HEAD -- .)
+          if [ -s "$RUNNER_TEMP/prettier_files.txt" ] || [ -s "$RUNNER_TEMP/eslint_files.txt" ]; then
+            echo "has_files=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "has_files=false" >> "$GITHUB_OUTPUT"
+            echo "No lintable UI files changed in this PR; nothing to check."
+          fi
+
+      - name: Setup Node.js
+        if: steps.changed.outputs.has_files == 'true'
+        uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
+        with:
+          node-version: "20"
+          cache: "npm"
+          cache-dependency-path: ui/litellm-dashboard/package-lock.json
+
+      - name: Install dependencies
+        if: steps.changed.outputs.has_files == 'true'
+        run: npm ci
+
+      - name: Lint changed files (prettier + eslint)
+        if: steps.changed.outputs.has_files == 'true'
+        run: |
+          prettier_files=()
+          eslint_files=()
+          while IFS= read -r f; do prettier_files+=("$f"); done < "$RUNNER_TEMP/prettier_files.txt"
+          while IFS= read -r f; do eslint_files+=("$f"); done < "$RUNNER_TEMP/eslint_files.txt"
+          status=0
+          if [ ${#prettier_files[@]} -gt 0 ]; then
+            echo "::group::Prettier (${#prettier_files[@]} files)"
+            npx prettier --check "${prettier_files[@]}" || { status=1; echo "::error::Unformatted files. Fix with: npm run format"; }
+            echo "::endgroup::"
+          fi
+          if [ ${#eslint_files[@]} -gt 0 ]; then
+            echo "::group::ESLint (${#eslint_files[@]} files)"
+            npx eslint --no-warn-ignored --pass-on-unpruned-suppressions "${eslint_files[@]}" || status=1
+            echo "::endgroup::"
+          fi
+          exit $status
+
+      - name: Check lint budgets
+        if: ${{ !cancelled() && steps.changed.outputs.has_files == 'true' }}
+        run: |
+          npx eslint . -f json -o "$RUNNER_TEMP/lint-report.json" || true
+          node scripts/check-lint-budgets.mjs "$RUNNER_TEMP/lint-report.json" eslint-budgets.json
+
+      - name: Check for dead code (knip)
+        if: ${{ !cancelled() && steps.changed.outputs.has_files == 'true' }}
+        run: npm run knip:ci

.github/workflows/test-mcp.yml

@@ -32,7 +32,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 

.github/workflows/test-semgrep.yml

@@ -31,7 +31,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 

.github/workflows/test-terraform-provider.yml

@@ -74,7 +74,7 @@ jobs:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 

.github/workflows/test-unit-documentation.yml

@@ -32,13 +32,17 @@ jobs:
           path: docs/my-website
           persist-credentials: false
 
+      - name: Detect backend-relevant changes
+        id: changes
+        uses: ./.github/actions/detect-backend-changes
+
       - name: Set up Python
         uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
         with:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 
@@ -53,10 +57,12 @@ jobs:
             ${{ runner.os }}-uv-
 
       - name: Install dependencies
+        if: steps.changes.outputs.decision != 'skip'
         run: |
           .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
 
       - name: Generate Prisma client
+        if: steps.changes.outputs.decision != 'skip'
         env:
           PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
         run: |
@@ -64,6 +70,7 @@ jobs:
 
       # Run the same documentation tests that CircleCI ran (as direct Python scripts)
       - name: Run documentation validation tests
+        if: steps.changes.outputs.decision != 'skip'
         run: |
           uv run --no-sync python ./tests/documentation_tests/test_env_keys.py
           uv run --no-sync python ./tests/documentation_tests/test_router_settings.py

.github/workflows/test-unit-proxy-legacy.yml

@@ -49,13 +49,17 @@ jobs:
         with:
           persist-credentials: false
 
+      - name: Detect backend-relevant changes
+        id: changes
+        uses: ./.github/actions/detect-backend-changes
+
       - name: Set up Python
         uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
         with:
           python-version: "3.12"
 
       - name: Set up uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
+        uses: ./.github/actions/setup-uv-with-retries
         with:
           version: "0.10.9"
 
@@ -70,16 +74,19 @@ jobs:
             ${{ runner.os }}-uv-
 
       - name: Install dependencies
+        if: steps.changes.outputs.decision != 'skip'
         run: |
           .github/scripts/uv_sync_with_retries.sh --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router
 
       - name: Generate Prisma client
+        if: steps.changes.outputs.decision != 'skip'
         env:
           PRISMA_BINARY_CACHE_DIR: ${{ runner.temp }}/prisma-cache
         run: |
           uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
 
       - name: Run tests - ${{ matrix.test-group.name }}
+        if: steps.changes.outputs.decision != 'skip'
         env:
           TEST_PATH: ${{ matrix.test-group.path }}
         run: |

.gitignore

@@ -106,6 +106,13 @@ STABILIZATION_TODO.md
 **/coverage
 test-config
 
+# Claude Code compatibility-matrix pytest artifact (CI-only output).
+compat-results.json
+compat-results.json.shards/
+compat-rate-limit-summary.json
+# Matrix JSON produced by the daily-cron publisher (pushed to litellm-docs).
+compatibility-matrix.json
+
 # ---------- Terraform ----------
 # Provider binaries + module cache — regenerated by `terraform init`.
 **/.terraform/

CLAUDE.md

@@ -19,7 +19,7 @@ Same thing for bug fixes. The tests should make it so that this specific bug can
 
 End-to-end tests belong in `tests/e2e/` and must follow the harness conventions documented in that directory's `CLAUDE.md`
 
-When creating PRs, don't set base to `main`. `litellm_internal_staging` serves that purpose
+When creating PRs, don't set base to `main`. `litellm_internal_staging` serves that purpose for internal contributors; external / OSS contributions target the current daily OSS branch instead, named `litellm_oss_daily_YYYY_MM_DD` (a fresh one is cut each weekday, so use the most recent)
 
 When writing a PR body, treat the comments and imperative instructions inside @.github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule
 
@@ -39,6 +39,8 @@ Don't hesitate to use values in .env to get needed API keys and other secrets, a
 
 Python max line length is 120, not 88
 
+On a fresh worktree or clone, run `make bootstrap` before anything else. It provisions everything tests, `make pre-commit`, and a local proxy need
+
 Run tests before you commit. Also, run `make pre-commit` right before each commit, which generates types (as needed) and formats/lints your code. Any errors found must be fixed. It only runs when there are staged frontend and/or backend changes and calculates violations, generates types, etc. based on the worktree, so stage what you need or stash/delete unwanted files in litellm/ or ui/ (where backend and frontend lint run, respectively) before running it. If it fails because dashboard api types are stale, it already regenerated them for you. You just need to stage the schema.d.ts, re-run `make pre-commit` to confirm it passes, and commit
 
 When you fix violations gated by `ruff-strict-budget.json`, `type-discipline-budget.json`, or `basedpyright-code-budget.json`, run `make lint-budget-update` and commit the lowered limits so the ceilings ratchet down instead of leaving stale headroom. It measures the working tree, so it must contain exactly the fixes you're committing

CONTRIBUTING.md

@@ -322,7 +322,7 @@ npm run build
 ## Submitting Your PR
 
 1. **Push your branch**: `git push origin your-feature-branch`
-2. **Create a PR**: Go to GitHub and create a pull request
+2. **Create a PR**: Go to GitHub and open a pull request against the current daily OSS branch, named `litellm_oss_daily_YYYY_MM_DD`. A fresh one is cut each weekday, so pick the most recent from the [branch list](https://github.com/BerriAI/litellm/branches/all?query=litellm_oss_daily). Do not target `main`.
 3. **Fill out the PR template**: Provide clear description of changes
 4. **Wait for review**: Maintainers will review and provide feedback
 5. **Address feedback**: Make requested changes and push updates

Makefile

@@ -5,15 +5,16 @@
 	test-unit-integrations test-unit-core-utils test-unit-other test-unit-root \
 	test-proxy-unit-a test-proxy-unit-b test-integration test-unit-helm \
 	info lint lint-dev lint-checks format \
-	lint-basedpyright lint-basedpyright-budget-update lint-type-discipline lint-type-discipline-budget-update \
+	lint-basedpyright lint-e2e-basedpyright lint-basedpyright-budget-update lint-type-discipline lint-type-discipline-budget-update \
 	lint-ruff-budget lint-ruff-budget-update lint-budget-update lint-gate \
 	install-dev install-proxy-dev install-test-deps install-hooks \
 	install-helm-unittest check-circular-imports check-import-safety pre-commit \
-	lint-install lint-fetch-base
+	lint-install lint-fetch-base bootstrap
 
 # Default target
 help:
 	@echo "Available commands:"
+	@echo "  make bootstrap          - Provision a fresh clone/worktree"
 	@echo "  make install-dev        - Install development dependencies"
 	@echo "  make install-proxy-dev  - Install proxy development dependencies"
 	@echo "  make install-dev-ci     - Install dev dependencies (CI-compatible, pins OpenAI)"
@@ -27,6 +28,7 @@ help:
 	@echo "  make lint               - Run all linting (Ruff, basedpyright, format check, circular imports, import safety)"
 	@echo "  make lint-ruff          - Run Ruff linting only"
 	@echo "  make lint-basedpyright  - Run basedpyright strict, gated by per-rule error counts"
+	@echo "  make lint-e2e-basedpyright - Run basedpyright over tests/e2e (zero errors allowed)"
 	@echo "  make lint-basedpyright-budget-update - Ratchet basedpyright limits down by what this branch fixed"
 	@echo "  make lint-format        - Check ruff format formatting (matches CI)"
 	@echo "  make lint-ruff-budget - Gate the codebase total of each strict ruff rule against its limit"
@@ -54,6 +56,7 @@ UV := uv
 UV_RUN := $(UV) run --no-sync
 
 LINT_DEP_INSTALL ?= install-dev
+LINT_E2E_DEP_INSTALL ?= lint-install
 LINT_DEP_BASE ?= lint-fetch-base
 LINT_JOBS := $(shell sysctl -n hw.ncpu 2>/dev/null || nproc 2>/dev/null || echo 4)
 LINT_OUTPUT_SYNC := $(if $(filter output-sync,$(.FEATURES)),--output-sync=target,)
@@ -69,6 +72,18 @@ info:
 install-dev:
 	$(UV) sync --inexact --frozen
 
+bootstrap:
+	$(UV) sync --inexact --frozen --extra proxy --group proxy-dev --group e2e-dev
+	$(UV_RUN) python scripts/prisma_generate_if_needed.py
+	cd ui/litellm-dashboard && npm ci --no-audit --no-fund
+	@main_root=$$(git worktree list --porcelain | head -1 | sed 's/^worktree //'); \
+	if [ "$$main_root" != "$$(git rev-parse --show-toplevel)" ] && [ -f "$$main_root/.env" ] && [ ! -f .env ]; then \
+		cp "$$main_root/.env" .env && echo "bootstrap: copied .env from $$main_root"; \
+	else \
+		echo "bootstrap: .env left untouched"; \
+	fi
+	@echo "bootstrap: done"
+
 install-proxy-dev:
 	$(UV) sync --frozen --group proxy-dev --extra proxy
 
@@ -111,7 +126,7 @@ lint-fetch-base:
 # CI's). --inexact tops up the venv instead of pruning the proxy extras gen:api and the
 # running proxy need.
 lint-install:
-	$(UV) sync --inexact --frozen --group proxy-dev
+	$(UV) sync --inexact --frozen --group proxy-dev --group e2e-dev
 	$(UV_RUN) python scripts/prisma_generate_if_needed.py
 
 # Diff-scoped format check, identical to test-linting.yml's "Check ruff format" step:
@@ -164,6 +179,9 @@ lint-ruff-FULL-dev: install-dev
 lint-basedpyright: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
 	($(UV_RUN) basedpyright --outputjson || true) | $(UV_RUN) python scripts/type_check_gate.py --base origin/litellm_internal_staging
 
+lint-e2e-basedpyright: $(LINT_E2E_DEP_INSTALL)
+	$(UV_RUN) basedpyright tests/e2e
+
 # Type-discipline budget (mutable collections / casts / type guards / kwargs /
 # unexplained suppressions), the test-linting.yml step `make lint` used to omit.
 lint-type-discipline: $(LINT_DEP_INSTALL) $(LINT_DEP_BASE)
@@ -208,9 +226,9 @@ check-import-safety: $(LINT_DEP_INSTALL)
 # base fetch) runs once up front; the checks themselves are independent, so a sub-make
 # fans them out with -j and the fast ones finish under basedpyright's shadow.
 lint: lint-install lint-fetch-base
-	$(MAKE) -j $(LINT_JOBS) $(LINT_OUTPUT_SYNC) LINT_DEP_INSTALL= LINT_DEP_BASE= lint-checks
+	$(MAKE) -j $(LINT_JOBS) $(LINT_OUTPUT_SYNC) LINT_DEP_INSTALL= LINT_E2E_DEP_INSTALL= LINT_DEP_BASE= lint-checks
 
-lint-checks: lint-format-check-changed lint-ruff lint-gate lint-type-discipline lint-basedpyright check-circular-imports check-import-safety
+lint-checks: lint-format-check-changed lint-ruff lint-gate lint-type-discipline lint-basedpyright lint-e2e-basedpyright check-circular-imports check-import-safety
 
 # Faster linting for local development (only checks changed code)
 lint-dev: lint-format-changed check-circular-imports check-import-safety

README.md

@@ -552,17 +552,12 @@ The Terraform modules live at [`terraform/litellm/aws/`](./terraform/litellm/aws
 2. Run dependent services `docker-compose up db prometheus`
 
 #### Backend
-1. (In root) create virtual environment `python -m venv .venv`
-2. Activate virtual environment `source .venv/bin/activate`
-3. Install dependencies `uv sync --all-extras --group proxy-dev`
-4. `uv run prisma generate`
-5. `prisma generate`
-6. Start proxy backend `python litellm/proxy/proxy_cli.py`
+1. Run `make bootstrap`
+2. Start proxy backend: `uv run python litellm/proxy/proxy_cli.py`
 
 #### Frontend
-1. Navigate to `ui/litellm-dashboard`
-2. Install dependencies `npm install`
-3. Run `npm run dev` to start the dashboard
+1. Navigate to `ui/litellm-dashboard` (dependencies were already installed w/ `make bootstrap`)
+2. Start dashboard: `npm run dev`
 
 ### Verify Docker Image Signatures
 

backend/routes/allowlist.py

@@ -46,6 +46,7 @@ BACKEND_PATH_PREFIXES: tuple[str, ...] = (
     "/fallback",
     "/fallbacks",
     "/cache_settings",
+    "/coordination_redis/",
     "/cost_tracking",
     "/cost/",
     "/credentials",

basedpyright-code-budget.json

@@ -57,7 +57,7 @@
     "limit": 5900
   },
   "reportMissingTypeArgument": {
-    "limit": 15918
+    "limit": 15903
   },
   "reportMissingTypeStubs": {
     "limit": 41
@@ -105,13 +105,13 @@
     "limit": 113
   },
   "reportUnknownMemberType": {
-    "limit": 40541
+    "limit": 40539
   },
   "reportUnknownParameterType": {
-    "limit": 20418
+    "limit": 20403
   },
   "reportUnknownVariableType": {
-    "limit": 32151
+    "limit": 32141
   },
   "reportUnnecessaryCast": {
     "limit": 177
@@ -123,7 +123,7 @@
     "limit": 7
   },
   "reportUnnecessaryIsInstance": {
-    "limit": 1212
+    "limit": 1209
   },
   "reportUntypedBaseClass": {
     "limit": 165

enterprise/litellm_enterprise/proxy/common_utils/check_batch_cost.py

@@ -29,7 +29,7 @@ class CheckBatchCost:
         proxy_logging_obj: "ProxyLogging",
         prisma_client: "PrismaClient",
         llm_router: "Router",
-        track_unmanaged_vertex_batch_cost: bool = False,
+        track_unmanaged_batch_cost: bool = False,
     ):
         from litellm.proxy.utils import PrismaClient, ProxyLogging
         from litellm.router import Router
@@ -37,7 +37,7 @@ class CheckBatchCost:
         self.proxy_logging_obj: ProxyLogging = proxy_logging_obj
         self.prisma_client: PrismaClient = prisma_client
         self.llm_router: Router = llm_router
-        self._track_unmanaged_vertex_batch_cost = track_unmanaged_vertex_batch_cost
+        self._track_unmanaged_batch_cost = track_unmanaged_batch_cost
         # Cached after the first poll cycle. Once we know the column is absent we skip
         # the guaranteed-failing primary query on every subsequent cycle.
         self._has_batch_processed_column: bool = True
@@ -118,11 +118,11 @@ class CheckBatchCost:
         Resolve (model_id, batch_id) for a managed-object row, where model_id is a router
         deployment id and batch_id is the raw provider batch id.
 
-        Managed batches encode both in a base64 unified id. Unmanaged Vertex batches, created with
-        a raw gs:// input_file_id, store the raw provider job id as unified_object_id; when
-        track_unmanaged_vertex_batch_cost is enabled the model is derived from the gs:// path and
-        mapped to a configured vertex_ai deployment. Returns None (recording a metric) when the row
-        can't be routed.
+        Managed batches encode both in a base64 unified id. Unmanaged batches (created outside
+        LiteLLM's own /v1/batches with a raw input_file_id) store the raw provider job id as
+        unified_object_id instead; when track_unmanaged_batch_cost is enabled the model is derived
+        from the provider-specific input_file_id layout (Vertex gs:// or Bedrock s3://) and mapped
+        to a matching deployment. Returns None (recording a metric) when the row can't be routed.
         """
         from litellm.proxy.openai_files_endpoints.common_utils import (
             _is_base64_encoded_unified_file_id,
@@ -142,8 +142,43 @@ class CheckBatchCost:
                 return None
             return model_id, get_batch_id_from_unified_batch_id(decoded)
 
-        if self._track_unmanaged_vertex_batch_cost:
-            return self._resolve_unmanaged_vertex_routing(job, prom_logger)
+        if self._track_unmanaged_batch_cost:
+            from litellm.llms.bedrock.batches.transformation import (
+                BedrockBatchesConfig,
+            )
+            from litellm.llms.vertex_ai.batches.transformation import (
+                VertexAIBatchTransformation,
+            )
+
+            input_file_id = self._get_input_file_id(job)
+            if VertexAIBatchTransformation.is_unmanaged_gcs_batch_input_file_id(
+                input_file_id
+            ):
+                assert input_file_id is not None  # narrowed by is_unmanaged_gcs_batch_input_file_id
+                return self._resolve_unmanaged_provider_routing(
+                    job=job,
+                    prom_logger=prom_logger,
+                    llm_provider="vertex_ai",
+                    bare_model_name=VertexAIBatchTransformation.get_bare_model_name_from_gcs_file(
+                        input_file_id
+                    ),
+                )
+            if BedrockBatchesConfig.is_unmanaged_s3_batch_input_file_id(input_file_id):
+                assert input_file_id is not None  # narrowed by is_unmanaged_s3_batch_input_file_id
+                return self._resolve_unmanaged_provider_routing(
+                    job=job,
+                    prom_logger=prom_logger,
+                    llm_provider="bedrock",
+                    bare_model_name=BedrockBatchesConfig.get_bare_model_name_from_s3_file(
+                        input_file_id
+                    ),
+                )
+            verbose_proxy_logger.info(
+                f"Skipping job {unified_object_id}: not a recognized unmanaged batch "
+                "(no gs:// or s3:// input_file_id with an embedded model)"
+            )
+            self._record_error(prom_logger, "invalid_unified_id")
+            return None
 
         verbose_proxy_logger.info(
             f"Skipping job {unified_object_id} because it is not a valid unified object id"
@@ -151,36 +186,17 @@ class CheckBatchCost:
         self._record_error(prom_logger, "invalid_unified_id")
         return None
 
-    def _resolve_unmanaged_vertex_routing(
+    def _resolve_unmanaged_provider_routing(
         self,
         job: "LiteLLM_ManagedObjectTable",
         prom_logger: Optional["PrometheusLogger"],
+        llm_provider: str,
+        bare_model_name: str,
     ) -> Optional[Tuple[str, str]]:
-        from litellm.llms.vertex_ai.batches.transformation import (
-            VertexAIBatchTransformation,
-        )
-
-        input_file_id = self._get_input_file_id(job)
-        if not VertexAIBatchTransformation.is_unmanaged_gcs_batch_input_file_id(
-            input_file_id
-        ):
-            verbose_proxy_logger.info(
-                f"Skipping job {job.unified_object_id}: not an unmanaged vertex batch "
-                "(no gs:// input_file_id with a publishers/ model path)"
-            )
-            self._record_error(prom_logger, "invalid_unified_id")
-            return None
-        assert input_file_id is not None  # narrowed by is_unmanaged_gcs_batch_input_file_id
-
-        bare_model_name = VertexAIBatchTransformation.get_bare_model_name_from_gcs_file(
-            input_file_id
-        )
-        deployment_id = self._get_vertex_ai_deployment_id_for_bare_model(
-            bare_model_name
-        )
+        deployment_id = self._get_deployment_id_for_bare_model(bare_model_name, llm_provider)
         if deployment_id is None:
             verbose_proxy_logger.info(
-                f"Skipping unmanaged vertex batch {job.unified_object_id}: no vertex_ai "
+                f"Skipping unmanaged {llm_provider} batch {job.unified_object_id}: no {llm_provider} "
                 f"deployment configured for model {bare_model_name}"
             )
             self._record_error(prom_logger, "unmanaged_no_matching_deployment")
@@ -188,22 +204,22 @@ class CheckBatchCost:
 
         return deployment_id, job.unified_object_id
 
-    def _get_vertex_ai_deployment_id_for_bare_model(
-        self, bare_model_name: str
+    def _get_deployment_id_for_bare_model(
+        self, bare_model_name: str, llm_provider: str
     ) -> Optional[str]:
         model_group = self.llm_router.resolve_model_name_from_model_id(bare_model_name)
         deployment_id = (
-            self._get_vertex_ai_deployment_id(model_group) if model_group else None
+            self._get_deployment_id_for_provider(model_group, llm_provider) if model_group else None
         )
         if deployment_id is not None:
             return deployment_id
 
-        return self._get_vertex_ai_deployment_id_from_matching_deployments(
-            bare_model_name
+        return self._get_deployment_id_from_matching_deployments(
+            bare_model_name, llm_provider
         )
 
-    def _get_vertex_ai_deployment_id_from_matching_deployments(
-        self, bare_model_name: str
+    def _get_deployment_id_from_matching_deployments(
+        self, bare_model_name: str, llm_provider: str
     ) -> Optional[str]:
         from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
 
@@ -215,13 +231,13 @@ class CheckBatchCost:
             if not self._is_bare_model_match(actual_model, bare_model_name):
                 continue
             try:
-                _, llm_provider, _, _ = get_llm_provider(
+                _, deployment_llm_provider, _, _ = get_llm_provider(
                     model=actual_model,
                     custom_llm_provider=litellm_params.get("custom_llm_provider"),
                 )
             except Exception:
                 continue
-            if llm_provider != "vertex_ai":
+            if deployment_llm_provider != llm_provider:
                 continue
             model_info = deployment.get("model_info") or {}
             deployment_id = model_info.get("id")
@@ -231,15 +247,21 @@ class CheckBatchCost:
 
     @staticmethod
     def _is_bare_model_match(actual_model: str, bare_model_name: str) -> bool:
+        # Bedrock model ids may have ":" replaced with "-" in the S3 object key (see
+        # BedrockBatchesConfig.get_bare_model_name_from_s3_file), so normalize both sides;
+        # a no-op for providers like vertex_ai whose model ids never contain a colon.
+        normalized_actual = actual_model.replace(":", "-")
+        normalized_bare = bare_model_name.replace(":", "-")
         return (
-            actual_model == bare_model_name
-            or actual_model.endswith(f"/{bare_model_name}")
-            or actual_model.endswith(f":{bare_model_name}")
+            normalized_actual == normalized_bare
+            or normalized_actual.endswith(f"/{normalized_bare}")
         )
 
-    def _get_vertex_ai_deployment_id(self, model_group: str) -> Optional[str]:
+    def _get_deployment_id_for_provider(
+        self, model_group: str, llm_provider: str
+    ) -> Optional[str]:
         """
-        Returns the first deployment id for `model_group` whose provider is vertex_ai,
+        Returns the first deployment id for `model_group` whose provider is `llm_provider`,
         skipping deployments from other providers that happen to share the model group name.
         """
         from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
@@ -249,13 +271,13 @@ class CheckBatchCost:
             if deployment_info is None:
                 continue
             try:
-                _, llm_provider, _, _ = get_llm_provider(
+                _, deployment_llm_provider, _, _ = get_llm_provider(
                     model=deployment_info.litellm_params.model,
                     custom_llm_provider=deployment_info.litellm_params.custom_llm_provider,
                 )
             except Exception:
                 continue
-            if llm_provider == "vertex_ai":
+            if deployment_llm_provider == llm_provider:
                 return deployment_id
         return None
 

enterprise/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "litellm-enterprise"
-version = "0.1.49"
+version = "0.1.50"
 description = "Package for LiteLLM Enterprise features"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -26,7 +26,7 @@ required-version = ">=0.10.9"
 module-root = ""
 
 [tool.commitizen]
-version = "0.1.49"
+version = "0.1.50"
 version_files = [
     "pyproject.toml:^version",
     "../pyproject.toml:litellm-enterprise==",

helm/litellm-helm/templates/_helpers.tpl

@@ -76,10 +76,13 @@ so fall back to "default" (or an explicit override) to avoid a cyclic dependency
 {{- end }}
 
 {{/*
-Get redis service name
+Get redis service name.
+The bundled Redis subchart only serves sentinel in "replication" architecture
+(it rejects standalone + sentinel outright), and in that mode the sentinel
+Service is named "<release>-redis", not "<release>-redis-master".
 */}}
 {{- define "litellm.redis.serviceName" -}}
-{{- if and (eq .Values.redis.architecture "standalone") .Values.redis.sentinel.enabled -}}
+{{- if .Values.redis.sentinel.enabled -}}
 {{- printf "%s-%s" .Release.Name (default "redis" .Values.redis.nameOverride | trunc 63 | trimSuffix "-") -}}
 {{- else -}}
 {{- printf "%s-%s-master" .Release.Name (default "redis" .Values.redis.nameOverride | trunc 63 | trimSuffix "-") -}}

helm/litellm-helm/templates/configmap-litellm.yaml

@@ -1,9 +1,22 @@
 {{- if .Values.proxyConfigMap.create }}
+{{- $config := deepCopy .Values.proxy_config }}
+{{- if and .Values.redis.enabled (dig "coordination" "enabled" true .Values.redis) }}
+{{- $generalSettings := (get $config "general_settings") | default dict }}
+{{- if not (hasKey $generalSettings "coordination_redis") }}
+{{- $coordinationRedis := dict "host" "os.environ/REDIS_HOST" "port" "os.environ/REDIS_PORT" "password" "os.environ/REDIS_PASSWORD" }}
+{{- if .Values.redis.sentinel.enabled }}
+{{- $sentinelNode := list (include "litellm.redis.serviceName" .) (include "litellm.redis.port" . | int) }}
+{{- $coordinationRedis = dict "sentinel_nodes" (list $sentinelNode) "service_name" (default "mymaster" .Values.redis.sentinel.masterSet) "password" "os.environ/REDIS_PASSWORD" }}
+{{- end }}
+{{- $_ := set $generalSettings "coordination_redis" $coordinationRedis }}
+{{- $_ := set $config "general_settings" $generalSettings }}
+{{- end }}
+{{- end }}
 apiVersion: v1
 kind: ConfigMap
 metadata:
   name: {{ include "litellm.fullname" . }}-config
 data:
   config.yaml: |
-{{ .Values.proxy_config | toYaml | indent 6 }}
+{{ $config | toYaml | indent 6 }}
 {{- end }}

helm/litellm-helm/tests/coordination_redis_tests.yaml

@@ -0,0 +1,143 @@
+suite: test coordination redis
+templates:
+  - configmap-litellm.yaml
+  - deployment.yaml
+tests:
+  - it: should not render coordination_redis when redis is disabled
+    template: configmap-litellm.yaml
+    set:
+      redis.enabled: false
+    asserts:
+      - notMatchRegex:
+          path: data["config.yaml"]
+          pattern: coordination_redis
+
+  - it: should not emit redis env vars when redis is disabled
+    template: deployment.yaml
+    set:
+      redis.enabled: false
+    asserts:
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_HOST
+            value: RELEASE-NAME-redis-master
+          any: true
+
+  - it: should render coordination_redis pointing at the bundled redis when enabled
+    template: configmap-litellm.yaml
+    set:
+      redis.enabled: true
+    asserts:
+      - matchRegex:
+          path: data["config.yaml"]
+          pattern: "coordination_redis:\n    host: os.environ/REDIS_HOST\n    password: os.environ/REDIS_PASSWORD\n    port: os.environ/REDIS_PORT\n"
+      - matchRegex:
+          path: data["config.yaml"]
+          pattern: "master_key: os.environ/PROXY_MASTER_KEY"
+
+  - it: should emit redis env vars backing the coordination_redis os.environ refs
+    template: deployment.yaml
+    set:
+      redis.enabled: true
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_HOST
+            value: RELEASE-NAME-redis-master
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_PORT
+            value: "6379"
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                name: RELEASE-NAME-redis
+                key: redis-password
+
+  - it: should not render coordination_redis when coordination is opted out
+    template: configmap-litellm.yaml
+    set:
+      redis.enabled: true
+      redis.coordination.enabled: false
+    asserts:
+      - notMatchRegex:
+          path: data["config.yaml"]
+          pattern: coordination_redis
+
+  - it: should keep emitting redis env vars when coordination is opted out
+    template: deployment.yaml
+    set:
+      redis.enabled: true
+      redis.coordination.enabled: false
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_HOST
+            value: RELEASE-NAME-redis-master
+
+  - it: should not clobber a user supplied coordination_redis block
+    template: configmap-litellm.yaml
+    set:
+      redis.enabled: true
+      proxy_config.general_settings.coordination_redis:
+        url: os.environ/COORDINATION_REDIS_URL
+    asserts:
+      - matchRegex:
+          path: data["config.yaml"]
+          pattern: "coordination_redis:\n    url: os.environ/COORDINATION_REDIS_URL\n"
+      - notMatchRegex:
+          path: data["config.yaml"]
+          pattern: "host: os.environ/REDIS_HOST"
+
+  - it: should render sentinel_nodes and service_name in sentinel mode
+    template: configmap-litellm.yaml
+    set:
+      redis.enabled: true
+      redis.architecture: replication
+      redis.sentinel.enabled: true
+    asserts:
+      # The sentinel Service the redis subchart renders is "<release>-redis", and a
+      # plain client cannot speak the sentinel protocol, so host/port must not appear
+      - matchRegex:
+          path: data["config.yaml"]
+          pattern: "coordination_redis:\n    password: os.environ/REDIS_PASSWORD\n    sentinel_nodes:\n    - - RELEASE-NAME-redis\n      - 26379\n    service_name: mymaster\n"
+      - notMatchRegex:
+          path: data["config.yaml"]
+          pattern: "host: os.environ/REDIS_HOST"
+
+  - it: should carry a custom sentinel masterSet into service_name
+    template: configmap-litellm.yaml
+    set:
+      redis.enabled: true
+      redis.architecture: replication
+      redis.sentinel.enabled: true
+      redis.sentinel.masterSet: litellm-master
+    asserts:
+      - matchRegex:
+          path: data["config.yaml"]
+          pattern: "service_name: litellm-master"
+
+  - it: should point REDIS_HOST at the sentinel service in sentinel mode
+    template: deployment.yaml
+    set:
+      redis.enabled: true
+      redis.architecture: replication
+      redis.sentinel.enabled: true
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_HOST
+            value: RELEASE-NAME-redis
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_PORT
+            value: "26379"

helm/litellm-helm/values.yaml

@@ -331,12 +331,28 @@ postgresql:
     # secretKeys:
     #   userPasswordKey: password
 
-# requires cache: true in config file
-# either enable this or pass a secret for REDIS_HOST, REDIS_PORT, REDIS_PASSWORD or REDIS_URL
-# with cache: true to use existing redis instance
+# Redis is the proxy's coordination store: cross-pod tpm/rpm rate limits, spend
+# tracking, and the pod lock manager. Enabling this deploys the bundled Redis
+# subchart, wires REDIS_HOST / REDIS_PORT / REDIS_PASSWORD into the proxy, and
+# renders a `general_settings.coordination_redis` block into the proxy config.
+#
+# To point at an existing Redis instead, leave `enabled: false` and pass a
+# secret for REDIS_HOST, REDIS_PORT, REDIS_PASSWORD or REDIS_URL; the proxy
+# falls back to those env vars for coordination. Set `cache: true` in the proxy
+# config only if you also want LLM response caching, which is independent of
+# coordination
+#
+# When `redis.sentinel.enabled` is set, the coordination block is rendered with
+# `sentinel_nodes` and `service_name` (from `redis.sentinel.masterSet`) instead
+# of host/port, because a plain Redis client cannot talk to the sentinel port
 redis:
   enabled: false
   architecture: standalone
+  coordination:
+    # Set to false to keep the bundled Redis for response caching only and leave
+    # `general_settings.coordination_redis` out of the rendered config. A
+    # `coordination_redis` block you define yourself in `proxy_config` always wins
+    enabled: true
 
 # Prisma migration job settings
 migrationJob:

helm/litellm/templates/_helpers.tpl

@@ -213,6 +213,10 @@ harmless no-op for the Job and authoritative for the app pods.
 */}}
 - name: DISABLE_SCHEMA_UPDATE
   value: "true"
+{{/* These feed the proxy's coordination Redis (cross-pod rate limits, spend
+     tracking, pod lock manager) via its REDIS_* env fallback. An explicit
+     `general_settings.coordination_redis` block in proxy_config takes
+     precedence over anything emitted here. */}}
 {{- if $root.Values.redis.host }}
 - name: REDIS_HOST
   value: {{ $root.Values.redis.host | quote }}
@@ -226,10 +230,11 @@ harmless no-op for the Job and authoritative for the app pods.
       key: {{ $root.Values.redis.passwordSecret.passwordKey | default "password" }}
 {{- end }}
 {{- if $root.Values.redis.cluster }}
-{{/* The proxy's Cache() reads REDIS_CLUSTER_NODES as JSON and constructs a
-     RedisClusterCache when it's set (litellm/caching/caching.py:169-192).
-     We seed with the single configured endpoint — the cluster client
-     discovers the remaining nodes from CLUSTER SLOTS at startup. */}}
+{{/* The proxy falls back to REDIS_CLUSTER_NODES (JSON) to build a cluster-mode
+     coordination client when `general_settings.coordination_redis` is absent
+     and no plain-Redis response cache is configured. We seed with the single
+     configured endpoint; the cluster client discovers the remaining nodes from
+     CLUSTER SLOTS at startup. */}}
 - name: REDIS_CLUSTER_NODES
   value: {{ printf "[{\"host\":%q,\"port\":%v}]" $root.Values.redis.host (int $root.Values.redis.port) | quote }}
 {{- end }}

helm/litellm/tests/redis_env_tests.yaml

@@ -0,0 +1,109 @@
+suite: test redis coordination env vars
+templates:
+  - gateway/deployment.yaml
+  - gateway/configmap.yaml
+  - backend/deployment.yaml
+values:
+  - ./values/required.yaml
+tests:
+  - it: gateway omits redis env vars when no host is configured
+    template: gateway/deployment.yaml
+    asserts:
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_HOST
+            value: redis.example.com
+          any: true
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_CLUSTER_NODES
+          any: true
+
+  - it: gateway emits host, port and password when redis is configured
+    template: gateway/deployment.yaml
+    set:
+      redis.host: redis.example.com
+      redis.port: 6380
+      redis.passwordSecret.name: redis-secret
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_HOST
+            value: redis.example.com
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_PORT
+            value: "6380"
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                name: redis-secret
+                key: password
+
+  - it: backend emits the same redis env vars so both pods coordinate on one redis
+    template: backend/deployment.yaml
+    set:
+      redis.host: redis.example.com
+      redis.passwordSecret.name: redis-secret
+      redis.passwordSecret.passwordKey: redis-password
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_HOST
+            value: redis.example.com
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                name: redis-secret
+                key: redis-password
+
+  - it: gateway omits REDIS_PASSWORD for an auth-less redis
+    template: gateway/deployment.yaml
+    set:
+      redis.host: redis.example.com
+    asserts:
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_PASSWORD
+          any: true
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_HOST
+            value: redis.example.com
+
+  - it: gateway seeds REDIS_CLUSTER_NODES from host and port in cluster mode
+    template: gateway/deployment.yaml
+    set:
+      redis.host: redis.example.com
+      redis.port: 6380
+      redis.cluster: true
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_CLUSTER_NODES
+            value: '[{"host":"redis.example.com","port":6380}]'
+
+  - it: gateway omits REDIS_CLUSTER_NODES when cluster mode is off
+    template: gateway/deployment.yaml
+    set:
+      redis.host: redis.example.com
+    asserts:
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: REDIS_CLUSTER_NODES
+          any: true

helm/litellm/values.yaml

@@ -100,7 +100,18 @@ database:
       usernameKey: username
       passwordKey: password
 
-# Optional Redis (caching, rate limiting). Leave host empty to disable.
+# Optional Redis. Leave host empty to disable.
+#
+# This is the proxy's coordination store: cross-pod tpm/rpm rate limits, spend
+# tracking, and the pod lock manager. The chart emits REDIS_HOST / REDIS_PORT /
+# REDIS_PASSWORD, which the proxy picks up through its coordination Redis env
+# fallback. Response caching is separate and off unless you enable it in
+# `proxy_config.litellm_settings.cache`.
+#
+# For full control, define `general_settings.coordination_redis` in
+# `proxy_config` (host/port/password/username/url/ssl/startup_nodes/
+# sentinel_nodes/sentinel_password/service_name, each accepting os.environ/VAR
+# refs). An explicit block overrides these env vars.
 #
 # Set `cluster: true` for Redis Cluster mode (e.g. AWS ElastiCache Cluster,
 # self-hosted Redis Cluster). The chart emits REDIS_CLUSTER_NODES from

litellm-proxy-extras/litellm_proxy_extras/migrations/20260710000000_add_dcr_bridge_to_mcp_server_table/migration.sql

@@ -0,0 +1,2 @@
+-- AlterTable
+ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN     "dcr_bridge" BOOLEAN;

litellm-proxy-extras/litellm_proxy_extras/migrations/20260713230852_add_key_type_to_litellm_verification_token/migration.sql

@@ -0,0 +1,6 @@
+-- AlterTable
+ALTER TABLE "LiteLLM_DeletedVerificationToken" ADD COLUMN     "key_type" TEXT;
+
+-- AlterTable
+ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN     "key_type" TEXT;
+

litellm-proxy-extras/litellm_proxy_extras/schema.prisma

@@ -339,6 +339,7 @@ model LiteLLM_MCPServerTable {
   available_on_public_internet Boolean @default(true)
   delegate_auth_to_upstream Boolean @default(false)
   oauth_passthrough Boolean @default(false)
+  dcr_bridge Boolean?
   is_byok               Boolean  @default(false)
   byok_description      String[] @default([])
   byok_api_key_help_url String?
@@ -421,6 +422,7 @@ model LiteLLM_VerificationToken {
     budget_reset_at DateTime?
     allowed_cache_controls String[] @default([])
     allowed_routes   String[] @default([])
+    key_type         String?
     policies         String[] @default([])
     access_group_ids String[] @default([])
     model_spend      Json @default("{}")
@@ -515,6 +517,7 @@ model LiteLLM_DeletedVerificationToken {
     budget_reset_at         DateTime?
     allowed_cache_controls  String[] @default([])
     allowed_routes          String[] @default([])
+    key_type                String?
     policies                String[] @default([])
     access_group_ids String[] @default([])
     model_spend             Json     @default("{}")

litellm-proxy-extras/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "litellm-proxy-extras"
-version = "0.4.75"
+version = "0.4.77"
 description = "Additional files for the LiteLLM Proxy. Reduces the size of the main litellm package."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -26,7 +26,7 @@ required-version = ">=0.10.9"
 module-root = ""
 
 [tool.commitizen]
-version = "0.4.75"
+version = "0.4.77"
 version_files = [
     "pyproject.toml:^version",
     "../pyproject.toml:litellm-proxy-extras==",

litellm/_redis.py

@@ -325,8 +325,19 @@ def _get_redis_client_logic(**env_overrides):
             value = get_secret(v)  # type: ignore
             env_overrides[k] = value
 
+    environment_kwargs = _redis_kwargs_from_environment()
+
+    # An explicitly configured connection target outranks REDIS_URL from the
+    # environment. Without this, the url branch below strips the caller's
+    # host/port/password and silently connects to whatever REDIS_URL names.
+    caller_named_a_target = any(
+        env_overrides.get(key) is not None for key in ("host", "startup_nodes", "sentinel_nodes")
+    )
+    if caller_named_a_target and env_overrides.get("url") is None:
+        environment_kwargs.pop("url", None)
+
     redis_kwargs = {
-        **_redis_kwargs_from_environment(),
+        **environment_kwargs,
         **env_overrides,
     }
 
@@ -678,9 +689,8 @@ def get_redis_connection_pool(
         redis_kwargs["credential_provider"] = GCPIAMCredentialProvider(redis_connect_func._gcp_service_account)
 
     connection_class = async_redis.Connection
-    if "ssl" in redis_kwargs:
+    if redis_kwargs.pop("ssl", False):
         connection_class = async_redis.SSLConnection
-        redis_kwargs.pop("ssl", None)
         redis_kwargs["connection_class"] = connection_class
     return async_redis.BlockingConnectionPool(timeout=REDIS_CONNECTION_POOL_TIMEOUT, **redis_kwargs)
 

litellm/batches/batch_utils.py

@@ -118,6 +118,7 @@ def _batch_cost_calculator(
     total_cost = _get_batch_job_cost_from_file_content(
         file_content_dictionary=file_content_dictionary,
         custom_llm_provider=custom_llm_provider,
+        model_name=model_name,
         model_info=model_info,
     )
     verbose_logger.debug("total_cost=%s", total_cost)
@@ -363,6 +364,7 @@ def _count_entry_tokens(
 def _get_batch_job_cost_from_file_content(
     file_content_dictionary: List[dict],
     custom_llm_provider: Literal["openai", "azure", "vertex_ai", "hosted_vllm", "anthropic"] = "openai",
+    model_name: Optional[str] = None,
     model_info: Optional[ModelInfo] = None,
 ) -> float:
     """
@@ -377,9 +379,15 @@ def _get_batch_job_cost_from_file_content(
         for _item in file_content_dictionary:
             if _batch_response_was_successful(_item, custom_llm_provider):
                 _response_body = _get_response_from_batch_job_output_file(_item, custom_llm_provider)
-                if model_info is not None or custom_llm_provider == "anthropic":
+                if model_info is not None or custom_llm_provider in ("anthropic", "bedrock"):
                     usage = _get_batch_job_usage_from_response_body(_response_body, custom_llm_provider)
-                    model = _response_body.get("model", "")
+                    # Bedrock batch output lines report a short internal model id
+                    # (e.g. "claude-sonnet-4-6") that is not in the cost map; use the
+                    # deployment model name for pricing when available.
+                    if custom_llm_provider == "bedrock" and model_name:
+                        model = model_name
+                    else:
+                        model = _response_body.get("model") or model_name or ""
                     prompt_cost, completion_cost = batch_cost_calculator(
                         usage=usage,
                         model=model,
@@ -485,7 +493,7 @@ def _get_batch_job_usage_from_response_body(response_body: dict, custom_llm_prov
     """
     Get the tokens of a batch job from the response body
     """
-    if custom_llm_provider == "anthropic":
+    if custom_llm_provider in ("anthropic", "bedrock"):
         from litellm.llms.anthropic.chat.transformation import AnthropicConfig
 
         return AnthropicConfig().calculate_usage(
@@ -513,6 +521,8 @@ def _get_response_from_batch_job_output_file(batch_job_output_file: dict, custom
     """
     if custom_llm_provider == "anthropic":
         return _get_anthropic_result_from_batch_results_line(batch_job_output_file).get("message", None) or {}
+    if custom_llm_provider == "bedrock":
+        return batch_job_output_file.get("modelOutput", None) or {}
     _response: dict = batch_job_output_file.get("response", None) or {}
     _response_body = _response.get("body", None) or {}
     return _response_body
@@ -523,9 +533,12 @@ def _batch_response_was_successful(batch_job_output_file: dict, custom_llm_provi
     Check if the batch job response was successful
 
     OpenAI-shaped output rows report ``response.status_code == 200``; Anthropic
-    message batch results lines report ``result.type == "succeeded"``.
+    message batch results lines report ``result.type == "succeeded"``; Bedrock
+    batch output lines report ``modelOutput`` (and no ``error``).
     """
     if custom_llm_provider == "anthropic":
         return _get_anthropic_result_from_batch_results_line(batch_job_output_file).get("type") == "succeeded"
+    if custom_llm_provider == "bedrock":
+        return batch_job_output_file.get("modelOutput") is not None and batch_job_output_file.get("error") is None
     _response: dict = batch_job_output_file.get("response", None) or {}
     return _response.get("status_code", None) == 200

litellm/constants.py

@@ -715,6 +715,7 @@ openai_compatible_endpoints: List = [
     "https://api.clarifai.com/v2/ext/openai/v1",
     "https://api.libertai.io/v1",
     "https://pinstripes.io/v1",
+    "https://api.meta.ai/v1",
 ]
 
 
@@ -781,6 +782,7 @@ openai_compatible_providers: List = [
     "ragflow",
     "pinstripes",  # Pinstripes - JSON-configured provider
     "darkbloom",
+    "meta",  # Meta Model API (Muse Spark) - JSON-configured provider
 ]
 openai_text_completion_compatible_providers: List = [  # providers that support `/v1/completions`
     "together_ai",

litellm/cost_calculator.py

@@ -760,7 +760,11 @@ def _select_model_name_for_cost_calc(
     if custom_pricing is True:
         if router_model_id is not None and router_model_id in litellm.model_cost:
             entry = litellm.model_cost[router_model_id]
-            if entry.get("input_cost_per_token") is not None or entry.get("input_cost_per_second") is not None:
+            if (
+                entry.get("input_cost_per_token") is not None
+                or entry.get("input_cost_per_second") is not None
+                or entry.get("tiered_pricing") is not None
+            ):
                 return_model = router_model_id
             else:
                 return_model = model

litellm/experimental_mcp_client/client.py

@@ -382,15 +382,25 @@ class MCPClient:
                 if root_cause is not None and isinstance(in_flight_error, asyncio.CancelledError):
                     raise root_cause from in_flight_error
 
-    async def run_with_session(self, operation: Callable[[ClientSession], Awaitable[TSessionResult]]) -> TSessionResult:
-        """Open a session, run the provided coroutine, and clean up."""
+    async def run_with_session(
+        self,
+        operation: Callable[[ClientSession], Awaitable[TSessionResult]],
+        *,
+        quiet_on_error: bool = False,
+    ) -> TSessionResult:
+        """Open a session, run the provided coroutine, and clean up.
+
+        quiet_on_error demotes the failure line to debug for callers that own the exception
+        (call_tool / list_tools under raise_on_error), so an expected pass-through re-auth does
+        not emit a warning per call; every other caller keeps the operator-visible warning."""
         http_client: Optional[httpx.AsyncClient] = None
         try:
             self._last_initialize_instructions = None
             transport_ctx, http_client = self._create_transport_context()
             return await self._execute_session_operation(transport_ctx, operation)
         except Exception:
-            verbose_logger.warning("MCP client run_with_session failed for %s", self.server_url or "stdio")
+            _log = verbose_logger.debug if quiet_on_error else verbose_logger.warning
+            _log("MCP client run_with_session failed for %s", self.server_url or "stdio")
             raise
         finally:
             if http_client is not None:
@@ -491,7 +501,7 @@ class MCPClient:
             return await session.list_tools()
 
         try:
-            result = await self.run_with_session(_list_tools_operation)
+            result = await self.run_with_session(_list_tools_operation, quiet_on_error=raise_on_error)
             tool_count = len(result.tools)
             tool_names = [tool.name for tool in result.tools]
             verbose_logger.info(f"MCP client listed {tool_count} tools from {self.server_url or 'stdio'}: {tool_names}")
@@ -501,7 +511,13 @@ class MCPClient:
             raise
         except Exception as e:
             error_type = type(e).__name__
-            verbose_logger.exception(
+            # Mirror call_tool: when the caller opted into raise_on_error it owns the exception and
+            # logs it at the fitting level (an expected pass-through re-auth 401 is info, not an
+            # error), so log at debug here to avoid an error-level line + traceback that would trip
+            # error-rate alerts on that expected signal. The swallow path still logs the full
+            # exception because nothing downstream will surface the failure.
+            _log = verbose_logger.debug if raise_on_error else verbose_logger.exception
+            _log(
                 f"MCP client list_tools failed - "
                 f"Error Type: {error_type}, "
                 f"Error: {str(e)}, "
@@ -510,7 +526,8 @@ class MCPClient:
             )
             # Check if it's a stream/connection error
             if "BrokenResourceError" in error_type or "Broken" in error_type:
-                verbose_logger.error(
+                _log_broken = verbose_logger.debug if raise_on_error else verbose_logger.error
+                _log_broken(
                     "MCP client detected broken connection/stream during list_tools - "
                     "the MCP server may have crashed, disconnected, or timed out"
                 )
@@ -567,7 +584,7 @@ class MCPClient:
             )
 
         try:
-            tool_result = await self.run_with_session(_call_tool_operation)
+            tool_result = await self.run_with_session(_call_tool_operation, quiet_on_error=raise_on_error)
             verbose_logger.info(f"MCP client tool call '{call_tool_request_params.name}' completed successfully")
             return tool_result
         except asyncio.CancelledError:
@@ -580,7 +597,13 @@ class MCPClient:
             verbose_logger.debug(f"MCP client tool call traceback:\n{error_trace}")
             # Log detailed error information
             error_type = type(e).__name__
-            verbose_logger.error(
+            # When the caller opted into raise_on_error it owns the exception and logs it at the
+            # level that fits (an expected pass-through re-auth 401 is info, not an operator-actionable
+            # error), so log at debug here to avoid an error-level line that would trip error-rate
+            # alerts on that expected signal. The swallow path (raise_on_error=False) still logs at
+            # error because nothing downstream will surface the failure.
+            _log = verbose_logger.debug if raise_on_error else verbose_logger.error
+            _log(
                 f"MCP client call_tool failed - "
                 f"Error Type: {error_type}, "
                 f"Error: {str(e)}, "
@@ -590,7 +613,7 @@ class MCPClient:
             )
             # Check if it's a stream/connection error
             if "BrokenResourceError" in error_type or "Broken" in error_type:
-                verbose_logger.error(
+                _log(
                     "MCP client detected broken connection/stream - "
                     "the MCP server may have crashed, disconnected, or timed out."
                 )

litellm/integrations/custom_guardrail.py

@@ -1,3 +1,4 @@
+import os
 import secrets
 from datetime import datetime
 from typing import (
@@ -17,6 +18,7 @@ from litellm._logging import verbose_logger
 from litellm.litellm_core_utils.core_helpers import redact_nested_match_and_regex_keys
 from litellm.caching import DualCache
 from litellm.integrations.custom_logger import CustomLogger
+from litellm.secret_managers.main import str_to_bool
 from litellm.types.guardrails import (
     DynamicGuardrailParams,
     GuardrailEventHooks,
@@ -59,6 +61,20 @@ from litellm.exceptions import (
 _PRE_CALL_EXECUTED_TOKEN = secrets.token_hex(16)
 
 
+def _strict_guardrail_modes_enabled() -> bool:
+    """Whether guardrail-mode validation raises (default) or logs a warning.
+
+    Set `LITELLM_STRICT_GUARDRAIL_MODES=false` to keep the pre-LIT-4226 behavior
+    for guardrails whose supported_event_hooks list newly includes their
+    configured mode: log the mismatch and continue instead of raising at boot.
+    """
+    raw = os.environ.get("LITELLM_STRICT_GUARDRAIL_MODES")
+    if raw is None:
+        return True
+    parsed = str_to_bool(raw)
+    return True if parsed is None else parsed
+
+
 def get_session_id_from_request_data(request_data: Dict[str, Any]) -> Optional[str]:
     """Extract session_id from request data (litellm_session_id or metadata)."""
     session_id = request_data.get("litellm_session_id")
@@ -132,7 +148,17 @@ class CustomGuardrail(CustomLogger):
 
         if supported_event_hooks:
             ## validate event_hook is in supported_event_hooks
-            self._validate_event_hook(event_hook, supported_event_hooks)
+            try:
+                self._validate_event_hook(event_hook, supported_event_hooks)
+            except ValueError as validation_error:
+                if _strict_guardrail_modes_enabled():
+                    raise
+                verbose_logger.warning(
+                    "%s. LITELLM_STRICT_GUARDRAIL_MODES=false; continuing "
+                    "with unsupported event_hook. Set the env var to true "
+                    "(default) to enforce validation and fail at startup.",
+                    validation_error,
+                )
         super().__init__(**kwargs)
 
     def render_violation_message(self, default: str, context: Optional[Dict[str, Any]] = None) -> str:
@@ -303,6 +329,18 @@ class CustomGuardrail(CustomLogger):
         """
         return None
 
+    @classmethod
+    def get_supported_event_hooks(cls) -> Optional[List[GuardrailEventHooks]]:
+        """
+        Returns the event hooks this guardrail supports, for the UI to render.
+
+        Subclasses should override to return their supported hooks list. When a
+        subclass returns None, the endpoint omits it from the per-provider map
+        and the UI is expected to fall back to the global `supported_modes`
+        list client-side.
+        """
+        return None
+
     def _validate_event_hook(
         self,
         event_hook: Optional[Union[GuardrailEventHooks, List[GuardrailEventHooks], Mode]],
@@ -477,6 +515,22 @@ class CustomGuardrail(CustomLogger):
                     return True
         return False
 
+    def uses_apply_guardrail_interface(self) -> bool:
+        return type(self).apply_guardrail is not CustomGuardrail.apply_guardrail
+
+    def _deployment_pre_call_target(self) -> "CustomLogger":
+        if not self.uses_apply_guardrail_interface():
+            return self
+        try:
+            from litellm.proxy.utils import unified_guardrail
+        except ImportError as e:
+            raise ImportError(
+                f"Guardrail {self.guardrail_name or type(self).__name__} implements apply_guardrail, which needs "
+                "the litellm proxy dependencies to run at the deployment level. "
+                "Install them with: pip install 'litellm[proxy]'"
+            ) from e
+        return unified_guardrail
+
     async def async_pre_call_deployment_hook(
         self, kwargs: Dict[str, Any], call_type: Optional[CallTypes]
     ) -> Optional[dict]:
@@ -495,7 +549,10 @@ class CustomGuardrail(CustomLogger):
 
         # CHECK IF GUARDRAIL REJECTS THE REQUEST
         if call_type == CallTypes.completion or call_type == CallTypes.acompletion:
-            result = await self.async_pre_call_hook(
+            target = self._deployment_pre_call_target()
+            if target is not self:
+                kwargs["guardrail_to_apply"] = self
+            result = await target.async_pre_call_hook(
                 user_api_key_dict=UserAPIKeyAuth(
                     user_id=kwargs.get("user_api_key_user_id"),
                     team_id=kwargs.get("user_api_key_team_id"),
@@ -505,7 +562,7 @@ class CustomGuardrail(CustomLogger):
                 ),
                 cache=dc,
                 data=kwargs,
-                call_type=call_type.value or "acompletion",  # type: ignore
+                call_type="completion" if call_type == CallTypes.completion else "acompletion",
             )
 
             if result is not None and isinstance(result, dict):
@@ -757,6 +814,12 @@ class CustomGuardrail(CustomLogger):
         # raw provider JSON so redaction is not duplicated upstream).
         clean_guardrail_response = redact_nested_match_and_regex_keys(clean_guardrail_response)
 
+        from litellm.litellm_core_utils.sensitive_data_masker import (
+            mask_credentials_in_payload,
+        )
+
+        clean_guardrail_response = mask_credentials_in_payload(clean_guardrail_response)
+
         slg = StandardLoggingGuardrailInformation(
             guardrail_name=self.guardrail_name,
             guardrail_provider=guardrail_provider,

litellm/integrations/datadog/datadog.py

@@ -19,7 +19,7 @@ import os
 import time
 import traceback
 from datetime import datetime as datetimeObj
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, List, Optional, Sequence, Union
 
 import httpx
 from httpx import Response
@@ -50,6 +50,7 @@ from litellm.types.integrations.base_health_check import IntegrationHealthCheckS
 from litellm.types.integrations.datadog import (
     DD_ERRORS,
     DD_MAX_BATCH_SIZE,
+    DD_MAX_PAYLOAD_SIZE_BYTES,
     DataDogStatus,
     DatadogInitParams,
     DatadogPayload,
@@ -384,8 +385,10 @@ class DataDogLogger(
 
     async def _send_with_413_split(self, batch: List) -> List:
         """
-        Send a batch, halving any sub-batch that 413s (payload too large) and retrying the
-        halves, since Datadog enforces a 5MB uncompressed limit per request.
+        Send a batch, halving any sub-batch that exceeds Datadog's intake limits before
+        sending, and halving again on a 413 (payload too large) response, since Datadog
+        enforces a 5MB uncompressed limit per request. The proactive split avoids paying
+        a serialize + gzip + round trip for a payload the intake is guaranteed to reject.
 
         A 413 surfaces as a raised MaskedHTTPStatusError (httpx raise_for_status), not a
         returned response, so both paths are handled. A lone event that still 413s is
@@ -398,6 +401,11 @@ class DataDogLogger(
             chunk = pending.pop()
             if not chunk:
                 continue
+            if len(chunk) > 1 and self._exceeds_intake_limits(chunk):
+                mid = len(chunk) // 2
+                pending.append(chunk[mid:])
+                pending.append(chunk[:mid])
+                continue
             try:
                 response = await self.async_send_compressed_data(chunk)
             except Exception as e:
@@ -436,6 +444,21 @@ class DataDogLogger(
     def _undelivered(chunk: List, pending: List[List]) -> List:
         return chunk + [event for remaining in reversed(pending) for event in remaining]
 
+    @staticmethod
+    def _exceeds_intake_limits(chunk: Sequence[DatadogPayload]) -> bool:
+        """
+        True when a chunk would breach Datadog's log intake limits: more than
+        DD_MAX_BATCH_SIZE events per payload, or a serialized size above
+        DD_MAX_PAYLOAD_SIZE_BYTES (held under Datadog's 5MB uncompressed cap so
+        the batch is split before the intake rejects it with a 413).
+        """
+        from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
+
+        if len(chunk) > DD_MAX_BATCH_SIZE:
+            return True
+        payload_size_bytes = len(safe_dumps(chunk).encode("utf-8"))
+        return payload_size_bytes > DD_MAX_PAYLOAD_SIZE_BYTES
+
     async def flush_queue(self):
         if self.flush_lock is None:
             return

litellm/integrations/otel/README.md

@@ -223,6 +223,15 @@ lives in [`plumbing/`](./plumbing):
   readers/exporters receive them alongside the server metrics, and one is built
   and registered as the global only when none is set (mirroring how V2 owns trace
   export).
+- [`events.py`](./plumbing/events.py) — GenAI client events. Gated on
+  `enable_events` (`LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS`), a failed LLM call
+  records the semconv `gen_ai.client.operation.exception` log event at severity
+  WARN, carrying `exception.type` / `exception.message` / `exception.stacktrace`
+  and correlated to the failed span through the trace and span ids. The
+  `LoggerProvider` is resolved like the meter provider, except that an explicit
+  `NoOpLoggerProvider` global is an operator opt-out that builds no recorder at
+  all. The deprecated `error.*` span attributes and the `exception` span event
+  are still stamped by the emitter for backwards compatibility.
 
 ### Adapter
 

litellm/integrations/otel/emitter.py

@@ -18,6 +18,7 @@ from litellm.integrations.otel.model.payloads import (
     ServiceSpanData,
     SpanError,
 )
+from litellm.integrations.otel.plumbing.events import GenAIEventRecorder
 from litellm.integrations.otel.plumbing.providers import to_otel_span_kind
 from litellm.integrations.otel.model.semconv import Error, ExceptionEvent, LiteLLMError
 from litellm.integrations.otel.model.spans import (
@@ -77,9 +78,11 @@ class SpanEmitter:
         tracer: Tracer,
         config: OpenTelemetryV2Config,
         mappers: Sequence[AttributeMapper] | None = None,
+        event_recorder: GenAIEventRecorder | None = None,
     ) -> None:
         self._tracer = tracer
         self._config = config
+        self._event_recorder = event_recorder
         # The mapper chain is the sole source of span attributes. When not
         # passed in, resolve it from the config so there's one source of truth.
         self._mappers: list[AttributeMapper] = (
@@ -223,6 +226,14 @@ class SpanEmitter:
                 ExceptionEvent.NAME,
                 {ExceptionEvent.TYPE: error_type, ExceptionEvent.MESSAGE: message},
             )
+            if self._event_recorder is not None and role is SpanRole.LLM_CALL:
+                self._event_recorder.record_operation_exception(
+                    span_context=span.get_span_context(),
+                    error_type=error_type,
+                    message=message,
+                    stack_trace=error.stack_trace,
+                    timestamp_ns=end_time_ns,
+                )
         # On success leave the status UNSET (the semconv default) rather than
         # forcing OK — that matches the FastAPI server span and avoids implying a
         # span-level health signal litellm doesn't actually evaluate. Only a

litellm/integrations/otel/logger.py

@@ -6,6 +6,7 @@ from datetime import datetime
 from typing import TYPE_CHECKING, Any, Callable, Iterator, Mapping, Sequence, cast
 
 from opentelemetry.context import Context, attach, get_current
+from opentelemetry.sdk._logs import LoggerProvider
 from opentelemetry.sdk.trace import TracerProvider
 from opentelemetry.trace import Span, Tracer, get_current_span, use_span
 
@@ -40,14 +41,17 @@ from litellm.integrations.otel.model.payloads import (
     is_mcp_list_tools,
     is_mcp_tool_call,
 )
+from litellm.integrations.otel.plumbing.events import GenAIEventRecorder
 from litellm.integrations.otel.plumbing.metrics import (
     GenAIMetricRecorder,
     create_genai_metrics,
 )
 from litellm.integrations.otel.plumbing.providers import (
     build_tracer_provider,
+    get_event_logger,
     get_meter,
     get_tracer,
+    resolve_logger_provider,
     resolve_meter_provider,
 )
 from litellm.integrations.otel.plumbing.routing import TenantTracerCache
@@ -104,7 +108,7 @@ class OpenTelemetryV2(CustomLogger):
         config: OpenTelemetryV2Config | None = None,
         callback_name: str | None = None,
         tracer_provider: TracerProvider | None = None,
-        logger_provider: Any | None = None,  # reserved for OTel logs
+        logger_provider: LoggerProvider | None = None,
         meter_provider: Any | None = None,
         **kwargs: Any,
     ) -> None:
@@ -117,7 +121,12 @@ class OpenTelemetryV2(CustomLogger):
         self.tracer: Tracer = get_tracer(self._tracer_provider, LITELLM_TRACER_NAME)
         self._metrics_recorder = self._init_metrics(meter_provider)
         self._metric_filter_error_logged = False
-        self._emitter = SpanEmitter(self.tracer, self.config, mappers=resolve_mappers(self.config.mapper_names))
+        self._emitter = SpanEmitter(
+            self.tracer,
+            self.config,
+            mappers=resolve_mappers(self.config.mapper_names),
+            event_recorder=self._init_events(logger_provider),
+        )
         self._tenant_tracers = TenantTracerCache(self.config, callback_name, LITELLM_TRACER_NAME)
         self._open_llm_calls: "OrderedDict[str, _LLMCallSpan]" = OrderedDict()
         self._init_otel_logger_on_litellm_proxy()
@@ -136,6 +145,22 @@ class OpenTelemetryV2(CustomLogger):
         meter = get_meter(provider, LITELLM_TRACER_NAME)
         return GenAIMetricRecorder(create_genai_metrics(meter), self.callback_name)
 
+    def _init_events(self, logger_provider: LoggerProvider | None) -> "GenAIEventRecorder | None":
+        """Create the GenAI event recorder when events are enabled, else ``None``.
+
+        ``logger_provider`` is an explicit override (tests inject one); otherwise the
+        provider is resolved from the OTel global so an operator-configured logs
+        pipeline receives the events, building and registering one only when no
+        global provider is set. A ``None`` resolution means the operator opted out
+        of the logs signal, so no recorder is built.
+        """
+        if not self.config.enable_events:
+            return None
+        provider = resolve_logger_provider(self.config, logger_provider)
+        if provider is None:
+            return None
+        return GenAIEventRecorder(get_event_logger(provider, LITELLM_TRACER_NAME))
+
     # ====================================================================== #
     #  Proxy global registration
     # ====================================================================== #

litellm/integrations/otel/model/semconv.py

@@ -177,6 +177,19 @@ class ExceptionEvent:
     NAME: Final = "exception"
     TYPE: Final = "exception.type"
     MESSAGE: Final = "exception.message"
+    STACKTRACE: Final = "exception.stacktrace"
+
+
+class GenAIEvent:
+    """GenAI semconv event names, from the GenAI registry's *events* section.
+
+    ``gen_ai.client.operation.exception`` is defined as a log-based event
+    (severity WARN) carrying the ``exception.*`` trio, correlated to the failed
+    span via the trace/span ids — the semconv-compliant home for GenAI failure
+    details, unlike the deprecated ``error.message`` span attribute.
+    """
+
+    OPERATION_EXCEPTION: Final = "gen_ai.client.operation.exception"
 
 
 class Server:

litellm/integrations/otel/plumbing/events.py

@@ -0,0 +1,52 @@
+"""GenAI client events: the ``gen_ai.client.operation.exception`` log event.
+
+The GenAI semantic conventions define exception recording for client
+operations as a log-based event (severity WARN) carrying the ``exception.*``
+attribute trio, correlated to the failed span through the trace/span ids —
+not as a span attribute or span event. This module owns building and
+emitting that event; the exporter pipeline it rides is built in
+:mod:`litellm.integrations.otel.plumbing.providers`.
+"""
+
+from dataclasses import dataclass
+
+from opentelemetry._events import Event, EventLogger
+from opentelemetry._logs.severity import SeverityNumber
+from opentelemetry.trace import SpanContext
+
+from litellm.integrations.otel.model.semconv import ExceptionEvent, GenAIEvent
+
+
+@dataclass(frozen=True, slots=True)
+class GenAIEventRecorder:
+    event_logger: EventLogger
+
+    def record_operation_exception(
+        self,
+        span_context: SpanContext,
+        error_type: str,
+        message: str,
+        stack_trace: str | None,
+        timestamp_ns: int | None,
+    ) -> None:
+        # ``exception.type`` and ``exception.message`` are the semconv-required
+        # pair and always ride the event; only the recommended stacktrace is
+        # conditional on the payload carrying one.
+        stacktrace = ((ExceptionEvent.STACKTRACE, stack_trace),) if stack_trace else ()
+        self.event_logger.emit(
+            Event(
+                name=GenAIEvent.OPERATION_EXCEPTION,
+                timestamp=timestamp_ns,
+                trace_id=span_context.trace_id,
+                span_id=span_context.span_id,
+                trace_flags=span_context.trace_flags,
+                severity_number=SeverityNumber.WARN,
+                attributes=dict(
+                    (
+                        (ExceptionEvent.TYPE, error_type),
+                        (ExceptionEvent.MESSAGE, message),
+                        *stacktrace,
+                    )
+                ),
+            )
+        )

litellm/integrations/otel/plumbing/providers.py

@@ -2,9 +2,20 @@
 
 from typing import TYPE_CHECKING, Any, Callable, Iterable
 
-from opentelemetry import baggage, metrics
+from opentelemetry import _logs, baggage, metrics
+from opentelemetry._events import EventLogger
+from opentelemetry._logs import LoggerProvider, NoOpLoggerProvider
 from opentelemetry.context import Context
 from opentelemetry.metrics import MeterProvider, NoOpMeterProvider
+from opentelemetry.sdk._events import EventLoggerProvider
+from opentelemetry.sdk._logs import LoggerProvider as SDKLoggerProvider
+from opentelemetry.sdk._logs.export import (
+    BatchLogRecordProcessor,
+    ConsoleLogExporter,
+    InMemoryLogExporter,
+    LogExporter,
+    SimpleLogRecordProcessor,
+)
 from opentelemetry.sdk.metrics import MeterProvider as SDKMeterProvider
 from opentelemetry.sdk.resources import Resource
 from opentelemetry.sdk.trace import ReadableSpan, SpanProcessor, TracerProvider
@@ -224,6 +235,112 @@ def build_metric_reader(config: OpenTelemetryV2Config) -> "MetricReader":
     return PeriodicExportingMetricReader(exporter, export_interval_millis=5000)
 
 
+def _otlp_logs_endpoint(endpoint: str | None) -> str | None:
+    """Point an OTLP/HTTP base endpoint at the ``/v1/logs`` signal path.
+
+    The OTLP/HTTP exporter only appends ``/v1/logs`` when it reads
+    ``OTEL_EXPORTER_OTLP_ENDPOINT`` itself; an explicitly passed endpoint is used
+    verbatim, so a base URL would POST to the root. Mirror ``_otlp_traces_endpoint``
+    for the logs signal (rewriting a sibling signal path when present).
+    """
+    if not endpoint:
+        return endpoint
+    endpoint = endpoint.rstrip("/")
+    if endpoint.endswith("/v1/logs"):
+        return endpoint
+    for other_signal in ("/v1/traces", "/v1/metrics"):
+        if endpoint.endswith(other_signal):
+            return endpoint[: -len(other_signal)] + "/v1/logs"
+    return endpoint + "/v1/logs"
+
+
+def build_log_exporter(config: OpenTelemetryV2Config) -> LogExporter:
+    """Build a log exporter mirroring the exporter selection of the other signals.
+
+    ``console`` (and any unrecognized kind) exports to the console; ``otlp_http``
+    and ``otlp_grpc`` export over OTLP with the configured endpoint/headers;
+    ``in_memory`` buffers for tests. Like GenAI metrics, events ride the
+    single-destination shorthand fields, not the multi-exporter ``exporters`` list.
+    """
+    kind = (config.exporter or "console").lower()
+    if kind in ("in_memory", "inmemory", "memory"):
+        return InMemoryLogExporter()
+    if kind in ("otlp_http", "http", "http/protobuf", "http/json"):
+        from opentelemetry.exporter.otlp.proto.http._log_exporter import (
+            OTLPLogExporter as HTTPLogExporter,
+        )
+
+        return HTTPLogExporter(
+            endpoint=_otlp_logs_endpoint(config.endpoint),
+            headers=parse_headers(config.headers),
+        )
+    if kind in ("otlp_grpc", "grpc"):
+        try:
+            from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
+                OTLPLogExporter as GRPCLogExporter,
+            )
+        except ImportError as exc:
+            raise ImportError(
+                "OpenTelemetry OTLP gRPC log exporter is not available. Install "
+                "`opentelemetry-exporter-otlp` and `grpcio` (or `litellm[grpc]`)."
+            ) from exc
+
+        return GRPCLogExporter(endpoint=config.endpoint, headers=parse_headers(config.headers))
+    return ConsoleLogExporter()
+
+
+def build_logger_provider(
+    config: OpenTelemetryV2Config,
+    log_exporter: LogExporter | None = None,
+) -> SDKLoggerProvider:
+    """Build the :class:`LoggerProvider` GenAI events export through.
+
+    ``log_exporter`` is an explicit override (tests inject an
+    ``InMemoryLogExporter``); otherwise the exporter is selected from the config's
+    exporter kind via :func:`build_log_exporter`. Console and in-memory exporters
+    get a Simple processor (synchronous export, which tests rely on), everything
+    else a Batch processor — the same split as span processing.
+    """
+    exporter = log_exporter if log_exporter is not None else build_log_exporter(config)
+    provider = SDKLoggerProvider(resource=build_resource(config))
+    use_simple = isinstance(exporter, (ConsoleLogExporter, InMemoryLogExporter))
+    provider.add_log_record_processor(
+        SimpleLogRecordProcessor(exporter) if use_simple else BatchLogRecordProcessor(exporter)
+    )
+    return provider
+
+
+def resolve_logger_provider(
+    config: OpenTelemetryV2Config,
+    logger_provider: SDKLoggerProvider | None = None,
+) -> SDKLoggerProvider | None:
+    """Resolve the :class:`LoggerProvider` GenAI events record through, or ``None``
+    when the operator has opted out of the logs signal.
+
+    Same resolution order as :func:`resolve_meter_provider`: an injected provider
+    wins (DI/tests); an operator-configured SDK global is reused so events ride
+    their pipeline; an explicit ``NoOpLoggerProvider`` global is an opt-out and
+    yields ``None``, so no event is ever built. Only the default placeholder
+    global makes V2 build a provider from the config and publish it as the global.
+    """
+    if logger_provider is not None:
+        return logger_provider
+
+    existing: LoggerProvider = _logs.get_logger_provider()
+    if isinstance(existing, SDKLoggerProvider):
+        return existing
+    if isinstance(existing, NoOpLoggerProvider):
+        return None
+
+    provider = build_logger_provider(config)
+    _logs.set_logger_provider(provider)
+    return provider
+
+
+def get_event_logger(provider: SDKLoggerProvider, name: str = "litellm") -> EventLogger:
+    return EventLoggerProvider(logger_provider=provider).get_event_logger(name, litellm_version)
+
+
 def build_meter_provider(
     config: OpenTelemetryV2Config,
     metric_reader: "MetricReader | None" = None,

litellm/integrations/prometheus.py

@@ -239,6 +239,18 @@ class PrometheusLogger(CustomLogger):
                 labelnames=self.get_labels_for_metric("litellm_output_audio_tokens_metric"),
             )
 
+            self.litellm_video_duration_seconds_metric = self._counter_factory(
+                "litellm_video_duration_seconds_metric",
+                "Seconds of video generated, from usage.duration_seconds on video generation calls",
+                labelnames=self.get_labels_for_metric("litellm_video_duration_seconds_metric"),
+            )
+
+            self.litellm_images_generated_metric = self._counter_factory(
+                "litellm_images_generated_metric",
+                "Number of images generated, from the image generation response",
+                labelnames=self.get_labels_for_metric("litellm_images_generated_metric"),
+            )
+
             # Remaining Budget for Team
             self.litellm_remaining_team_budget_metric = self._gauge_factory(
                 "litellm_remaining_team_budget_metric",
@@ -1336,6 +1348,12 @@ class PrometheusLogger(CustomLogger):
             label_context=label_context,
         )
 
+        self._increment_media_generation_metrics(
+            standard_logging_payload=standard_logging_payload,
+            enum_values=enum_values,
+            label_context=label_context,
+        )
+
         # MCP tool call metrics
         self._increment_mcp_tool_call_metrics(
             standard_logging_payload=standard_logging_payload,
@@ -1459,8 +1477,65 @@ class PrometheusLogger(CustomLogger):
             ),
         ]
 
-        for counter, metric_name, value in detail_metrics:
-            if not isinstance(value, (int, float)) or value <= 0:
+        PrometheusLogger._inc_sparse_usage_counters(
+            self,
+            detail_metrics,
+            enum_values=enum_values,
+            label_context=label_context,
+        )
+
+    def _increment_media_generation_metrics(
+        self,
+        standard_logging_payload: StandardLoggingPayload,
+        enum_values: UserAPIKeyLabelValues,
+        label_context: PrometheusLabelFactoryContext | None = None,
+    ) -> None:
+        """
+        Increment video-seconds and images-generated counters from
+        ``standard_logging_payload["metadata"]["usage_object"]``. Video
+        providers report ``duration_seconds`` there; image generation calls
+        report ``output_image_count``. Both are sparse: only emitted when the
+        value is present and > 0, so token-only call types are unaffected.
+        """
+        metadata = standard_logging_payload.get("metadata") or {}
+        usage_object = metadata.get("usage_object") if isinstance(metadata, dict) else None
+        if not isinstance(usage_object, dict):
+            return
+
+        media_metrics: list[tuple[Any, DEFINED_PROMETHEUS_METRICS, Any]] = [
+            (
+                self.litellm_video_duration_seconds_metric,
+                "litellm_video_duration_seconds_metric",
+                usage_object.get("duration_seconds"),
+            ),
+            (
+                self.litellm_images_generated_metric,
+                "litellm_images_generated_metric",
+                usage_object.get("output_image_count"),
+            ),
+        ]
+
+        PrometheusLogger._inc_sparse_usage_counters(
+            self,
+            media_metrics,
+            enum_values=enum_values,
+            label_context=label_context,
+        )
+
+    def _inc_sparse_usage_counters(
+        self,
+        counters_with_values: list[tuple[Any, DEFINED_PROMETHEUS_METRICS, Any]],
+        enum_values: UserAPIKeyLabelValues,
+        label_context: PrometheusLabelFactoryContext | None = None,
+    ) -> None:
+        """
+        Increment each ``(counter, metric_name, value)`` entry whose value is
+        a positive number. Non-numeric values (including booleans from
+        malformed provider usage dicts) and values <= 0 are skipped, keeping
+        scrape output sparse.
+        """
+        for counter, metric_name, value in counters_with_values:
+            if isinstance(value, bool) or not isinstance(value, (int, float)) or value <= 0:
                 continue
             PrometheusLogger._inc_labeled_counter(
                 self,
@@ -1618,6 +1693,14 @@ class PrometheusLogger(CustomLogger):
         user_id: Optional[str] = None,
         user_api_key_org_id: Optional[str] = None,
     ):
+        if (
+            isinstance(self.litellm_remaining_team_budget_metric, NoOpMetric)
+            and isinstance(self.litellm_remaining_api_key_budget_metric, NoOpMetric)
+            and isinstance(self.litellm_remaining_user_budget_metric, NoOpMetric)
+            and isinstance(self.litellm_remaining_org_budget_metric, NoOpMetric)
+        ):
+            return
+
         _metadata = litellm_params.get("metadata") or {}
         _team_spend = _metadata.get("user_api_key_team_spend", None)
         _team_max_budget = _metadata.get("user_api_key_team_max_budget", None)
@@ -1708,6 +1791,35 @@ class PrometheusLogger(CustomLogger):
             amount=float(response_cost),
         )
 
+    @staticmethod
+    def _get_remaining_from_v3_rate_limit_headers(
+        standard_logging_payload: StandardLoggingPayload | None,
+        rate_limit_type: Literal["requests", "tokens"],
+    ) -> int | None:
+        """
+        Read the per-(key, model) remaining value emitted by the v3 rate
+        limiter (``parallel_request_limiter_v3.py``), which writes
+        ``x-ratelimit-model_per_key-remaining-{requests,tokens}`` into
+        ``standard_logging_object.hidden_params.additional_headers`` instead
+        of the ``litellm-key-remaining-*`` metadata keys the legacy limiter
+        sets. The header carries no model group; it always refers to this
+        request's model group, which is what the gauges are labeled with.
+        Values are written in-process as plain ints (never HTTP-serialized
+        strings), so anything else is rejected rather than coerced.
+        """
+        if standard_logging_payload is None:
+            return None
+        hidden_params = standard_logging_payload.get("hidden_params")
+        if hidden_params is None:
+            return None
+        additional_headers = hidden_params.get("additional_headers")
+        if additional_headers is None:
+            return None
+        value = dict(additional_headers).get(f"x-ratelimit-model_per_key-remaining-{rate_limit_type}")
+        if isinstance(value, bool) or not isinstance(value, int):
+            return None
+        return value
+
     def _set_virtual_key_rate_limit_metrics(
         self,
         user_api_key: Optional[str],
@@ -1725,11 +1837,20 @@ class PrometheusLogger(CustomLogger):
         model_group = get_model_group_from_litellm_kwargs(kwargs)
         remaining_requests_variable_name = f"litellm-key-remaining-requests-{model_group}"
         remaining_tokens_variable_name = f"litellm-key-remaining-tokens-{model_group}"
+        standard_logging_payload: StandardLoggingPayload | None = kwargs.get("standard_logging_object")
 
         remaining_requests = metadata.get(remaining_requests_variable_name)
+        if remaining_requests is None:
+            remaining_requests = self._get_remaining_from_v3_rate_limit_headers(
+                standard_logging_payload=standard_logging_payload, rate_limit_type="requests"
+            )
         if remaining_requests is None:
             remaining_requests = sys.maxsize
         remaining_tokens = metadata.get(remaining_tokens_variable_name)
+        if remaining_tokens is None:
+            remaining_tokens = self._get_remaining_from_v3_rate_limit_headers(
+                standard_logging_payload=standard_logging_payload, rate_limit_type="tokens"
+            )
         if remaining_tokens is None:
             remaining_tokens = sys.maxsize
 
@@ -3332,6 +3453,9 @@ class PrometheusLogger(CustomLogger):
             - looks up team info from db if not available in metadata
         - Set team budget metrics
         """
+        if isinstance(self.litellm_remaining_team_budget_metric, NoOpMetric):
+            return
+
         if user_api_team:
             team_object = await self._assemble_team_object(
                 team_id=user_api_team,
@@ -3453,6 +3577,9 @@ class PrometheusLogger(CustomLogger):
         - Fetches org info via cache (get_org_object)
         - Sets org budget metrics
         """
+        if isinstance(self.litellm_remaining_org_budget_metric, NoOpMetric):
+            return
+
         if not org_id:
             return
 
@@ -3582,6 +3709,9 @@ class PrometheusLogger(CustomLogger):
         key_max_budget: Optional[float],
         key_spend: Optional[float],
     ):
+        if isinstance(self.litellm_remaining_api_key_budget_metric, NoOpMetric):
+            return
+
         if user_api_key:
             user_api_key_dict = await self._assemble_key_object(
                 user_api_key=user_api_key,
@@ -3642,6 +3772,9 @@ class PrometheusLogger(CustomLogger):
             - looks up user info from db if not available in metadata
         - Set user budget metrics
         """
+        if isinstance(self.litellm_remaining_user_budget_metric, NoOpMetric):
+            return
+
         if user_id:
             user_object = await self._assemble_user_object(
                 user_id=user_id,

litellm/integrations/rubrik.py

@@ -7,7 +7,7 @@ import time
 import urllib.parse
 import uuid
 from collections import Counter
-from typing import TYPE_CHECKING, Any, Literal, Optional
+from typing import TYPE_CHECKING, Any, List, Literal, Optional
 
 import httpx
 from litellm._logging import verbose_logger
@@ -52,6 +52,10 @@ class _MalformedToolBlockingResponseError(Exception):
 
 
 class RubrikLogger(CustomGuardrail, CustomBatchLogger):
+    @classmethod
+    def get_supported_event_hooks(cls) -> List[GuardrailEventHooks]:
+        return [GuardrailEventHooks.pre_call, GuardrailEventHooks.post_call]
+
     def __init__(
         self,
         api_key: str | None = None,
@@ -69,6 +73,7 @@ class RubrikLogger(CustomGuardrail, CustomBatchLogger):
         kwargs["event_hook"] = kwargs.get("event_hook") or GuardrailEventHooks.post_call
         if kwargs.get("default_on") is None:
             kwargs["default_on"] = True
+        kwargs.setdefault("supported_event_hooks", list(self.get_supported_event_hooks()))
         super().__init__(
             flush_lock=self.flush_lock,
             **kwargs,

litellm/integrations/s3.py

@@ -161,8 +161,13 @@ def get_s3_object_key(
     start_time: datetime,
     s3_file_name: str,
 ) -> str:
+    sanitized_s3_file_name = s3_file_name.replace("/", "_")
     s3_object_key = (
-        (s3_path.rstrip("/") + "/" if s3_path else "") + prefix + start_time.strftime("%Y-%m-%d") + "/" + s3_file_name
+        (s3_path.rstrip("/") + "/" if s3_path else "")
+        + prefix
+        + start_time.strftime("%Y-%m-%d")
+        + "/"
+        + sanitized_s3_file_name
     )  # we need the s3 key to include the time, so we log cache hits too
     s3_object_key += ".json"
     return s3_object_key

litellm/litellm_core_utils/cli_token_utils.py

@@ -7,6 +7,7 @@ This module has no dependencies on proxy code and can be safely imported at the
 
 import json
 import os
+import time
 from pathlib import Path
 from typing import Optional
 
@@ -68,3 +69,17 @@ def get_litellm_gateway_api_key(
         if stored_url != expected_base_url.rstrip("/"):
             return None
     return token_data["key"]
+
+
+def is_cli_token_fresh(token_data: dict, buffer_hours: float = 0.1) -> bool:
+    """Check whether a cached CLI token (as stored in token.json) is still
+    within its expiration window. Used by `lite auth print-token` to fail
+    fast, without a network round trip, once the cached token is past
+    `LITELLM_CLI_JWT_EXPIRATION_HOURS`."""
+    from litellm.constants import CLI_JWT_EXPIRATION_HOURS
+
+    timestamp = token_data.get("timestamp")
+    if not isinstance(timestamp, (int, float)):
+        return False
+    age_hours = (time.time() - timestamp) / 3600
+    return age_hours < (CLI_JWT_EXPIRATION_HOURS - buffer_hours)

litellm/litellm_core_utils/exception_mapping_utils.py

@@ -95,6 +95,9 @@ class ExceptionCheckers:
         if "current length is" in _error_str_lowercase and "while limit is" in _error_str_lowercase:
             return True
 
+        if "maximum input length is" in _error_str_lowercase and "tokens" in _error_str_lowercase:
+            return True
+
         return False
 
     @staticmethod

litellm/litellm_core_utils/fallback_generalizations.py

@@ -3,52 +3,69 @@ Declarative fallback generalizations for unknown / newly-released models.
 
 The ``fallback_generalizations`` block in ``model_prices_and_context_window.json``
 holds an ordered list of rules. Each rule pairs a single case-insensitive regex
-with the metadata to apply when a model name has no exact entry in the cost map.
-The metadata is a partial cost-map entry: ``litellm_provider`` drives provider
-routing, and the remaining fields (``mode``, ``supports_*``, context window,
-pricing, ...) drive ``get_model_info`` / ``supports_*``.
+with a ``model_info`` dict, and the structure of ``model_info`` decides which of
+two kinds the rule is.
 
-Precedence: rules are evaluated in file order and the first match wins. They are
-consulted only after exact and case-insensitive lookups miss, so an exact entry
-always takes precedence over a rule.
+A ROUTING rule carries exactly one ``model_info`` key, ``litellm_provider``. It is
+consumed only by ``get_llm_provider`` bare-id inference: the first routing rule
+whose regex matches decides the provider. Routing rules never contribute to model
+info.
+
+A CAPABILITY rule carries any ``model_info`` keys except ``litellm_provider``
+(``mode``, ``supports_*``, context window, pricing, ...). It is consumed by
+``get_model_info`` fallback resolution: the ``model_info`` of ALL capability rules
+whose regex matches is unioned in file order, with later rules overriding earlier
+ones on key conflicts, and the caller backfills ``litellm_provider`` with the
+provider it requested. If no capability rule matches, model-info resolution misses
+as if no rules existed.
+
+LEGACY-SCHEMA SHIM (temporary, until the new-schema JSON reaches main): released
+proxies fetch this JSON remotely from main, whose block still ships the old schema
+where a rule mixes ``litellm_provider`` with capability keys and may inherit a
+parent's ``model_info`` via ``extends``. Such a legacy rule is tolerated rather
+than skipped: ``extends`` is resolved once at install time (single level, against
+raw parents), and the resolved rule acts as BOTH kinds, a routing rule (its
+``litellm_provider`` participates in first-hit inference) and a capability rule
+(its full ``model_info``, provider included, participates in the union). New-schema
+rules never mix the two and never use ``extends``. A rule whose
+``litellm_provider`` is not a string is invalid and is warned about and skipped
+(a warning rather than a crash, for the same remote-fetch reason).
+
+Rules are only consulted after exact and case-insensitive lookups miss, so an
+exact cost-map entry always takes precedence over any rule.
 
 Patterns are matched case-insensitively with ``re.search`` and are not implicitly
-anchored: a rule must include ``^`` and ``$`` (as the shipped rules do) to bind to
-the whole model name, otherwise it matches as a substring. Keeping anchoring in the
-regex makes the rule the single, self-contained source of truth for what it matches.
-
-A rule may set ``extends`` to the ``name`` of another rule to inherit that rule's
-``model_info``; the rule's own ``model_info`` overrides the inherited keys, so a
-narrow rule (for example a version-gated capability flag) carries only its delta
-instead of duplicating the parent's pricing block. Inheritance is resolved once,
-at install time, against each rule's raw (unresolved) ``model_info``; it is a
-single level (a parent that itself extends is not chained).
+anchored: a rule must include ``^`` and ``$`` to bind to the whole model name,
+otherwise it matches as a substring. Keeping anchoring in the regex makes the rule
+the single, self-contained source of truth for what it matches.
 
 Any other keys on a rule (for example a free-text ``description`` documenting what
 the regex matches) are ignored by the engine and exist only for the reader.
 
-The compiled-regex list is built once and cached. ``match_fallback_generalization``
-is O(number of rules); callers must only invoke it on a cache miss.
+Rules are compiled and classified once, at install time. The match functions are
+O(number of rules); callers must only invoke them on a cache miss.
 """
 
 import re
-from typing import Optional
+from dataclasses import dataclass
+from typing import Optional, Union
 
 from litellm._logging import verbose_logger
 
 NAME_FIELD = "name"
 PATTERN_FIELD = "pattern"
 MODEL_INFO_FIELD = "model_info"
-EXTENDS_FIELD = "extends"
+PROVIDER_KEY = "litellm_provider"
+LEGACY_EXTENDS_FIELD = "extends"
 
 
-def _resolve_extends(rules: list) -> list:
-    """Expand ``extends`` inheritance so each rule's ``model_info`` is self-contained.
+def _resolve_legacy_extends(rules: list) -> list:
+    """Expand legacy ``extends`` inheritance so each rule's ``model_info`` is self-contained.
 
-    A rule with ``extends: <name>`` is rewritten with ``model_info`` set to the parent's
-    ``model_info`` overlaid by its own. Resolution is single-level and uses each rule's
-    raw ``model_info`` as the parent source. Non-dict rules and dangling parents are
-    passed through unchanged.
+    Compatibility shim for the old remote schema: single level, resolved against each
+    parent's raw ``model_info``, with the child's own keys winning on conflict. Non-dict
+    rules and dangling parents pass through unchanged; new-schema rules carry no
+    ``extends`` and are untouched.
     """
     base_by_name = {
         rule[NAME_FIELD]: rule[MODEL_INFO_FIELD]
@@ -58,84 +75,138 @@ def _resolve_extends(rules: list) -> list:
         and isinstance(rule.get(MODEL_INFO_FIELD), dict)
     }
 
-    def resolved(rule: dict) -> dict:
-        parent_name = rule.get(EXTENDS_FIELD)
+    def resolved(rule: object) -> object:
+        if not isinstance(rule, dict):
+            return rule
+        parent_name = rule.get(LEGACY_EXTENDS_FIELD)
         own_info = rule.get(MODEL_INFO_FIELD)
         parent_info = base_by_name.get(parent_name) if isinstance(parent_name, str) else None
         if parent_info is None or not isinstance(own_info, dict):
             return rule
         return {**rule, MODEL_INFO_FIELD: {**parent_info, **own_info}}
 
-    return [resolved(rule) if isinstance(rule, dict) else rule for rule in rules]
+    return [resolved(rule) for rule in rules]
+
+
+@dataclass(frozen=True, slots=True)
+class _RoutingRule:
+    pattern: re.Pattern
+    provider: str
+
+
+@dataclass(frozen=True, slots=True)
+class _CapabilityRule:
+    pattern: re.Pattern
+    model_info: dict
+
+
+_CompiledRule = Union[_RoutingRule, _CapabilityRule]
+
+
+def _compile_rule(rule: object) -> tuple[_CompiledRule, ...]:
+    if not isinstance(rule, dict):
+        return ()
+    pattern = rule.get(PATTERN_FIELD)
+    model_info = rule.get(MODEL_INFO_FIELD)
+    if not isinstance(pattern, str) or not isinstance(model_info, dict):
+        verbose_logger.warning(
+            "LiteLLM: skipping malformed fallback generalization rule %s (needs string '%s' and dict '%s').",
+            rule.get(NAME_FIELD, pattern),
+            PATTERN_FIELD,
+            MODEL_INFO_FIELD,
+        )
+        return ()
+    try:
+        compiled = re.compile(pattern, re.IGNORECASE)
+    except re.error as e:
+        verbose_logger.warning(
+            "LiteLLM: skipping fallback generalization rule with invalid regex %r: %s",
+            pattern,
+            e,
+        )
+        return ()
+    if PROVIDER_KEY not in model_info:
+        return (_CapabilityRule(pattern=compiled, model_info=model_info),)
+    provider = model_info[PROVIDER_KEY]
+    if not isinstance(provider, str):
+        verbose_logger.warning(
+            "LiteLLM: skipping invalid fallback generalization rule %s: '%s' in '%s' must be a string.",
+            rule.get(NAME_FIELD, pattern),
+            PROVIDER_KEY,
+            MODEL_INFO_FIELD,
+        )
+        return ()
+    if len(model_info) == 1:
+        return (_RoutingRule(pattern=compiled, provider=provider),)
+    return (
+        _RoutingRule(pattern=compiled, provider=provider),
+        _CapabilityRule(pattern=compiled, model_info=model_info),
+    )
 
 
 class _FallbackGeneralizations:
-    """Holds the active rule list and its lazily-compiled regex cache."""
+    """Holds the raw rule list and its install-time-compiled routing and capability rules."""
 
     def __init__(self) -> None:
-        self.rules: list[dict] = []
-        self._compiled: Optional[list[tuple[re.Pattern, dict]]] = None
+        self.rules: list = []
+        self.routing_rules: tuple = ()
+        self.capability_rules: tuple = ()
 
-    def set_rules(self, rules: Optional[list[dict]]) -> None:
-        self.rules = rules if isinstance(rules, list) else []
-        self._compiled = None
+    def set_rules(self, rules: Optional[list]) -> None:
+        installed = rules if isinstance(rules, list) else []
+        compiled = tuple(kind for rule in _resolve_legacy_extends(installed) for kind in _compile_rule(rule))
+        self.rules = installed
+        self.routing_rules = tuple(rule for rule in compiled if isinstance(rule, _RoutingRule))
+        self.capability_rules = tuple(rule for rule in compiled if isinstance(rule, _CapabilityRule))
 
-    def _compile(self) -> list[tuple[re.Pattern, dict]]:
-        compiled: list[tuple[re.Pattern, dict]] = []
-        for rule in self.rules:
-            if not isinstance(rule, dict):
-                continue
-            pattern = rule.get(PATTERN_FIELD)
-            model_info = rule.get(MODEL_INFO_FIELD)
-            if not isinstance(pattern, str) or not isinstance(model_info, dict):
-                verbose_logger.warning(
-                    "LiteLLM: skipping malformed fallback generalization rule %s (needs string '%s' and dict '%s').",
-                    rule.get("name", pattern),
-                    PATTERN_FIELD,
-                    MODEL_INFO_FIELD,
-                )
-                continue
-            try:
-                compiled.append((re.compile(pattern, re.IGNORECASE), model_info))
-            except re.error as e:
-                verbose_logger.warning(
-                    "LiteLLM: skipping fallback generalization rule with invalid regex %r: %s",
-                    pattern,
-                    e,
-                )
-        return compiled
-
-    def match(self, model: str) -> Optional[dict]:
+    def match_routing(self, model: str) -> Optional[str]:
         if not model:
             return None
-        if self._compiled is None:
-            self._compiled = self._compile()
-        for pattern, model_info in self._compiled:
-            if pattern.search(model) is not None:
-                return dict(model_info)
-        return None
+        return next(
+            (rule.provider for rule in self.routing_rules if rule.pattern.search(model) is not None),
+            None,
+        )
+
+    def match_capabilities(self, model: str) -> Optional[dict]:
+        if not model:
+            return None
+        matched = tuple(rule.model_info for rule in self.capability_rules if rule.pattern.search(model) is not None)
+        if not matched:
+            return None
+        return {key: value for model_info in matched for key, value in model_info.items()}
 
 
 _registry = _FallbackGeneralizations()
 
 
-def set_fallback_generalizations(rules: Optional[list[dict]]) -> None:
-    """Install the active rule list and invalidate the compiled-regex cache.
+def set_fallback_generalizations(rules: Optional[list]) -> None:
+    """Install the active rule list, compiling and classifying each rule.
 
-    ``extends`` inheritance is resolved here, once, before the rules are stored.
-    Called once when the model cost map is loaded (and again on any reload).
+    Legacy ``extends`` inheritance is resolved here, once, before classification;
+    a legacy rule mixing ``litellm_provider`` with capability keys installs as both
+    kinds. Malformed and invalid-regex rules are warned about and skipped. Called
+    once when the model cost map is loaded (and again on any reload).
     """
-    _registry.set_rules(_resolve_extends(rules) if isinstance(rules, list) else rules)
+    _registry.set_rules(rules)
 
 
-def get_fallback_generalization_rules() -> list[dict]:
+def get_fallback_generalization_rules() -> list:
     """Return the raw rule list (read-only view for callers/tests)."""
     return _registry.rules
 
 
-def match_fallback_generalization(model: str) -> Optional[dict]:
-    """Return the ``model_info`` of the first rule whose regex matches ``model``.
+def match_routing_generalization(model: str) -> Optional[str]:
+    """Return the provider of the first routing rule whose regex matches ``model``.
 
     O(number of rules). Only call this once exact lookups have missed.
     """
-    return _registry.match(model)
+    return _registry.match_routing(model)
+
+
+def match_capability_generalizations(model: str) -> Optional[dict]:
+    """Return the union of the ``model_info`` of every capability rule matching ``model``.
+
+    Later rules override earlier ones on key conflicts. Returns ``None`` when no
+    capability rule matches. O(number of rules); only call once exact lookups have missed.
+    """
+    return _registry.match_capabilities(model)

litellm/litellm_core_utils/get_litellm_params.py

@@ -2,26 +2,8 @@ from typing import Optional
 
 from litellm.llms.openai.data_residency import infer_openai_data_residency
 
-# Pre-define optional kwargs keys as frozenset for O(1) lookups
-# These are extracted from kwargs only if present, avoiding unnecessary .get() calls
-OPTIONAL_KWARGS_KEYS = frozenset(
+AWS_CREDENTIAL_KWARGS_KEYS = frozenset(
     {
-        "azure_ad_token",
-        "tenant_id",
-        "client_id",
-        "client_secret",
-        "azure_username",
-        "azure_password",
-        "azure_scope",
-        "timeout",
-        "gcs_bucket_name",
-        "bucket_name",
-        "vertex_credentials",
-        "vertex_project",
-        "vertex_location",
-        "vertex_ai_project",
-        "vertex_ai_location",
-        "vertex_ai_credentials",
         "aws_region_name",
         "aws_access_key_id",
         "aws_secret_access_key",
@@ -34,14 +16,40 @@ OPTIONAL_KWARGS_KEYS = frozenset(
         "aws_external_id",
         "aws_bedrock_runtime_endpoint",
         "aws_bedrock_project_id",
-        "tpm",
-        "rpm",
-        "itpm",
-        "otpm",
-        "use_xai_oauth",
     }
 )
 
+# Pre-define optional kwargs keys as frozenset for O(1) lookups
+# These are extracted from kwargs only if present, avoiding unnecessary .get() calls
+OPTIONAL_KWARGS_KEYS = (
+    frozenset(
+        {
+            "azure_ad_token",
+            "tenant_id",
+            "client_id",
+            "client_secret",
+            "azure_username",
+            "azure_password",
+            "azure_scope",
+            "timeout",
+            "gcs_bucket_name",
+            "bucket_name",
+            "vertex_credentials",
+            "vertex_project",
+            "vertex_location",
+            "vertex_ai_project",
+            "vertex_ai_location",
+            "vertex_ai_credentials",
+            "tpm",
+            "rpm",
+            "itpm",
+            "otpm",
+            "use_xai_oauth",
+        }
+    )
+    | AWS_CREDENTIAL_KWARGS_KEYS
+)
+
 # Backward-compatible alias for existing imports/tests.
 _OPTIONAL_KWARGS_KEYS = OPTIONAL_KWARGS_KEYS
 

litellm/litellm_core_utils/get_llm_provider_logic.py

@@ -4,7 +4,7 @@ from urllib.parse import urlparse
 import litellm
 from litellm.constants import REPLICATE_MODEL_NAME_WITH_ID_LENGTH
 from litellm.litellm_core_utils.fallback_generalizations import (
-    match_fallback_generalization,
+    match_routing_generalization,
 )
 from litellm.llms.openai_like.json_loader import JSONProviderRegistry
 from litellm.secret_managers.main import get_secret, get_secret_str
@@ -346,6 +346,9 @@ def get_llm_provider(
                     elif endpoint == "https://pinstripes.io/v1":
                         custom_llm_provider = "pinstripes"
                         dynamic_api_key = get_secret_str("PINSTRIPES_API_KEY")
+                    elif endpoint == "https://api.meta.ai/v1":
+                        custom_llm_provider = "meta"
+                        dynamic_api_key = get_secret_str("META_API_KEY")
 
                     if api_base is not None and not isinstance(api_base, str):
                         raise Exception("api base needs to be a string. api_base={}".format(api_base))
@@ -471,12 +474,10 @@ def get_llm_provider(
             custom_llm_provider = "sap"
 
         # Last resort for an otherwise-unknown model: a declarative
-        # fallback-generalization rule (e.g. routes future claude-* to anthropic).
+        # fallback-generalization routing rule (e.g. routes future claude-* to anthropic).
         # Exact provider matches above always win; this only runs on a miss.
         if not custom_llm_provider:
-            generalization = match_fallback_generalization(model)
-            if generalization is not None:
-                custom_llm_provider = generalization.get("litellm_provider") or None
+            custom_llm_provider = match_routing_generalization(model)
 
         if not custom_llm_provider:
             if litellm.suppress_debug_info is False:

litellm/litellm_core_utils/litellm_logging.py

@@ -73,6 +73,7 @@ from litellm.litellm_core_utils.model_param_helper import ModelParamHelper
 from litellm.litellm_core_utils.redact_messages import (
     redact_message_input_output_from_custom_logger,
     redact_message_input_output_from_logging,
+    redact_streaming_responses_for_custom_logger,
 )
 from litellm.llms.base_llm.ocr.transformation import OCRResponse
 from litellm.llms.base_llm.search.transformation import SearchResponse
@@ -2576,6 +2577,9 @@ class Logging(LiteLLMLoggingBaseClass):
                     model_call_details = callback.redact_standard_logging_payload_from_model_call_details(
                         model_call_details=model_call_details
                     )
+                    model_call_details = redact_streaming_responses_for_custom_logger(
+                        model_call_details=model_call_details, custom_logger=callback
+                    )
                     ##################################
                     if self.stream is True:
                         if "async_complete_streaming_response" in model_call_details:
@@ -5208,10 +5212,15 @@ def get_standard_logging_object_payload(
         call_type = kwargs.get("call_type")
         cache_hit = kwargs.get("cache_hit", False)
         # Extract usage as a plain dict, avoiding Pydantic round-trip
-        usage_dict = StandardLoggingPayloadSetup.get_usage_as_dict(
+        raw_usage_dict = StandardLoggingPayloadSetup.get_usage_as_dict(
             response_obj=response_obj,
             combined_usage_object=cast(Optional[Usage], kwargs.get("combined_usage_object")),
         )
+        usage_dict = (
+            {**raw_usage_dict, "output_image_count": len(init_response_obj.data)}
+            if isinstance(init_response_obj, ImageResponse) and init_response_obj.data
+            else raw_usage_dict
+        )
 
         id = response_obj.get("id", kwargs.get("litellm_call_id"))
 

litellm/litellm_core_utils/llm_cost_calc/tiered_pricing.py

@@ -0,0 +1,139 @@
+"""
+Provider-neutral graduated tiered pricing calculation.
+
+Shared by provider cost calculators (e.g. Dashscope) and the proxy budget
+reservation logic so neither has to depend on the other.
+"""
+
+from typing import List, Optional, Union
+
+
+def _coerce_cost_per_token(value: Union[float, int, str, None]) -> float:
+    """
+    Coerce a per-token cost into a float.
+
+    Model cost values loaded from YAML config may arrive as strings (e.g.
+    scientific notation like "4e-07"), which would break arithmetic.
+    """
+    if value is None:
+        return 0.0
+    if isinstance(value, str):
+        try:
+            return float(value)
+        except ValueError:
+            return 0.0
+    return float(value)
+
+
+def calculate_tiered_cost(
+    tokens: int,
+    tiered_pricing: List[dict],
+    cost_key: str,
+    fallback_cost_key: Optional[str] = None,
+) -> float:
+    """
+    Calculate cost for a given number of tokens based on a true tiered pricing structure.
+
+    This function iterates through sorted pricing tiers, calculates the cost for the
+    number of tokens that fall into each tier's range, and sums them up to get the total cost.
+
+    Args:
+        tokens (int): The total number of tokens to calculate the cost for.
+        tiered_pricing (List[dict]): A list of dictionaries, where each dictionary
+            represents a pricing tier.
+        cost_key (str): The key in the tier dictionary that holds the per-token cost
+            (e.g., 'input_cost_per_token').
+        fallback_cost_key (Optional[str], optional): A fallback key to use if the
+            primary `cost_key` is not found in a tier. Defaults to None.
+
+    Returns:
+        float: The total calculated cost for the given tokens.
+
+    Example:
+        >>> tiered_pricing = [
+        ...     {"range": [0, 100000], "input_cost_per_token": 0.0001},
+        ...     {"range": [100000, 500000], "input_cost_per_token": 0.00005},
+        ... ]
+
+        Calculating cost for 150,000 tokens:
+        (100,000 * 0.0001) + (50,000 * 0.00005) = $12.5
+    """
+    if not tiered_pricing or tokens <= 0:
+        return 0.0
+
+    total_cost = 0.0
+    tokens_processed = 0
+
+    sorted_tiers = sorted(tiered_pricing, key=lambda x: x.get("range", [0, 0])[0])
+
+    for tier in sorted_tiers:
+        if tokens_processed >= tokens:
+            break
+
+        tier_range = tier.get("range", [])
+        if len(tier_range) != 2:
+            continue
+
+        range_start, range_end = tier_range
+
+        if tokens <= range_start:
+            continue
+
+        tier_start = max(range_start, tokens_processed)
+        tier_end = min(range_end, tokens)
+
+        if tier_end > tier_start:
+            tokens_in_tier = tier_end - tier_start
+            cost_per_token = tier.get(cost_key) or tier.get(fallback_cost_key, 0)
+            total_cost += tokens_in_tier * _coerce_cost_per_token(cost_per_token)
+            tokens_processed = tier_end
+
+    # After loop, check if any tokens remain (i.e., tokens > highest tier's end range)
+    # and charge them at the last tier's rate.
+    if tokens_processed < tokens and sorted_tiers:
+        last_tier = sorted_tiers[-1]
+        remaining_tokens = tokens - tokens_processed
+        cost_per_token = last_tier.get(cost_key) or last_tier.get(fallback_cost_key, 0)
+        total_cost += remaining_tokens * _coerce_cost_per_token(cost_per_token)
+
+    return total_cost
+
+
+def select_tier_for_input(
+    tiered_pricing: List[dict],
+    input_tokens: int,
+) -> Optional[dict]:
+    """
+    Select the pricing tier for a request based on its total input token count.
+
+    Alibaba Model Studio (Dashscope) tiered pricing is all-or-nothing: the tier is
+    chosen by the total input tokens of a single request and every token in the
+    request (input and output) is billed at that one tier's rate, rather than
+    graduated income-tax-style slicing. A tier matches when
+    ``range_start < input_tokens <= range_end`` (so a request of exactly
+    ``range_end`` tokens stays in the lower tier, matching the official
+    ``0 < Token <= 32K`` phrasing). Requests above the highest declared range fall
+    back to the last (most expensive) tier.
+    """
+    if not tiered_pricing or input_tokens <= 0:
+        return None
+
+    sorted_tiers = sorted(tiered_pricing, key=lambda t: t.get("range", [0, 0])[0])
+    valid_tiers = [tier for tier in sorted_tiers if len(tier.get("range", [])) == 2]
+    if not valid_tiers:
+        return None
+
+    matching = [tier for tier in valid_tiers if tier["range"][0] < input_tokens <= tier["range"][1]]
+    if matching:
+        return matching[0]
+    return valid_tiers[-1]
+
+
+def tier_rate(
+    tier: dict,
+    cost_key: str,
+    fallback_cost_key: Optional[str] = None,
+) -> float:
+    """Read a per-token rate from a tier, coercing YAML string costs to float."""
+    raw = tier.get(cost_key) or tier.get(fallback_cost_key, 0)
+    return _coerce_cost_per_token(raw)

litellm/litellm_core_utils/llm_cost_calc/utils.py

@@ -445,6 +445,7 @@ class PromptTokensDetailsResult(TypedDict):
     text_tokens: int
     audio_tokens: int
     image_tokens: int
+    video_tokens: int
     character_count: int
     image_count: int
     video_length_seconds: float
@@ -473,6 +474,7 @@ def _parse_prompt_tokens_details(usage: Usage) -> PromptTokensDetailsResult:
     )
     audio_tokens = cast(Optional[int], getattr(usage.prompt_tokens_details, "audio_tokens", 0)) or 0
     image_tokens = cast(Optional[int], getattr(usage.prompt_tokens_details, "image_tokens", 0)) or 0
+    video_tokens = _coerce_token_count(getattr(usage.prompt_tokens_details, "video_tokens", 0))
     character_count = (
         cast(
             Optional[int],
@@ -503,6 +505,7 @@ def _parse_prompt_tokens_details(usage: Usage) -> PromptTokensDetailsResult:
         text_tokens=text_tokens,
         audio_tokens=audio_tokens,
         image_tokens=image_tokens,
+        video_tokens=video_tokens,
         character_count=character_count,
         image_count=image_count,
         video_length_seconds=float(video_length_seconds),
@@ -515,6 +518,7 @@ class CompletionTokensDetailsResult(TypedDict):
     text_tokens: int
     reasoning_tokens: int
     image_tokens: int
+    video_tokens: int
 
 
 def _parse_completion_tokens_details(usage: Usage) -> CompletionTokensDetailsResult:
@@ -546,12 +550,14 @@ def _parse_completion_tokens_details(usage: Usage) -> CompletionTokensDetailsRes
         )
         or 0
     )
+    video_tokens = _coerce_token_count(getattr(usage.completion_tokens_details, "video_tokens", 0))
 
     return CompletionTokensDetailsResult(
         audio_tokens=audio_tokens,
         text_tokens=text_tokens,
         reasoning_tokens=reasoning_tokens,
         image_tokens=image_tokens,
+        video_tokens=video_tokens,
     )
 
 
@@ -586,6 +592,13 @@ def _calculate_input_cost(
             image_token_cost_key = "input_cost_per_token"
         prompt_cost += calculate_cost_component(model_info, image_token_cost_key, prompt_tokens_details["image_tokens"])
 
+    ### VIDEO TOKEN COST
+    if prompt_tokens_details["video_tokens"]:
+        video_token_cost_key = "input_cost_per_video_token"
+        if model_info.get(video_token_cost_key) is None:
+            video_token_cost_key = "input_cost_per_token"
+        prompt_cost += calculate_cost_component(model_info, video_token_cost_key, prompt_tokens_details["video_tokens"])
+
     ### CACHE WRITING COST - Now uses tiered pricing
     if (
         prompt_tokens_details["cache_creation_tokens"]
@@ -698,6 +711,7 @@ def generic_cost_per_token(
         text_tokens=usage.prompt_tokens,
         audio_tokens=0,
         image_tokens=0,
+        video_tokens=0,
         character_count=0,
         image_count=0,
         video_length_seconds=0.0,
@@ -716,13 +730,14 @@ def generic_cost_per_token(
     audio_tokens = prompt_tokens_details["audio_tokens"]
     cache_creation = prompt_tokens_details["cache_creation_tokens"]
     image_tokens = prompt_tokens_details["image_tokens"]
+    video_tokens = prompt_tokens_details["video_tokens"]
 
     # Check for double-counting: sum of details > prompt_tokens means overlap
-    total_details = text_tokens + cache_hit + audio_tokens + cache_creation + image_tokens
+    total_details = text_tokens + cache_hit + audio_tokens + cache_creation + image_tokens + video_tokens
     has_double_counting = cache_hit > 0 and total_details > usage.prompt_tokens
 
     if (text_tokens == 0 and prompt_tokens_details["image_count"] == 0) or has_double_counting:
-        text_tokens = usage.prompt_tokens - cache_hit - audio_tokens - cache_creation - image_tokens
+        text_tokens = usage.prompt_tokens - cache_hit - audio_tokens - cache_creation - image_tokens - video_tokens
         # Clamp to zero: inconsistent streaming usage
         if text_tokens < 0:
             text_tokens = 0
@@ -751,6 +766,7 @@ def generic_cost_per_token(
     audio_tokens = 0
     reasoning_tokens = 0
     image_tokens = 0
+    video_tokens = 0
     is_text_tokens_total = False
     if usage.completion_tokens_details is not None:
         completion_tokens_details = _parse_completion_tokens_details(usage)
@@ -758,19 +774,20 @@ def generic_cost_per_token(
         text_tokens = completion_tokens_details["text_tokens"]
         reasoning_tokens = completion_tokens_details["reasoning_tokens"]
         image_tokens = completion_tokens_details["image_tokens"]
+        video_tokens = completion_tokens_details["video_tokens"]
 
     # Handle text_tokens calculation:
     # 1. If text_tokens is explicitly provided and > 0, use it
-    # 2. If there's a breakdown (reasoning/audio/image tokens), calculate text_tokens as the remainder
+    # 2. If there's a breakdown (reasoning/audio/image/video tokens), calculate text_tokens as the remainder
     # 3. If no breakdown at all, assume all completion_tokens are text_tokens
-    has_token_breakdown = image_tokens > 0 or audio_tokens > 0 or reasoning_tokens > 0
+    has_token_breakdown = image_tokens > 0 or audio_tokens > 0 or reasoning_tokens > 0 or video_tokens > 0
     if text_tokens == 0:
         if has_token_breakdown:
             # Calculate text tokens as remainder when we have a breakdown
             # This handles cases like OpenAI's reasoning models where text_tokens isn't provided
             text_tokens = max(
                 0,
-                usage.completion_tokens - reasoning_tokens - audio_tokens - image_tokens,
+                usage.completion_tokens - reasoning_tokens - audio_tokens - image_tokens - video_tokens,
             )
         else:
             # No breakdown at all, all tokens are text tokens
@@ -803,6 +820,14 @@ def generic_cost_per_token(
         )
         completion_cost += float(image_tokens) * _output_cost_per_image_token
 
+    ## VIDEO COST
+    if not is_text_tokens_total and video_tokens and video_tokens > 0:
+        _output_cost_per_video_token = _get_cost_per_unit(model_info, "output_cost_per_video_token", None)
+        _output_cost_per_video_token = (
+            _output_cost_per_video_token if _output_cost_per_video_token is not None else completion_base_cost
+        )
+        completion_cost += float(video_tokens) * _output_cost_per_video_token
+
     ## REGIONAL DATA-RESIDENCY UPLIFT
     # Applied as a flat multiplier across all token costs for the request
     # when the upstream is a regionalized OpenAI host (eu./us.api.openai.com).

litellm/litellm_core_utils/prompt_templates/factory.py

@@ -5494,3 +5494,56 @@ def has_tool_with_name(tools: Any, tool_name: str) -> bool:
         elif tool.get("name") == tool_name:
             return True
     return False
+
+
+def resolve_structured_messages(
+    messages: list[dict[str, Any]] | None,
+    request_kwargs: dict[str, Any],
+) -> list[dict[str, Any]] | None:
+    """
+    Normalize a request's messages to OpenAI-spec chat-completions shape,
+    regardless of which API surface produced them (chat completions,
+    Anthropic /v1/messages, Responses API ``input``, etc).
+
+    Returns ``messages`` unchanged if already present. Otherwise dispatches
+    through the guardrail translation handlers (the same per-surface
+    conversion logic guardrails use) to convert e.g. Responses API ``input``
+    into a message list. Returns ``None`` if no messages could be resolved.
+    """
+    if messages:
+        return messages
+
+    from litellm.litellm_core_utils.api_route_to_call_types import (
+        get_call_types_for_route,
+    )
+    from litellm.llms import load_guardrail_translation_mappings
+    from litellm.types.utils import CallTypes
+
+    mappings = load_guardrail_translation_mappings()
+    call_type: CallTypes | None = None
+
+    # 1. Try route-based inference from proxy metadata
+    route = request_kwargs.get("litellm_metadata", {}).get("user_api_key_request_route")
+    if route:
+        call_types_list = get_call_types_for_route(route)
+        if call_types_list:
+            for ct in call_types_list:
+                if ct in mappings:
+                    call_type = ct
+                    break
+
+    # 2. Fallback: try each mapped handler until one produces messages
+    handlers_to_try: list[Any] = []
+    if call_type is not None and call_type in mappings:
+        handlers_to_try.append(mappings[call_type]())
+    else:
+        handlers_to_try.extend(handler_cls() for handler_cls in mappings.values())
+
+    for handler in handlers_to_try:
+        structured = handler.get_structured_messages(request_kwargs)
+        if structured:
+            return [
+                msg if isinstance(msg, dict) else msg.model_dump()  # type: ignore
+                for msg in structured
+            ]
+    return None

litellm/litellm_core_utils/redact_messages.py

@@ -38,10 +38,45 @@ def redact_message_input_output_from_custom_logger(
     litellm_logging_obj: LiteLLMLoggingObject, result, custom_logger: CustomLogger
 ):
     if hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True:
-        return perform_redaction(litellm_logging_obj.model_call_details, result)
+        return perform_redaction(litellm_logging_obj.model_call_details, result, redact_streaming_responses=False)
     return result
 
 
+def redact_streaming_responses_for_custom_logger(model_call_details: dict, custom_logger: CustomLogger) -> dict:
+    """
+    Returns a copy of model_call_details whose streaming response entries are redacted deepcopies
+    when the custom logger has opted out of message logging. The shared model_call_details is left
+    untouched so other callbacks still receive the unredacted response.
+    """
+    if not (hasattr(custom_logger, "message_logging") and custom_logger.message_logging is not True):
+        return model_call_details
+    redacted_entries = {
+        streaming_key: _redacted_streaming_response_copy(model_call_details[streaming_key])
+        for streaming_key in ("complete_streaming_response", "async_complete_streaming_response")
+        if model_call_details.get(streaming_key) is not None
+    }
+    if not redacted_entries:
+        return model_call_details
+    return {**model_call_details, **redacted_entries}
+
+
+def _redacted_streaming_response_copy(streaming_response):
+    redacted_response = copy.deepcopy(streaming_response)
+    _redact_streaming_response(redacted_response)
+    return redacted_response
+
+
+def _redact_streaming_response(streaming_response):
+    if hasattr(streaming_response, "choices"):
+        for choice in streaming_response.choices:
+            _redact_choice_content(choice)
+        redact_vertex_ai_metadata_from_logged_object(streaming_response)
+    elif hasattr(streaming_response, "output"):
+        _redact_responses_api_output(streaming_response.output)
+        if hasattr(streaming_response, "reasoning") and streaming_response.reasoning is not None:
+            streaming_response.reasoning = None
+
+
 def _redact_choice_content(choice):
     """Helper to redact content in a choice (message or delta)."""
     if isinstance(choice, litellm.Choices):
@@ -150,9 +185,13 @@ def _redact_model_response_dict_choices(choices, redacted_str: str):
             _redact_choice_content(choice)
 
 
-def perform_redaction(model_call_details: dict, result):
+def perform_redaction(model_call_details: dict, result, redact_streaming_responses: bool = True):
     """
     Performs the actual redaction on the logging object and result.
+
+    redact_streaming_responses=False skips the in-place redaction of the shared streaming
+    response entries; per-callback redaction hands each opted-out callback its own redacted
+    copy via redact_streaming_responses_for_custom_logger instead.
     """
     # Redact model_call_details
     model_call_details["messages"] = [{"role": "user", "content": "redacted-by-litellm"}]
@@ -162,17 +201,9 @@ def perform_redaction(model_call_details: dict, result):
     redact_vertex_ai_metadata_from_litellm_params(model_call_details)
 
     # Redact streaming response
-    if model_call_details.get("stream", False) is True and "complete_streaming_response" in model_call_details:
-        _streaming_response = model_call_details["complete_streaming_response"]
-        if hasattr(_streaming_response, "choices"):
-            for choice in _streaming_response.choices:
-                _redact_choice_content(choice)
-            redact_vertex_ai_metadata_from_logged_object(_streaming_response)
-        elif hasattr(_streaming_response, "output"):
-            _redact_responses_api_output(_streaming_response.output)
-            # Redact reasoning field in ResponsesAPIResponse
-            if hasattr(_streaming_response, "reasoning") and _streaming_response.reasoning is not None:
-                _streaming_response.reasoning = None
+    if redact_streaming_responses and model_call_details.get("stream", False) is True:
+        for _streaming_key in ("complete_streaming_response", "async_complete_streaming_response"):
+            _redact_streaming_response(model_call_details.get(_streaming_key))
 
     # Redact result
     if result is not None:

litellm/litellm_core_utils/sensitive_data_masker.py

@@ -1,6 +1,8 @@
 from collections.abc import Mapping
 from typing import Any, Dict, List, Optional, Set
 
+from pydantic import BaseModel
+
 from litellm.constants import DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER
 
 
@@ -153,6 +155,39 @@ def mask_sensitive_structure(data: object) -> object:
     return _error_masker.mask(data)
 
 
+def mask_credentials_in_payload(data: object) -> object:
+    """Return a copy of ``data`` where string values under sensitive-named keys
+    are masked but every other value (``None``, ``int``, ``float``, ``bool``,
+    ``bytes``, ``datetime``, tuples, sets, typed objects) is preserved by
+    identity, and dicts/lists are rebuilt structurally.
+
+    Use this for logging payloads that carry response data through to
+    SpendLogs / OTel / Langfuse, where :meth:`SensitiveDataMasker.mask`'s
+    config-dump semantics (``None`` -> ``"None"``, tuples stringified,
+    objects flattened via ``__dict__``) would silently distort the record.
+
+    Sensitive-key detection is delegated to the shared
+    :class:`SensitiveDataMasker` so pattern updates stay in one place.
+    """
+    return _walk_payload(data, key_is_sensitive=False, depth=0)
+
+
+def _walk_payload(node: object, key_is_sensitive: bool, depth: int) -> object:
+    if depth >= DEFAULT_MAX_RECURSE_DEPTH_SENSITIVE_DATA_MASKER:
+        return node
+    if isinstance(node, Mapping):
+        return {k: _walk_payload(v, _default_masker.is_sensitive_key(k), depth + 1) for k, v in node.items()}
+    if isinstance(node, list):
+        return [_walk_payload(item, key_is_sensitive, depth + 1) for item in node]
+    if isinstance(node, tuple):
+        return tuple(_walk_payload(item, key_is_sensitive, depth + 1) for item in node)
+    if isinstance(node, BaseModel):
+        return _walk_payload(node.model_dump(), key_is_sensitive, depth)
+    if key_is_sensitive and isinstance(node, str) and node:
+        return _default_masker._mask_value(node)
+    return node
+
+
 def mask_sensitive_keys(data: Dict[str, Any], sensitive_fields: Set[str]) -> Dict[str, Any]:
     """Return a new dict with values masked for keys listed in ``sensitive_fields``.
 

litellm/llms/anthropic/chat/transformation.py

@@ -227,6 +227,10 @@ DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING = (
     "Sonnet 4.6+, and Mythos Preview."
 )
 
+DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING = (
+    "Dropping adaptive `thinking` for model=%s: max_tokens is too small to fit the minimum thinking budget."
+)
+
 DROP_UNSUPPORTED_SPEED_WARNING = (
     "Dropping unsupported `speed` for model=%s (drop_params=True). Fast mode is only supported on select Opus models."
 )
@@ -266,6 +270,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
     def custom_llm_provider(self) -> Optional[str]:
         return "anthropic"
 
+    @property
+    def _resolved_provider(self) -> str:
+        return self.custom_llm_provider or "anthropic"
+
     @classmethod
     def get_config(cls, *, model: Optional[str] = None):
         config = super().get_config()
@@ -335,23 +343,26 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
         return any(v in model_lower for v in ("opus-4-7", "opus_4_7", "opus-4.7", "opus_4.7"))
 
     @staticmethod
-    def _supports_effort_level(model: str, level: str) -> bool:
+    def _supports_effort_level(model: str, level: str, custom_llm_provider: str) -> bool:
         """Check ``supports_{level}_reasoning_effort`` in the model map."""
-        return AnthropicConfig._supports_model_capability(model, f"supports_{level}_reasoning_effort")
+        return AnthropicConfig._supports_model_capability(
+            model, f"supports_{level}_reasoning_effort", custom_llm_provider
+        )
 
     @staticmethod
-    def _validate_effort_for_model(model: str, effort: Optional[str]) -> Optional[str]:
+    def _validate_effort_for_model(model: str, effort: Optional[str], custom_llm_provider: str) -> Optional[str]:
         """Return ``None`` if ``effort`` is allowed on ``model``, else an error message."""
         if effort == "max" and not (
-            AnthropicConfig._is_adaptive_thinking_model(model) or AnthropicConfig._supports_effort_level(model, "max")
+            AnthropicConfig._is_adaptive_thinking_model(model, custom_llm_provider)
+            or AnthropicConfig._supports_effort_level(model, "max", custom_llm_provider)
         ):
             return f"effort='max' is not supported by this model. Got model: {model}"
-        if effort == "xhigh" and not AnthropicConfig._supports_effort_level(model, "xhigh"):
+        if effort == "xhigh" and not AnthropicConfig._supports_effort_level(model, "xhigh", custom_llm_provider):
             return f"effort='xhigh' is not supported by this model. Got model: {model}"
         return None
 
     @staticmethod
-    def _model_supports_effort_param(model: str) -> bool:
+    def _model_supports_effort_param(model: str, custom_llm_provider: str) -> bool:
         """Whether the model accepts ``output_config.effort`` at all.
 
         A model qualifies if its map entry advertises ``supports_output_config``
@@ -359,10 +370,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
         signals: e.g. Claude Opus 4.5 supports ``output_config`` without
         advertising a non-default (max/xhigh) effort level.
         """
-        if AnthropicConfig._supports_model_capability(model, "supports_output_config"):
+        if AnthropicConfig._supports_model_capability(model, "supports_output_config", custom_llm_provider):
             return True
         return any(
-            AnthropicConfig._supports_effort_level(model, level)
+            AnthropicConfig._supports_effort_level(model, level, custom_llm_provider)
             for level in ("low", "minimal", "medium", "high", "xhigh", "max")
         )
 
@@ -451,7 +462,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
 
         if (
             "claude-3-7-sonnet" in model
-            or AnthropicConfig._is_adaptive_thinking_model(model)
+            or AnthropicConfig._is_adaptive_thinking_model(model, self._resolved_provider)
             or supports_reasoning(
                 model=model,
                 custom_llm_provider=self.custom_llm_provider,
@@ -1159,11 +1170,13 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
     def _map_reasoning_effort(
         reasoning_effort: Optional[Union[REASONING_EFFORT, str]],
         model: str,
+        custom_llm_provider: str,
         llm_provider: str = "anthropic",
     ) -> Optional[AnthropicThinkingParam]:
+        """Capability probes read the cost map under ``custom_llm_provider``; ``llm_provider`` only tags raised exceptions."""
         if reasoning_effort is None or reasoning_effort == "none":
             return None
-        if AnthropicConfig._is_adaptive_thinking_model(model):
+        if AnthropicConfig._is_adaptive_thinking_model(model, custom_llm_provider):
             return AnthropicThinkingParam(
                 type="adaptive",
             )
@@ -1211,6 +1224,23 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
                 llm_provider=llm_provider,
             )
 
+    @staticmethod
+    def _cap_thinking_budget_to_max_tokens(
+        thinking: AnthropicThinkingParam, max_tokens: Optional[int]
+    ) -> Optional[AnthropicThinkingParam]:
+        """Cap a legacy ``thinking.budget_tokens`` below ``max_tokens`` (Anthropic
+        requires ``max_tokens > budget_tokens``). Returns the (possibly capped)
+        thinking dict, or ``None`` when ``max_tokens`` is too small to fit even the
+        minimum thinking budget and thinking should be dropped."""
+        budget = thinking.get("budget_tokens")
+        if max_tokens is None or not isinstance(budget, int):
+            return thinking
+        if max_tokens <= ANTHROPIC_MIN_THINKING_BUDGET_TOKENS:
+            return None
+        if budget < max_tokens:
+            return thinking
+        return AnthropicThinkingParam(type=thinking.get("type", "enabled"), budget_tokens=max_tokens - 1)
+
     def _extract_json_schema_from_response_format(self, value: Optional[dict]) -> Optional[dict]:
         if value is None:
             return None
@@ -1411,24 +1441,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
                     output_key=param,
                 )
             elif param == "response_format" and isinstance(value, dict):
-                if any(
-                    substring in model
-                    for substring in {
-                        "sonnet-4.5",
-                        "sonnet-4-5",
-                        "opus-4.1",
-                        "opus-4-1",
-                        "opus-4.5",
-                        "opus-4-5",
-                        "opus-4.6",
-                        "opus-4-6",
-                        "opus-4.7",
-                        "opus-4-7",
-                        "sonnet-4.6",
-                        "sonnet-4-6",
-                        "sonnet_4.6",
-                        "sonnet_4_6",
-                    }
+                if AnthropicConfig._supports_model_capability(
+                    model,
+                    "supports_native_structured_output",
+                    self._resolved_provider,
                 ):
                     _output_format = self.map_response_format_to_anthropic_output_format(value)
                     if _output_format is not None:
@@ -1454,7 +1470,38 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
             ):
                 optional_params["metadata"] = {"user_id": value}
             elif param == "thinking":
-                optional_params["thinking"] = value
+                if (
+                    isinstance(value, dict)
+                    and value.get("type") == "adaptive"
+                    and not AnthropicConfig._is_adaptive_thinking_model(model, self._resolved_provider)
+                ):
+                    # Callers (e.g. Claude Code) send adaptive thinking
+                    # unconditionally; translate it down to the legacy
+                    # `thinking={type: enabled, budget_tokens}` interface a
+                    # pre-4.6 model actually supports instead of forwarding a
+                    # shape the model will reject.
+                    max_tokens = non_default_params.get("max_completion_tokens") or non_default_params.get("max_tokens")
+                    legacy_thinking = AnthropicConfig._map_reasoning_effort(
+                        reasoning_effort="medium",
+                        model=model,
+                        custom_llm_provider=self._resolved_provider,
+                        llm_provider=self._resolved_provider,
+                    )
+                    capped_thinking = (
+                        AnthropicConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
+                        if legacy_thinking is not None
+                        else None
+                    )
+                    if capped_thinking is not None:
+                        optional_params["thinking"] = capped_thinking
+                    else:
+                        litellm.verbose_logger.warning(
+                            DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING,
+                            model,
+                        )
+                        optional_params.pop("thinking", None)
+                else:
+                    optional_params["thinking"] = value
             elif param == "reasoning_effort":
                 # Accept both string ("low") and dict ({"effort": "low",
                 # "summary": "concise"}). The Responses->Chat parser keeps the
@@ -1471,20 +1518,21 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
                 mapped_thinking = AnthropicConfig._map_reasoning_effort(
                     reasoning_effort=effort_value,
                     model=model,
-                    llm_provider=self.custom_llm_provider or "anthropic",
+                    custom_llm_provider=self._resolved_provider,
+                    llm_provider=self._resolved_provider,
                 )
                 if mapped_thinking is None:
                     optional_params.pop("thinking", None)
                     optional_params.pop("output_config", None)
                 else:
                     optional_params["thinking"] = mapped_thinking
-                    if AnthropicConfig._is_adaptive_thinking_model(model):
+                    if AnthropicConfig._is_adaptive_thinking_model(model, self._resolved_provider):
                         mapped_effort = REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT.get(effort_value)
                         if mapped_effort is None:
                             AnthropicConfig._raise_invalid_reasoning_effort(
                                 model=model,
                                 value=effort_value,
-                                llm_provider=self.custom_llm_provider or "anthropic",
+                                llm_provider=self._resolved_provider,
                             )
                         optional_params["output_config"] = {"effort": mapped_effort}
             elif param == "web_search_options" and isinstance(value, dict):
@@ -1813,7 +1861,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
             anthropic_messages = anthropic_messages_pt(
                 model=model,
                 messages=messages,
-                llm_provider=self.custom_llm_provider or "anthropic",
+                llm_provider=self._resolved_provider,
             )
         except Exception as e:
             raise AnthropicError(
@@ -1902,7 +1950,7 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
         output_config = optional_params.get("output_config")
         if not output_config or not isinstance(output_config, dict):
             return
-        if litellm.drop_params is True and not self._model_supports_effort_param(model):
+        if litellm.drop_params is True and not self._model_supports_effort_param(model, self._resolved_provider):
             litellm.verbose_logger.warning(
                 DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,
                 model,
@@ -1916,14 +1964,14 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig):
             raise litellm.exceptions.BadRequestError(
                 message=(f"Invalid effort value: {effort!r}. Must be one of: 'high', 'medium', 'low', 'xhigh', 'max'"),
                 model=model,
-                llm_provider=self.custom_llm_provider or "anthropic",
+                llm_provider=self._resolved_provider,
             )
-        gate_error = self._validate_effort_for_model(model, effort)
+        gate_error = self._validate_effort_for_model(model, effort, self._resolved_provider)
         if gate_error is not None:
             raise litellm.exceptions.BadRequestError(
                 message=gate_error,
                 model=model,
-                llm_provider=self.custom_llm_provider or "anthropic",
+                llm_provider=self._resolved_provider,
             )
         data["output_config"] = output_config
 

litellm/llms/anthropic/common_utils.py

@@ -289,6 +289,13 @@ class AnthropicModelInfo(BaseLLMModelInfo):
                 status_code=400,
             )
 
+    @staticmethod
+    def _strip_version_suffix(model: str) -> str:
+        at = model.rfind("@")
+        if at > 0:
+            return model[:at]
+        return model
+
     @staticmethod
     def _model_map_lookup_candidates(model: str) -> List[str]:
         """Model-map keys to try for ``model``: the id itself, the same id with a
@@ -324,6 +331,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
                 _DATED_RELEASE_SUFFIX_RE.sub("", cand),
                 _DOTTED_VERSION_RE.sub(r"\1-\2", cand),
                 _strip_bedrock_id_suffixes(cand),
+                AnthropicModelInfo._strip_version_suffix(cand),
             )
         )
         return list(dict.fromkeys((*primary, *normalized)))
@@ -332,11 +340,15 @@ class AnthropicModelInfo(BaseLLMModelInfo):
     def _get_model_capability(model: str, key: str) -> Optional[bool]:
         """Read boolean capability ``key`` from the model map, or None when
         no entry declares it."""
+        from litellm.utils import _get_bundled_model_cost_map
+
         try:
-            for cand in AnthropicModelInfo._model_map_lookup_candidates(model):
-                value = litellm.model_cost.get(cand, {}).get(key)
-                if isinstance(value, bool):
-                    return value
+            candidates = AnthropicModelInfo._model_map_lookup_candidates(model)
+            for model_cost in (litellm.model_cost, _get_bundled_model_cost_map()):
+                for cand in candidates:
+                    value = model_cost.get(cand, {}).get(key)
+                    if isinstance(value, bool):
+                        return value
         except Exception:
             pass
         return None
@@ -352,18 +364,43 @@ class AnthropicModelInfo(BaseLLMModelInfo):
         return value if isinstance(value, bool) else None
 
     @staticmethod
-    def _supports_model_capability(model: str, key: str) -> bool:
-        """Check a boolean capability ``key`` in the model map.
+    def _get_provider_resolved_capability(model: str, key: str, custom_llm_provider: str) -> Optional[bool]:
+        """Resolve boolean capability ``key`` for ``model`` under the caller's provider.
 
-        Strips bedrock/vertex prefixes so a provider-routed Claude still
-        resolves to the Anthropic model-map entry.
+        Returns the flag when the provider-aware lookup resolves ``model`` to an
+        entry (or fallback rule) that sets it explicitly, and ``None`` when the
+        model does not resolve under that provider or the resolved entry has no
+        opinion on ``key``.
+        """
+        from litellm.utils import _get_model_info_helper
+
+        try:
+            resolved_model, resolved_provider, _, _ = litellm.get_llm_provider(
+                model=model, custom_llm_provider=custom_llm_provider
+            )
+            value = _get_model_info_helper(model=resolved_model, custom_llm_provider=resolved_provider).get(key)
+        except Exception:  # noqa: BLE001  # _get_model_info_helper raises bare Exception for unmapped models
+            return None
+        return value if isinstance(value, bool) else None
+
+    @staticmethod
+    def _supports_model_capability(model: str, key: str, custom_llm_provider: str) -> bool:
+        """Check a boolean capability ``key`` in the model map under the caller's provider.
+
+        The provider-aware lookup is authoritative when it resolves an explicit flag,
+        so ``key: false`` on the provider-namespaced entry wins over every fallback.
+        Otherwise ``_supports_factory``'s provider-level fallbacks and the raw
+        model-map walk remain as backstops for alias forms the lookup misses.
         """
         from litellm.utils import _supports_factory
 
+        resolved = AnthropicModelInfo._get_provider_resolved_capability(model, key, custom_llm_provider)
+        if resolved is not None:
+            return resolved
         try:
             if _supports_factory(
                 model=model,
-                custom_llm_provider="anthropic",
+                custom_llm_provider=custom_llm_provider,
                 key=key,
             ):
                 return True
@@ -372,17 +409,24 @@ class AnthropicModelInfo(BaseLLMModelInfo):
         return AnthropicModelInfo._get_model_capability(model, key) is True
 
     @staticmethod
-    def _is_adaptive_thinking_model(model: str) -> bool:
+    def _is_adaptive_thinking_model(model: str, custom_llm_provider: str) -> bool:
         """Whether ``model`` uses adaptive thinking (``output_config.effort``).
 
         The model cost map is authoritative: an explicit ``supports_adaptive_thinking``
-        entry, or a ``fallback_generalizations`` rule for unknown Claude models. The
-        version gate (>= 4.6, including provider-prefixed Bedrock/Vertex ids that map to
-        no exact entry) lives entirely in that declarative rule, not here.
+        entry resolved under ``custom_llm_provider``, or a ``fallback_generalizations``
+        rule for unknown Claude models. The version gate (>= 4.6, including
+        provider-prefixed Bedrock/Vertex ids that map to no exact entry) lives entirely
+        in that declarative rule, not here.
         """
-        return AnthropicModelInfo._supports_model_capability(model, "supports_adaptive_thinking")
+        return AnthropicModelInfo._supports_model_capability(model, "supports_adaptive_thinking", custom_llm_provider)
 
-    def is_effort_used(self, optional_params: Optional[dict], model: Optional[str] = None) -> bool:
+    def is_effort_used(
+        self,
+        optional_params: Optional[dict],
+        model: Optional[str] = None,
+        *,
+        custom_llm_provider: str,
+    ) -> bool:
         """
         Check if effort parameter is being used and requires a beta header.
 
@@ -394,7 +438,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
             return False
 
         # Claude 4.6+ models use output_config as a stable API feature — no beta header needed
-        if model and self._is_adaptive_thinking_model(model):
+        if model and self._is_adaptive_thinking_model(model, custom_llm_provider):
             return False
 
         # Check if reasoning_effort is provided for Claude Opus 4.5
@@ -475,6 +519,8 @@ class AnthropicModelInfo(BaseLLMModelInfo):
         prompt_caching_set: bool = False,
         file_id_used: bool = False,
         mcp_server_used: bool = False,
+        *,
+        custom_llm_provider: str,
     ) -> List[str]:
         """
         Get list of common beta headers based on the features that are active.
@@ -487,7 +533,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
         betas = []
 
         # Detect features
-        effort_used = self.is_effort_used(optional_params, model)
+        effort_used = self.is_effort_used(optional_params, model, custom_llm_provider=custom_llm_provider)
 
         if effort_used:
             betas.append(ANTHROPIC_EFFORT_BETA_HEADER)  # effort-2025-11-24
@@ -643,7 +689,7 @@ class AnthropicModelInfo(BaseLLMModelInfo):
         tool_search_used = self.is_tool_search_used(tools=tools)
         programmatic_tool_calling_used = self.is_programmatic_tool_calling_used(tools=tools)
         input_examples_used = self.is_input_examples_used(tools=tools)
-        effort_used = self.is_effort_used(optional_params=optional_params, model=model)
+        effort_used = self.is_effort_used(optional_params=optional_params, model=model, custom_llm_provider="anthropic")
         code_execution_tool_used = self.is_code_execution_tool_used(tools=tools)
         container_with_skills_used = self.is_container_with_skills_used(optional_params=optional_params)
         user_anthropic_beta_headers = self._get_user_anthropic_beta_headers(

litellm/llms/anthropic/experimental_pass_through/messages/transformation.py

@@ -32,8 +32,22 @@ from ...common_utils import (
 
 DEFAULT_ANTHROPIC_API_VERSION = "2023-06-01"
 
+DROP_UNSUPPORTED_ADAPTIVE_EFFORT_WARNING = (
+    "Dropping adaptive `thinking`/`output_config.effort` for model=%s: the model "
+    "does not support extended thinking, or max_tokens is too small to fit the "
+    "minimum thinking budget."
+)
+
 
 class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
+    @property
+    def custom_llm_provider(self) -> Optional[str]:
+        return "anthropic"
+
+    @property
+    def _resolved_provider(self) -> str:
+        return self.custom_llm_provider or "anthropic"
+
     def get_supported_anthropic_messages_params(self, model: str) -> list:
         return [
             "messages",
@@ -174,7 +188,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
         return headers, api_base
 
     @staticmethod
-    def _translate_reasoning_effort_to_anthropic(model: str, optional_params: Dict) -> None:
+    def _translate_reasoning_effort_to_anthropic(model: str, optional_params: Dict, custom_llm_provider: str) -> None:
         """Map OpenAI-style ``reasoning_effort`` to native Anthropic params.
 
         Caller-supplied ``thinking`` / ``output_config`` win over the alias.
@@ -191,7 +205,11 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
             return
 
         try:
-            mapped_thinking = AnthropicConfig._map_reasoning_effort(reasoning_effort=reasoning_effort, model=model)
+            mapped_thinking = AnthropicConfig._map_reasoning_effort(
+                reasoning_effort=reasoning_effort,
+                model=model,
+                custom_llm_provider=custom_llm_provider,
+            )
         except _BadRequestError as e:
             raise AnthropicError(message=str(e.message), status_code=400)
 
@@ -201,7 +219,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
             return
 
         optional_params.setdefault("thinking", mapped_thinking)
-        if AnthropicModelInfo._is_adaptive_thinking_model(model):
+        if AnthropicModelInfo._is_adaptive_thinking_model(model, custom_llm_provider):
             mapped_effort = REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT.get(reasoning_effort)
             if mapped_effort is None:
                 raise AnthropicError(
@@ -212,7 +230,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
                     ),
                     status_code=400,
                 )
-            gate_error = AnthropicConfig._validate_effort_for_model(model, mapped_effort)
+            gate_error = AnthropicConfig._validate_effort_for_model(model, mapped_effort, custom_llm_provider)
             if gate_error is not None:
                 raise AnthropicError(message=gate_error, status_code=400)
             existing_output_config = optional_params.get("output_config")
@@ -222,13 +240,15 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
             optional_params["output_config"] = existing_output_config
 
     @staticmethod
-    def _translate_legacy_thinking_for_adaptive_model(model: str, optional_params: Dict) -> None:
+    def _translate_legacy_thinking_for_adaptive_model(
+        model: str, optional_params: Dict, custom_llm_provider: str
+    ) -> None:
         """Translate legacy ``thinking.type=enabled`` to adaptive for 4.6/4.7.
         Caller-provided ``output_config.effort`` is never overridden.
         """
         from litellm.llms.anthropic.chat.transformation import AnthropicConfig
 
-        if not AnthropicModelInfo._is_adaptive_thinking_model(model):
+        if not AnthropicModelInfo._is_adaptive_thinking_model(model, custom_llm_provider):
             return
         thinking = optional_params.get("thinking")
         if not isinstance(thinking, dict) or thinking.get("type") != "enabled":
@@ -236,7 +256,7 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
 
         budget = int(thinking.get("budget_tokens") or 0)
         if budget >= DEFAULT_REASONING_EFFORT_XHIGH_THINKING_BUDGET and (
-            AnthropicConfig._supports_effort_level(model, "xhigh")
+            AnthropicConfig._supports_effort_level(model, "xhigh", custom_llm_provider)
         ):
             effort = "xhigh"
         elif budget >= DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET:
@@ -253,6 +273,138 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
         existing_output_config.setdefault("effort", effort)
         optional_params["output_config"] = existing_output_config
 
+    @staticmethod
+    def _translate_adaptive_effort_for_non_adaptive_model(
+        model: str, optional_params: Dict, max_tokens: Optional[int], custom_llm_provider: str
+    ) -> None:
+        """Translate the 4.6+ adaptive-thinking interface (``thinking.type=adaptive``
+        and/or ``output_config.effort``) down to what an older Anthropic model
+        supports. Clients like Claude Code send this interface unconditionally, so
+        without translation it reaches a pre-4.6 model and Anthropic rejects it with
+        "This model does not support the effort parameter".
+
+        The reshape is silent, matching how the messages path already strips
+        unsupported ``output_config`` for older models (bedrock invoke, issue
+        #22797): the goal is to keep the request working, not to fail it.
+
+        ``thinking.type=adaptive`` and ``output_config.effort`` are independent
+        capabilities. Adaptive thinking needs ``supports_adaptive_thinking`` (4.6+);
+        ``output_config.effort`` needs ``supports_output_config``, which some
+        non-adaptive models (e.g. Claude Opus 4.5) advertise on its own. So the two
+        are handled separately:
+
+        - Adaptive-thinking models (4.6+): both are native, left untouched.
+        - ``supports_output_config`` but non-adaptive (Opus 4.5): keep
+          ``output_config.effort`` (native), only drop the unsupported adaptive
+          ``thinking`` block. When adaptive thinking is being dropped and the
+          effort level itself isn't supported by the model (e.g. ``xhigh``/``max``
+          on Opus 4.5, which only accepts low/medium/high, while ``xhigh`` is
+          Claude Code's default), fall through to the legacy translation below
+          instead of forwarding a level Anthropic would reject. Effort-only
+          requests are always left untouched: provider subclasses own their level
+          normalization (bedrock clamps ``xhigh`` to the model's ceiling after
+          this base transform runs).
+        - Thinking-capable but neither (``supports_reasoning``, e.g. Haiku/Sonnet
+          4.5): map effort to legacy ``thinking={type: enabled, budget_tokens}`` via
+          ``AnthropicConfig._map_reasoning_effort``, capped below ``max_tokens``
+          (Anthropic requires ``max_tokens > budget_tokens``) and dropped when
+          ``max_tokens`` can't fit even the minimum budget.
+        - No reasoning support: ``thinking`` is dropped.
+
+        For the last two, only the consumed ``effort`` key is removed from
+        ``output_config``; any residual (e.g. ``format``) is left for provider
+        subclasses to handle.
+        """
+        from litellm.exceptions import BadRequestError as _BadRequestError
+        from litellm.llms.anthropic.chat.transformation import AnthropicConfig
+
+        if AnthropicConfig._is_adaptive_thinking_model(model, custom_llm_provider):
+            return
+
+        output_config = optional_params.get("output_config")
+        thinking = optional_params.get("thinking")
+        effort = output_config.get("effort") if isinstance(output_config, dict) else None
+        adaptive_thinking = isinstance(thinking, dict) and thinking.get("type") == "adaptive"
+        if effort is None and not adaptive_thinking:
+            return
+
+        # Models that natively accept `output_config.effort` but are not adaptive (Claude Opus 4.5).
+        # Keep the native effort and only drop the adaptive `thinking` block, which these models
+        # reject. Effort-only requests pass through so provider subclasses (bedrock/vertex) keep
+        # owning level clamping; an adaptive request only stays here when its effort level is one
+        # the model supports, otherwise it falls through to the legacy budget translation below.
+        if AnthropicConfig._model_supports_effort_param(model, custom_llm_provider) and (
+            not adaptive_thinking
+            or AnthropicConfig._validate_effort_for_model(model, effort, custom_llm_provider) is None
+        ):
+            if adaptive_thinking:
+                optional_params.pop("thinking", None)
+            return
+
+        supports_thinking = AnthropicModelInfo._supports_model_capability(
+            model, "supports_reasoning", custom_llm_provider
+        )
+        try:
+            legacy_thinking = (
+                AnthropicConfig._map_reasoning_effort(
+                    reasoning_effort=effort or "medium",
+                    model=model,
+                    custom_llm_provider=custom_llm_provider,
+                )
+                if supports_thinking
+                else None
+            )
+        except _BadRequestError as e:
+            raise AnthropicError(message=str(e.message), status_code=400)
+        capped_thinking = (
+            AnthropicConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
+            if legacy_thinking is not None
+            else None
+        )
+
+        if capped_thinking is not None:
+            optional_params["thinking"] = capped_thinking
+        else:
+            verbose_logger.warning(DROP_UNSUPPORTED_ADAPTIVE_EFFORT_WARNING, model)
+            optional_params.pop("thinking", None)
+
+        if isinstance(output_config, dict) and "effort" in output_config:
+            residual = {k: v for k, v in output_config.items() if k != "effort"}
+            if residual:
+                optional_params["output_config"] = residual
+            else:
+                optional_params.pop("output_config", None)
+
+    @staticmethod
+    def _drop_incompatible_temperature_for_thinking(
+        model: str, optional_params: dict, custom_llm_provider: str
+    ) -> None:
+        """Anthropic rejects any ``temperature`` other than 1 while extended thinking
+        is enabled ("temperature may only be set to 1 when thinking is enabled").
+
+        Clients like Claude Code send ``thinking``/``output_config.effort`` together
+        with a pinned ``temperature`` (e.g. the safety classifier uses ``temperature=0``
+        for determinism). When the request lands on a non-adaptive model, the effort
+        interface is reshaped above into legacy ``thinking={type: enabled}`` (or kept
+        as ``output_config.effort`` on Opus 4.5), and the leftover ``temperature`` would
+        400. Preserving the thinking the caller asked for wins over an unhonorable
+        sampling value (Anthropic forces ``temperature=1`` under thinking regardless),
+        so drop it and let the API default apply.
+
+        Adaptive models (4.6+) own this natively and are left untouched.
+        """
+        if AnthropicModelInfo._is_adaptive_thinking_model(model, custom_llm_provider):
+            return
+        temperature = optional_params.get("temperature")
+        if temperature is None or temperature == 1:
+            return
+        thinking = optional_params.get("thinking")
+        output_config = optional_params.get("output_config")
+        thinking_enabled = isinstance(thinking, dict) and thinking.get("type") == "enabled"
+        effort_enabled = isinstance(output_config, dict) and output_config.get("effort") is not None
+        if thinking_enabled or effort_enabled:
+            optional_params.pop("temperature", None)
+
     def transform_anthropic_messages_request(
         self,
         model: str,
@@ -277,11 +429,26 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig):
         self._translate_reasoning_effort_to_anthropic(
             model=model,
             optional_params=anthropic_messages_optional_request_params,
+            custom_llm_provider=self._resolved_provider,
         )
 
         self._translate_legacy_thinking_for_adaptive_model(
             model=model,
             optional_params=anthropic_messages_optional_request_params,
+            custom_llm_provider=self._resolved_provider,
+        )
+
+        self._translate_adaptive_effort_for_non_adaptive_model(
+            model=model,
+            optional_params=anthropic_messages_optional_request_params,
+            max_tokens=max_tokens,
+            custom_llm_provider=self._resolved_provider,
+        )
+
+        self._drop_incompatible_temperature_for_thinking(
+            model=model,
+            optional_params=anthropic_messages_optional_request_params,
+            custom_llm_provider=self._resolved_provider,
         )
 
         system_param = anthropic_messages_optional_request_params.get("system")

litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py

@@ -198,14 +198,16 @@ class LiteLLMAnthropicToResponsesAPIAdapter:
     @staticmethod
     def translate_tool_choice_to_responses_api(
         tool_choice: AnthropicMessagesToolChoice,
-    ) -> Dict[str, Any]:
+    ) -> Union[str, dict[str, Any]]:
         """Convert Anthropic tool_choice to Responses API tool_choice."""
         tc_type = tool_choice.get("type")
         if tc_type == "any":
-            return {"type": "required"}
+            return "required"
         elif tc_type == "tool":
             return {"type": "function", "name": tool_choice.get("name", "")}
-        return {"type": "auto"}
+        elif tc_type == "none":
+            return "none"
+        return "auto"
 
     @staticmethod
     def translate_context_management_to_responses_api(

litellm/llms/azure_ai/anthropic/messages_transformation.py

@@ -21,6 +21,10 @@ class AzureAnthropicMessagesConfig(AnthropicMessagesConfig):
     and Azure endpoint format.
     """
 
+    @property
+    def custom_llm_provider(self) -> Optional[str]:
+        return "azure_ai"
+
     def should_strip_billing_metadata(self) -> bool:
         return True
 

litellm/llms/base_llm/guardrail_translation/base_translation.py

@@ -1,4 +1,5 @@
 from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
 from typing import TYPE_CHECKING, Any, Dict, List, Optional
 
 if TYPE_CHECKING:
@@ -11,10 +12,30 @@ if TYPE_CHECKING:
     from litellm.types.llms.openai import AllMessageValues
 
 
+@dataclass(slots=True)
+class StreamTransformSink:
+    """Out-parameter used by ``process_output_streaming_response`` to hand the
+    guardrailed streaming state back to the caller.
+
+    The streaming text-transform path must not mutate ``responses_so_far`` (it is
+    the raw accumulator the guardrail re-reads every round), so the guardrailed
+    accumulated text per choice (``mutated_text_per_choice``, keyed by
+    ``StreamingChoices.index``) and the per-choice trailing holdback the guardrail
+    requested (``holdback_per_choice``, from ``stream_holdback_chars``) are
+    reported here instead of in place. Only the OpenAI chat handler populates this
+    today; the hook passes a fresh sink per round and reads it afterwards. A
+    mutable dataclass is deliberate: it is a write-once output parameter for a
+    single call, not shared state.
+    """
+
+    mutated_text_per_choice: dict[int, str] = field(default_factory=dict)
+    holdback_per_choice: dict[int, int] = field(default_factory=dict)
+
+
 class BaseTranslation(ABC):
     @staticmethod
     def transform_user_api_key_dict_to_metadata(
-        user_api_key_dict: Optional[Any],
+        user_api_key_dict: Any | None,
     ) -> Dict[str, Any]:
         """
         Transform user_api_key_dict to a metadata dict with prefixed keys.
@@ -73,7 +94,7 @@ class BaseTranslation(ABC):
         guardrail_to_apply: "CustomGuardrail",
         litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
         user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
-        request_data: Optional[dict] = None,
+        request_data: dict | None = None,
     ) -> Any:
         """
         Process output response with guardrails.
@@ -92,12 +113,15 @@ class BaseTranslation(ABC):
         guardrail_to_apply: "CustomGuardrail",
         litellm_logging_obj: Optional["LiteLLMLoggingObj"] = None,
         user_api_key_dict: Optional["UserAPIKeyAuth"] = None,
-        request_data: Optional[dict] = None,
+        request_data: dict | None = None,
+        stream_transform_sink: StreamTransformSink | None = None,
     ) -> Any:
         """
         Process output streaming response with guardrails.
 
-        Optional to override in subclasses.
+        Optional to override in subclasses. ``stream_transform_sink`` is the
+        out-parameter used by handlers that support streaming text
+        transformations (see ``StreamTransformSink``); base handlers ignore it.
         """
         return responses_so_far
 
@@ -105,8 +129,8 @@ class BaseTranslation(ABC):
         self,
         exc: "ModifyResponseException",
         stream_started: bool = False,
-        responses_so_far: Optional[list[Any]] = None,
-    ) -> Optional[list[bytes]]:
+        responses_so_far: list[Any] | None = None,
+    ) -> list[bytes] | None:
         """
         Build the streaming chunks that deliver a guardrail block message and
         cleanly terminate the stream in this provider's wire format.
@@ -125,7 +149,7 @@ class BaseTranslation(ABC):
         """
         return None
 
-    def get_structured_messages(self, data: dict) -> Optional[List["AllMessageValues"]]:
+    def get_structured_messages(self, data: dict) -> List["AllMessageValues"] | None:
         """
         Convert request data to OpenAI-spec structured messages.
 

litellm/llms/bedrock/base_aws_llm.py

@@ -877,6 +877,15 @@ class BaseAWSLLM:
                     "Resource": "*",
                     "Condition": {"Bool": {"aws:SecureTransport": "true"}},
                 },
+                {
+                    "Sid": "BedrockMantleLiteLLM",
+                    "Effect": "Allow",
+                    "Action": [
+                        "bedrock-mantle:CreateInference",
+                    ],
+                    "Resource": "*",
+                    "Condition": {"Bool": {"aws:SecureTransport": "true"}},
+                },
             ],
         }
         assume_role_params = {

litellm/llms/bedrock/batches/transformation.py

@@ -5,6 +5,9 @@ from typing import Any, Dict, List, Literal, Optional, Union, cast
 
 from httpx import Headers, Response
 
+from litellm.litellm_core_utils.cloud_storage_security import (
+    BEDROCK_MANAGED_S3_BATCH_PREFIX,
+)
 from litellm.litellm_core_utils.safe_json_dumps import safe_dumps
 from litellm.llms.base_llm.batches.transformation import BaseBatchesConfig
 from litellm.llms.base_llm.chat.transformation import BaseLLMException
@@ -26,6 +29,15 @@ from litellm.types.utils import LiteLLMBatch, LlmProviders
 from ..base_aws_llm import BaseAWSLLM
 from ..common_utils import CommonBatchFilesUtils
 
+# Bedrock batch input files are uploaded as
+# s3://bucket/litellm-bedrock-files-{model, ":" -> "-"}-{uuid4}.jsonl (see
+# BedrockFilesTransformation._get_s3_object_name). A uuid4 is always 36 hex/dash
+# characters, so it can be stripped off the end unambiguously even though the
+# model name itself may contain dashes.
+_S3_BATCH_FILE_UUID_SUFFIX_PATTERN = re.compile(
+    r"-[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\.jsonl$"
+)
+
 
 class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
     """
@@ -40,6 +52,41 @@ class BedrockBatchesConfig(BaseAWSLLM, BaseBatchesConfig):
     def custom_llm_provider(self) -> LlmProviders:
         return LlmProviders.BEDROCK
 
+    @classmethod
+    def _get_bare_model_name_from_s3_key(cls, object_key: str) -> Optional[str]:
+        if not object_key.startswith(BEDROCK_MANAGED_S3_BATCH_PREFIX):
+            return None
+        model_part = object_key[len(BEDROCK_MANAGED_S3_BATCH_PREFIX) :]
+        match = _S3_BATCH_FILE_UUID_SUFFIX_PATTERN.search(model_part)
+        if not match or match.start() == 0:
+            return None
+        return model_part[: match.start()]
+
+    @classmethod
+    def is_unmanaged_s3_batch_input_file_id(cls, input_file_id: Optional[str]) -> bool:
+        """
+        Returns True if `input_file_id` is a raw s3:// Bedrock batch input file (i.e. not a
+        LiteLLM-managed unified file id) whose object key embeds the model name in the
+        `litellm-bedrock-files-{model}-{uuid}.jsonl` layout.
+        """
+        if input_file_id is None or not input_file_id.startswith("s3://"):
+            return False
+        object_key = input_file_id.rsplit("/", 1)[-1]
+        return cls._get_bare_model_name_from_s3_key(object_key) is not None
+
+    @classmethod
+    def get_bare_model_name_from_s3_file(cls, input_file_id: str) -> str:
+        """
+        Extracts the bare model name (e.g. "us.anthropic.claude-sonnet-4-20250514-v1-0") from
+        an unmanaged batch's s3:// input file id. Note any ":" in the original model id was
+        replaced with "-" at upload time, so callers must fuzzy-match against configured
+        deployments rather than expect an exact string match.
+        """
+        object_key = input_file_id.rsplit("/", 1)[-1]
+        bare_model_name = cls._get_bare_model_name_from_s3_key(object_key)
+        assert bare_model_name is not None  # narrowed by is_unmanaged_s3_batch_input_file_id
+        return bare_model_name
+
     def validate_environment(
         self,
         headers: dict,

litellm/llms/bedrock/chat/converse_transformation.py

@@ -33,6 +33,7 @@ from litellm.litellm_core_utils.prompt_templates.factory import (
     make_valid_bedrock_tool_name,
 )
 from litellm.llms.anthropic.chat.transformation import (
+    DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING,
     DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,
     REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT,
     AnthropicConfig,
@@ -423,6 +424,7 @@ class AmazonConverseConfig(BaseConfig):
             mapped_thinking = AnthropicConfig._map_reasoning_effort(
                 reasoning_effort=reasoning_effort,
                 model=model,
+                custom_llm_provider="bedrock",
                 llm_provider="bedrock_converse",
             )
             if mapped_thinking is None:
@@ -430,7 +432,7 @@ class AmazonConverseConfig(BaseConfig):
                 optional_params.pop("output_config", None)
             else:
                 optional_params["thinking"] = mapped_thinking
-                if AnthropicConfig._is_adaptive_thinking_model(model):
+                if AnthropicConfig._is_adaptive_thinking_model(model, "bedrock"):
                     mapped_effort = REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT.get(reasoning_effort)
                     if mapped_effort is None:
                         AnthropicConfig._raise_invalid_reasoning_effort(
@@ -465,7 +467,7 @@ class AmazonConverseConfig(BaseConfig):
                 model=model,
                 llm_provider="bedrock_converse",
             )
-        error = AnthropicConfig._validate_effort_for_model(model=model, effort=effort)
+        error = AnthropicConfig._validate_effort_for_model(model=model, effort=effort, custom_llm_provider="bedrock")
         if error is not None:
             raise litellm.exceptions.BadRequestError(
                 message=error,
@@ -898,7 +900,28 @@ class AmazonConverseConfig(BaseConfig):
                     "tool_choice": {"disable_parallel_tool_use": disable_parallel}
                 }
             if param == "thinking":
-                optional_params["thinking"] = value
+                if (
+                    isinstance(value, dict)
+                    and value.get("type") == "adaptive"
+                    and not AnthropicConfig._is_adaptive_thinking_model(model, "bedrock")
+                ):
+                    max_tokens = non_default_params.get("max_completion_tokens") or non_default_params.get("max_tokens")
+                    legacy_thinking = AnthropicConfig._map_reasoning_effort(
+                        reasoning_effort="medium",
+                        model=model,
+                        custom_llm_provider="bedrock",
+                    )
+                    capped = (
+                        AnthropicConfig._cap_thinking_budget_to_max_tokens(legacy_thinking, max_tokens)
+                        if legacy_thinking is not None
+                        else None
+                    )
+                    if capped is not None:
+                        optional_params["thinking"] = capped
+                    else:
+                        litellm.verbose_logger.warning(DROP_UNSUPPORTED_ADAPTIVE_THINKING_WARNING, model)
+                else:
+                    optional_params["thinking"] = value
             elif param == "reasoning_effort" and isinstance(value, str):
                 self._handle_reasoning_effort_parameter(
                     model=model, reasoning_effort=value, optional_params=optional_params
@@ -1279,7 +1302,7 @@ class AmazonConverseConfig(BaseConfig):
 
         if anthropic_output_config is not None and isinstance(anthropic_output_config, dict):
             if base_model.startswith("anthropic"):
-                if litellm.drop_params is True and not AnthropicConfig._model_supports_effort_param(model):
+                if litellm.drop_params is True and not AnthropicConfig._model_supports_effort_param(model, "bedrock"):
                     litellm.verbose_logger.warning(
                         DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,
                         model,
@@ -1422,7 +1445,7 @@ class AmazonConverseConfig(BaseConfig):
             if (
                 isinstance(output_config, dict)
                 and output_config.get("effort") is not None
-                and not AnthropicConfig._is_adaptive_thinking_model(model)
+                and not AnthropicConfig._is_adaptive_thinking_model(model, "bedrock")
             ):
                 from litellm.types.llms.anthropic import (
                     ANTHROPIC_EFFORT_BETA_HEADER,

litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py

@@ -115,7 +115,7 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
         keeps working. Non-adaptive models and models without a ceiling are
         left untouched.
         """
-        if not AnthropicConfig._is_adaptive_thinking_model(model):
+        if not AnthropicConfig._is_adaptive_thinking_model(model, "bedrock"):
             return
         effort = params.get("reasoning_effort")
         if not isinstance(effort, str):
@@ -228,7 +228,7 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
                 custom_llm_provider="bedrock",
                 key="supports_output_config",
             )
-            or AnthropicConfig._model_supports_effort_param(model)
+            or AnthropicConfig._model_supports_effort_param(model, "bedrock")
         ):
             if anthropic_request.pop("output_config", None) is not None:
                 verbose_logger.warning(
@@ -269,6 +269,7 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig):
             prompt_caching_set=False,
             file_id_used=self.is_file_id_used(messages),
             mcp_server_used=self.is_mcp_server_used(optional_params.get("mcp_servers")),
+            custom_llm_provider="bedrock",
         )
         beta_set.update(auto_betas)
 

litellm/llms/bedrock/claude_platform/transformation.py

@@ -54,7 +54,9 @@ class BedrockClaudePlatformConfig(BedrockClaudePlatformMixin, AnthropicConfig):
             tool_search_used=self.is_tool_search_used(tools=optional_params.get("tools")),
             programmatic_tool_calling_used=self.is_programmatic_tool_calling_used(tools=optional_params.get("tools")),
             input_examples_used=self.is_input_examples_used(tools=optional_params.get("tools")),
-            effort_used=self.is_effort_used(optional_params=optional_params, model=model),
+            effort_used=self.is_effort_used(
+                optional_params=optional_params, model=model, custom_llm_provider="anthropic"
+            ),
             user_anthropic_beta_headers=self._get_user_anthropic_beta_headers(
                 anthropic_beta_header=headers.get("anthropic-beta")
             ),

litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py

@@ -77,6 +77,10 @@ class AmazonAnthropicClaudeMessagesConfig(
 
     DEFAULT_BEDROCK_ANTHROPIC_API_VERSION = "bedrock-2023-05-31"
 
+    @property
+    def custom_llm_provider(self) -> Optional[str]:
+        return "bedrock"
+
     BEDROCK_INVOKE_ALLOWED_TOP_LEVEL_FIELDS = frozenset(BedrockInvokeAnthropicMessagesRequest.__annotations__.keys())
 
     def __init__(self, **kwargs):
@@ -93,26 +97,48 @@ class AmazonAnthropicClaudeMessagesConfig(
             return [{"type": "text", "text": value}]
         return [value]
 
-    def _normalize_system_role_messages_for_bedrock(self, anthropic_messages_request: dict) -> None:
-        """Bedrock Invoke rejects ``role: "system"`` entries inside ``messages`` on
-        some Claude aliases; Anthropic Messages carries that content in the
-        top-level ``system`` field. Move any such entries into ``system`` before
-        the Invoke request is built."""
+    @staticmethod
+    def _is_system_role_message(message: Any) -> bool:
+        return isinstance(message, dict) and message.get("role") == "system"
+
+    def _normalize_system_role_messages_for_bedrock(self, anthropic_messages_request: dict, model: str) -> None:
+        """Bedrock Invoke validates ``role: "system"`` entries inside ``messages``
+        per model. Models carrying ``supports_mid_conversation_system`` in the
+        cost map (the Opus 4.8 family) only reject a leading run ("messages.0:
+        use the top-level 'system' parameter for the initial system prompt") and
+        accept mid-conversation entries (e.g. Claude Code's
+        ``mid-conversation-system-2026-04-07`` reminders) in place, where they
+        MUST stay: hoisting one mutates the ``system`` prefix and invalidates the
+        prompt cache for the entire message history. Older Claude models (Opus
+        4.7, Sonnet 4.6, Haiku 4.5, ...) reject the role in every position
+        ("role 'system' is not supported on this model"), so without the flag
+        every system entry is hoisted into the top-level ``system`` field.
+        Billing-header system blocks are stripped from the top-level ``system``
+        field regardless of whether anything was hoisted."""
         messages = anthropic_messages_request.get("messages")
         if not isinstance(messages, list):
             return
-        system_role_messages = [m for m in messages if isinstance(m, dict) and m.get("role") == "system"]
-        if not system_role_messages:
-            return
-
-        anthropic_messages_request["messages"] = [
-            m for m in messages if not (isinstance(m, dict) and m.get("role") == "system")
-        ]
+        if _supports_factory(
+            model=model,
+            custom_llm_provider="bedrock",
+            key="supports_mid_conversation_system",
+        ):
+            leading_count = next(
+                (i for i, m in enumerate(messages) if not self._is_system_role_message(m)),
+                len(messages),
+            )
+            hoisted = messages[:leading_count]
+            remaining = messages[leading_count:]
+        else:
+            hoisted = [m for m in messages if self._is_system_role_message(m)]
+            remaining = [m for m in messages if not self._is_system_role_message(m)]
+        if hoisted:
+            anthropic_messages_request["messages"] = remaining
         system_content = [
             block
             for source in (
                 anthropic_messages_request.get("system"),
-                *(m.get("content") for m in system_role_messages),
+                *(m.get("content") for m in hoisted),
             )
             for block in self._as_system_content_blocks(source)
         ]
@@ -247,7 +273,7 @@ class AmazonAnthropicClaudeMessagesConfig(
         Returns:
             True if the model supports extended thinking on Bedrock
         """
-        if AnthropicModelInfo._is_adaptive_thinking_model(model):
+        if AnthropicModelInfo._is_adaptive_thinking_model(model, "bedrock"):
             return True
 
         model_lower = model.lower()
@@ -297,7 +323,7 @@ class AmazonAnthropicClaudeMessagesConfig(
         if not self._supports_extended_thinking_on_bedrock(model):
             return False
 
-        is_adaptive_thinking_model = AnthropicModelInfo._is_adaptive_thinking_model(model)
+        is_adaptive_thinking_model = AnthropicModelInfo._is_adaptive_thinking_model(model, "bedrock")
 
         thinking = anthropic_messages_request.get("thinking")
         if isinstance(thinking, dict):
@@ -574,6 +600,7 @@ class AmazonAnthropicClaudeMessagesConfig(
             mcp_server_used=anthropic_model_info.is_mcp_server_used(
                 anthropic_messages_optional_request_params.get("mcp_servers")
             ),
+            custom_llm_provider="bedrock",
         )
         beta_set.update(auto_betas)
 
@@ -640,7 +667,7 @@ class AmazonAnthropicClaudeMessagesConfig(
         path degrades ``xhigh`` -> ``max`` rather than 400-ing. Non-adaptive models
         and models without a ceiling are left untouched.
         """
-        if not AnthropicModelInfo._is_adaptive_thinking_model(model):
+        if not AnthropicModelInfo._is_adaptive_thinking_model(model, "bedrock"):
             return
         effort = optional_params.get("reasoning_effort")
         if not isinstance(effort, str):
@@ -669,7 +696,7 @@ class AmazonAnthropicClaudeMessagesConfig(
             litellm_params=litellm_params,
             headers=headers,
         )
-        self._normalize_system_role_messages_for_bedrock(anthropic_messages_request)
+        self._normalize_system_role_messages_for_bedrock(anthropic_messages_request, model=model)
         #########################################################
         ############## BEDROCK Invoke SPECIFIC TRANSFORMATION ###
         #########################################################
@@ -728,7 +755,7 @@ class AmazonAnthropicClaudeMessagesConfig(
                 custom_llm_provider="bedrock",
                 key="supports_output_config",
             )
-            or AnthropicConfig._model_supports_effort_param(model)
+            or AnthropicConfig._model_supports_effort_param(model, "bedrock")
         ):
             if anthropic_messages_request.pop("output_config", None) is not None:
                 verbose_logger.warning(
@@ -765,7 +792,7 @@ class AmazonAnthropicClaudeMessagesConfig(
         if (
             litellm.drop_params is True
             and "output_config" in anthropic_messages_request
-            and not AnthropicConfig._model_supports_effort_param(model)
+            and not AnthropicConfig._model_supports_effort_param(model, "bedrock")
         ):
             verbose_logger.warning(
                 DROP_UNSUPPORTED_OUTPUT_CONFIG_WARNING,

litellm/llms/dashscope/cost_calculator.py

@@ -7,6 +7,7 @@ Handles tiered pricing and prompt caching scenarios.
 from dataclasses import dataclass
 from typing import List, Optional, Tuple
 
+from litellm.litellm_core_utils.llm_cost_calc.tiered_pricing import calculate_tiered_cost
 from litellm.types.utils import ModelInfo, Usage
 from litellm.utils import get_model_info
 
@@ -42,80 +43,6 @@ def _extract_token_breakdown(usage: Usage) -> TokenBreakdown:
     return TokenBreakdown(text_tokens, cached_tokens, completion_tokens, reasoning_tokens)
 
 
-def _calculate_tiered_cost(
-    tokens: int,
-    tiered_pricing: List[dict],
-    cost_key: str,
-    fallback_cost_key: Optional[str] = None,
-) -> float:
-    """
-    Calculate cost for a given number of tokens based on a true tiered pricing structure.
-
-    This function iterates through sorted pricing tiers, calculates the cost for the
-    number of tokens that fall into each tier's range, and sums them up to get the total cost.
-
-    Args:
-        tokens (int): The total number of tokens to calculate the cost for.
-        tiered_pricing (List[dict]): A list of dictionaries, where each dictionary
-            represents a pricing tier.
-        cost_key (str): The key in the tier dictionary that holds the per-token cost
-            (e.g., 'input_cost_per_token').
-        fallback_cost_key (Optional[str], optional): A fallback key to use if the
-            primary `cost_key` is not found in a tier. Defaults to None.
-
-    Returns:
-        float: The total calculated cost for the given tokens.
-
-    Example:
-        >>> tiered_pricing = [
-        ...     {"range": [0, 100000], "input_cost_per_token": 0.0001},
-        ...     {"range": [100000, 500000], "input_cost_per_token": 0.00005},
-        ... ]
-
-        Calculating cost for 150,000 tokens:
-        (100,000 * 0.0001) + (50,000 * 0.00005) = $12.5
-    """
-    if not tiered_pricing or tokens <= 0:
-        return 0.0
-
-    total_cost = 0.0
-    tokens_processed = 0
-
-    sorted_tiers = sorted(tiered_pricing, key=lambda x: x.get("range", [0, 0])[0])
-
-    for tier in sorted_tiers:
-        if tokens_processed >= tokens:
-            break
-
-        tier_range = tier.get("range", [])
-        if len(tier_range) != 2:
-            continue
-
-        range_start, range_end = tier_range
-
-        if tokens <= range_start:
-            continue
-
-        tier_start = max(range_start, tokens_processed)
-        tier_end = min(range_end, tokens)
-
-        if tier_end > tier_start:
-            tokens_in_tier = tier_end - tier_start
-            cost_per_token = tier.get(cost_key) or tier.get(fallback_cost_key, 0)
-            total_cost += tokens_in_tier * cost_per_token
-            tokens_processed = tier_end
-
-    # After loop, check if any tokens remain (i.e., tokens > highest tier's end range)
-    # and charge them at the last tier's rate.
-    if tokens_processed < tokens and sorted_tiers:
-        last_tier = sorted_tiers[-1]
-        remaining_tokens = tokens - tokens_processed
-        cost_per_token = last_tier.get(cost_key) or last_tier.get(fallback_cost_key, 0)
-        total_cost += remaining_tokens * cost_per_token
-
-    return total_cost
-
-
 def _calculate_prompt_cost(
     breakdown: TokenBreakdown,
     model_info: ModelInfo,
@@ -123,12 +50,12 @@ def _calculate_prompt_cost(
 ) -> float:
     """Calculate total prompt cost including cached tokens."""
     if tiered_pricing:
-        text_cost = _calculate_tiered_cost(
+        text_cost = calculate_tiered_cost(
             tokens=breakdown.text_tokens,
             tiered_pricing=tiered_pricing,
             cost_key="input_cost_per_token",
         )
-        cache_cost = _calculate_tiered_cost(
+        cache_cost = calculate_tiered_cost(
             tokens=breakdown.cached_tokens,
             tiered_pricing=tiered_pricing,
             cost_key="cache_read_input_token_cost",
@@ -155,12 +82,12 @@ def _calculate_completion_cost(
 ) -> float:
     """Calculate total completion cost including reasoning tokens."""
     if tiered_pricing:
-        completion_cost = _calculate_tiered_cost(
+        completion_cost = calculate_tiered_cost(
             tokens=breakdown.completion_tokens,
             tiered_pricing=tiered_pricing,
             cost_key="output_cost_per_token",
         )
-        reasoning_cost = _calculate_tiered_cost(
+        reasoning_cost = calculate_tiered_cost(
             tokens=breakdown.reasoning_tokens,
             tiered_pricing=tiered_pricing,
             cost_key="output_cost_per_reasoning_token",

litellm/llms/databricks/chat/transformation.py

@@ -181,6 +181,10 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
             if key != "self" and value is not None:
                 setattr(self.__class__, key, value)
 
+    @property
+    def custom_llm_provider(self) -> Optional[str]:
+        return "databricks"
+
     @classmethod
     def get_config(cls):
         return super().get_config()
@@ -372,6 +376,7 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
             mapped_thinking = AnthropicConfig._map_reasoning_effort(
                 reasoning_effort=reasoning_effort_value,
                 model=model,
+                custom_llm_provider="databricks",
                 llm_provider="databricks",
             )
             if mapped_thinking is None:
@@ -379,7 +384,7 @@ class DatabricksConfig(DatabricksBase, OpenAILikeChatConfig, AnthropicConfig):
                 optional_params.pop("output_config", None)
             else:
                 optional_params["thinking"] = mapped_thinking
-                if AnthropicConfig._is_adaptive_thinking_model(model):
+                if AnthropicConfig._is_adaptive_thinking_model(model, "databricks"):
                     mapped_effort: Optional[str] = None
                     if isinstance(reasoning_effort_value, str):
                         mapped_effort = REASONING_EFFORT_TO_OUTPUT_CONFIG_EFFORT.get(reasoning_effort_value)

litellm/llms/github_copilot/messages/transformation.py

@@ -25,6 +25,10 @@ class GithubCopilotAnthropicMessagesConfig(AnthropicMessagesConfig):
         super().__init__()
         self.authenticator = Authenticator()
 
+    @property
+    def custom_llm_provider(self) -> Optional[str]:
+        return "github_copilot"
+
     def handles_web_search_natively(self) -> bool:
         """
         Copilot's /v1/messages endpoint does not execute ``web_search`` tools, so

litellm/llms/openai/chat/guardrail_translation/handler.py

@@ -14,11 +14,14 @@ Pattern Overview:
 This pattern can be replicated for other message formats (e.g., Anthropic).
 """
 
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union, cast
+from typing import TYPE_CHECKING, Any, Dict, List, Tuple, Union, cast
 
 import litellm
 from litellm._logging import verbose_proxy_logger
-from litellm.llms.base_llm.guardrail_translation.base_translation import BaseTranslation
+from litellm.llms.base_llm.guardrail_translation.base_translation import (
+    BaseTranslation,
+    StreamTransformSink,
+)
 from litellm.llms.base_llm.guardrail_translation.utils import (
     effective_skip_system_message_for_guardrail,
     effective_skip_tool_message_for_guardrail,
@@ -27,6 +30,9 @@ from litellm.llms.base_llm.guardrail_translation.utils import (
 )
 from litellm.main import stream_chunk_builder
 from litellm.types.llms.openai import AllMessageValues, ChatCompletionToolParam
+from litellm.types.proxy.guardrails.guardrail_hooks.generic_guardrail_api import (
+    coerce_stream_holdback_value,
+)
 from litellm.types.utils import (
     Choices,
     GenericGuardrailAPIInputs,
@@ -50,7 +56,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
     Methods can be overridden to customize behavior for different message formats.
     """
 
-    def get_structured_messages(self, data: dict) -> Optional[List[AllMessageValues]]:
+    def get_structured_messages(self, data: dict) -> List[AllMessageValues] | None:
         """
         Convert chat completions request data to OpenAI-spec structured messages.
 
@@ -65,7 +71,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         self,
         data: dict,
         guardrail_to_apply: "CustomGuardrail",
-        litellm_logging_obj: Optional[Any] = None,
+        litellm_logging_obj: Any | None = None,
     ) -> Any:
         """
         Process input messages by applying guardrails to text content.
@@ -80,7 +86,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         texts_to_check: List[str] = []
         images_to_check: List[str] = []
         tool_calls_to_check: List[ChatCompletionToolParam] = []
-        text_task_mappings: List[Tuple[int, Optional[int]]] = []
+        text_task_mappings: List[Tuple[int, int | None]] = []
         tool_call_task_mappings: List[Tuple[int, int]] = []
 
         # Step 1: Extract all text content, images, and tool calls
@@ -184,7 +190,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         texts_to_check: List[str],
         images_to_check: List[str],
         tool_calls_to_check: List[ChatCompletionToolParam],
-        text_task_mappings: List[Tuple[int, Optional[int]]],
+        text_task_mappings: List[Tuple[int, int | None]],
         tool_call_task_mappings: List[Tuple[int, int]],
         skip_system_message: bool = False,
         skip_tool_message: bool = False,
@@ -239,7 +245,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         self,
         messages: List[Dict[str, Any]],
         responses: List[str],
-        task_mappings: List[Tuple[int, Optional[int]]],
+        task_mappings: List[Tuple[int, int | None]],
     ) -> None:
         """
         Apply guardrail responses back to input message text content.
@@ -249,7 +255,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         for task_idx, guardrail_response in enumerate(responses):
             mapping = task_mappings[task_idx]
             msg_idx = cast(int, mapping[0])
-            content_idx_optional = cast(Optional[int], mapping[1])
+            content_idx_optional = cast(int | None, mapping[1])
 
             # Handle content
             content = messages[msg_idx].get("content", None)
@@ -291,9 +297,9 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         self,
         response: "ModelResponse",
         guardrail_to_apply: "CustomGuardrail",
-        litellm_logging_obj: Optional[Any] = None,
-        user_api_key_dict: Optional[Any] = None,
-        request_data: Optional[dict] = None,
+        litellm_logging_obj: Any | None = None,
+        user_api_key_dict: Any | None = None,
+        request_data: dict | None = None,
     ) -> Any:
         """
         Process output response by applying guardrails to text content.
@@ -320,7 +326,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         texts_to_check: List[str] = []
         images_to_check: List[str] = []
         tool_calls_to_check: List[Dict[str, Any]] = []
-        text_task_mappings: List[Tuple[int, Optional[int]]] = []
+        text_task_mappings: List[Tuple[int, int | None]] = []
         tool_call_task_mappings: List[Tuple[int, int]] = []
         # text_task_mappings: Track (choice_index, content_index) for each text
         # content_index is None for string content, int for list content
@@ -402,9 +408,10 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         self,
         responses_so_far: List["ModelResponseStream"],
         guardrail_to_apply: "CustomGuardrail",
-        litellm_logging_obj: Optional[Any] = None,
-        user_api_key_dict: Optional[Any] = None,
-        request_data: Optional[dict] = None,
+        litellm_logging_obj: Any | None = None,
+        user_api_key_dict: Any | None = None,
+        request_data: dict | None = None,
+        stream_transform_sink: StreamTransformSink | None = None,
     ) -> List["ModelResponseStream"]:
         """
         Process output streaming responses by applying guardrails to text content.
@@ -414,14 +421,50 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
             guardrail_to_apply: The guardrail instance to apply
             litellm_logging_obj: Optional logging object
             user_api_key_dict: User API key metadata to pass to guardrails
+            stream_transform_sink: Optional out-parameter for the streaming text
+                transformation path. When provided, the guardrail runs over the raw
+                accumulated text (``responses_so_far`` is left untouched so it stays
+                a correct raw accumulator across rounds) and the guardrailed text
+                plus requested holdback are reported per choice on the sink.
 
         Returns:
-            Modified list of responses with guardrail applied to content
+            The (unmodified) list of responses.
 
         Response Format Support:
             - String content: choice.message.content = "text here"
             - List content: choice.message.content = [{"type": "text", "text": "text here"}, ...]
         """
+        if stream_transform_sink is not None:
+            await self._process_streaming_transform(
+                responses_so_far=responses_so_far,
+                guardrail_to_apply=guardrail_to_apply,
+                litellm_logging_obj=litellm_logging_obj,
+                user_api_key_dict=user_api_key_dict,
+                request_data=request_data,
+                sink=stream_transform_sink,
+            )
+            return responses_so_far
+
+        return await self._process_streaming_block_only(
+            responses_so_far=responses_so_far,
+            guardrail_to_apply=guardrail_to_apply,
+            litellm_logging_obj=litellm_logging_obj,
+            user_api_key_dict=user_api_key_dict,
+            request_data=request_data,
+        )
+
+    async def _process_streaming_block_only(
+        self,
+        *,
+        responses_so_far: list["ModelResponseStream"],
+        guardrail_to_apply: "CustomGuardrail",
+        litellm_logging_obj: Any | None,
+        user_api_key_dict: Any | None,
+        request_data: dict | None,
+    ) -> list["ModelResponseStream"]:
+        """Block-only streaming path: run the guardrail so an in-flight BLOCK can
+        terminate the stream. Text rewrites are not propagated to the client here
+        (see ``_process_streaming_transform`` for the incremental_diff path)."""
         # check if the stream has ended
         has_stream_ended = False
         for chunk in responses_so_far:
@@ -467,7 +510,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         # Step 2: Create lists for guardrail processing
         texts_to_check: List[str] = []
         images_to_check: List[str] = []
-        task_mappings: List[Tuple[int, Optional[int]]] = []
+        task_mappings: List[Tuple[int, int | None]] = []
         # Track (choice_index, content_index) for each combined text
 
         for (map_choice_idx, map_content_idx), combined_text in combined_texts.items():
@@ -520,9 +563,109 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
 
         return responses_so_far
 
+    @staticmethod
+    def _accumulate_string_content_by_choice_index(
+        responses_so_far: list["ModelResponseStream"],
+    ) -> dict[int, str]:
+        """Accumulate raw string ``delta.content`` per choice, keyed by
+        ``StreamingChoices.index`` (not enumerate position, which collapses to 0
+        when each chunk carries a single non-zero-indexed choice for ``n > 1``).
+
+        Only string content participates; list-of-blocks content is out of scope
+        for the incremental transform path. Reads ``responses_so_far`` without
+        mutating it so it stays a correct raw accumulator across rounds.
+        """
+        accumulated: dict[int, str] = {}
+        for response in responses_so_far:
+            for choice in response.choices:
+                if isinstance(choice, litellm.StreamingChoices):
+                    content = choice.delta.content
+                elif isinstance(choice, litellm.Choices):
+                    content = choice.message.content
+                else:
+                    continue
+                if isinstance(content, str) and content:
+                    idx = getattr(choice, "index", 0) or 0
+                    accumulated[idx] = accumulated.get(idx, "") + content
+        return accumulated
+
+    async def _process_streaming_transform(
+        self,
+        *,
+        responses_so_far: list["ModelResponseStream"],
+        guardrail_to_apply: "CustomGuardrail",
+        litellm_logging_obj: Any | None,
+        user_api_key_dict: Any | None,
+        request_data: dict | None,
+        sink: StreamTransformSink,
+    ) -> None:
+        """Run the guardrail over the raw accumulated text and report the
+        guardrailed text plus requested holdback per choice on ``sink``.
+
+        Unlike the block-only path this never mutates ``responses_so_far``: it
+        re-derives the raw accumulated text every round (so a rewrite guardrail
+        always sees consistent input) and hands the result back out of band.
+        """
+        raw_by_index = self._accumulate_string_content_by_choice_index(responses_so_far)
+        if not raw_by_index:
+            sink.mutated_text_per_choice = {}
+            sink.holdback_per_choice = {}
+            return
+
+        # Fix #2 — sort by StreamingChoices.index so an n>1 stream that emits
+        # choice 1 before choice 0 still hands the guardrail texts in a
+        # deterministic index order. Without this, the guardrail's returned
+        # texts (aligned to the input order it received) would map back to the
+        # wrong choice indices when we rebuild the sink dicts by
+        # ``enumerate(indices)``.
+        indices = sorted(raw_by_index.keys())
+        texts_to_check = [raw_by_index[i] for i in indices]
+
+        if request_data is None:
+            request_data = {"responses": responses_so_far}
+        elif "responses" not in request_data:
+            request_data["responses"] = responses_so_far
+        if "litellm_metadata" not in request_data:
+            user_metadata = self.transform_user_api_key_dict_to_metadata(user_api_key_dict)
+            if user_metadata:
+                request_data["litellm_metadata"] = user_metadata
+
+        inputs = GenericGuardrailAPIInputs(texts=texts_to_check)
+        if responses_so_far and getattr(responses_so_far[0], "model", None):
+            inputs["model"] = responses_so_far[0].model
+        guardrailed_inputs = await guardrail_to_apply.apply_guardrail(
+            inputs=inputs,
+            request_data=request_data,
+            input_type="response",
+            logging_obj=litellm_logging_obj,
+        )
+
+        returned_texts = guardrailed_inputs.get("texts")
+        # No "texts" key means the guardrail made no change (action NONE): the raw
+        # accumulated text is the guardrailed text. A present-but-shorter list is a
+        # guardrail contract violation; those choices are omitted below (withheld,
+        # not emitted raw) so a malformed response fails closed instead of leaking.
+        if returned_texts is None:
+            returned_texts = texts_to_check
+        elif len(returned_texts) < len(texts_to_check):
+            verbose_proxy_logger.warning(
+                "OpenAI Chat Completions: guardrail returned %s transformed texts for %s inputs on the "
+                "streaming transform path; withholding the unmatched choices to fail closed.",
+                len(returned_texts),
+                len(texts_to_check),
+            )
+
+        holdback = guardrailed_inputs.get("stream_holdback_chars") or []
+        sink.mutated_text_per_choice = {
+            idx: returned_texts[i] for i, idx in enumerate(indices) if i < len(returned_texts)
+        }
+        sink.holdback_per_choice = {
+            indices[i]: coerce_stream_holdback_value(holdback[i]) for i in range(len(indices)) if i < len(holdback)
+        }
+
     def _combine_streaming_texts(
         self, responses_so_far: List["ModelResponseStream"]
-    ) -> Dict[Tuple[int, Optional[int]], str]:
+    ) -> Dict[Tuple[int, int | None], str]:
         """
         Combine all streaming chunks into complete text per choice.
 
@@ -534,7 +677,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         Returns:
             Dict mapping (choice_idx, content_idx) to combined text string
         """
-        combined_texts: Dict[Tuple[int, Optional[int]], str] = {}
+        combined_texts: Dict[Tuple[int, int | None], str] = {}
 
         for response_idx, response in enumerate(responses_so_far):
             for choice_idx, choice in enumerate(response.choices):
@@ -550,7 +693,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
 
                 if isinstance(content, str):
                     # String content - accumulate for this choice
-                    str_key: Tuple[int, Optional[int]] = (choice_idx, None)
+                    str_key: Tuple[int, int | None] = (choice_idx, None)
                     if str_key not in combined_texts:
                         combined_texts[str_key] = ""
                     combined_texts[str_key] += content
@@ -560,7 +703,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                     for content_idx, content_item in enumerate(content):
                         text_str = content_item.get("text")
                         if text_str:
-                            list_key: Tuple[int, Optional[int]] = (
+                            list_key: Tuple[int, int | None] = (
                                 choice_idx,
                                 content_idx,
                             )
@@ -607,7 +750,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         texts_to_check: List[str],
         images_to_check: List[str],
         tool_calls_to_check: List[Dict[str, Any]],
-        text_task_mappings: List[Tuple[int, Optional[int]]],
+        text_task_mappings: List[Tuple[int, int | None]],
         tool_call_task_mappings: List[Tuple[int, int]],
     ) -> None:
         """
@@ -619,7 +762,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
 
         # Determine content source and tool calls based on choice type
         content = None
-        tool_calls: Optional[List[Any]] = None
+        tool_calls: List[Any] | None = None
         if isinstance(choice, litellm.Choices):
             content = choice.message.content
             tool_calls = choice.message.tool_calls
@@ -662,7 +805,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                     tool_calls_to_check.append(tool_call_dict)
                     tool_call_task_mappings.append((choice_idx, int(tool_call_idx)))
 
-    def _convert_tool_call_to_dict(self, tool_call: Union[Dict[str, Any], Any]) -> Optional[Dict[str, Any]]:
+    def _convert_tool_call_to_dict(self, tool_call: Union[Dict[str, Any], Any]) -> Dict[str, Any] | None:
         """
         Convert a tool call object to dictionary format.
 
@@ -691,7 +834,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         self,
         response: "ModelResponse",
         responses: List[str],
-        task_mappings: List[Tuple[int, Optional[int]]],
+        task_mappings: List[Tuple[int, int | None]],
     ) -> None:
         """
         Apply guardrail text responses back to output response.
@@ -701,7 +844,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         for task_idx, guardrail_response in enumerate(responses):
             mapping = task_mappings[task_idx]
             choice_idx = cast(int, mapping[0])
-            content_idx_optional = cast(Optional[int], mapping[1])
+            content_idx_optional = cast(int | None, mapping[1])
 
             choice = cast(Choices, response.choices[choice_idx])
 
@@ -755,7 +898,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         self,
         responses: List["ModelResponseStream"],
         guardrailed_texts: List[str],
-        task_mappings: List[Tuple[int, Optional[int]]],
+        task_mappings: List[Tuple[int, int | None]],
     ) -> None:
         """
         Apply guardrail responses back to output streaming responses.
@@ -771,16 +914,16 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
         Override this method to customize how responses are applied to streaming responses.
         """
         # Build a mapping of what guardrailed text to use for each (choice_idx, content_idx)
-        guardrail_map: Dict[Tuple[int, Optional[int]], str] = {}
+        guardrail_map: Dict[Tuple[int, int | None], str] = {}
         for task_idx, guardrail_response in enumerate(guardrailed_texts):
             mapping = task_mappings[task_idx]
             choice_idx = cast(int, mapping[0])
-            content_idx_optional = cast(Optional[int], mapping[1])
+            content_idx_optional = cast(int | None, mapping[1])
             guardrail_map[(choice_idx, content_idx_optional)] = guardrail_response
 
         # Track which choices we've already set the guardrailed text for
         # Key: (choice_idx, content_idx), Value: boolean (True if already set)
-        already_set: Dict[Tuple[int, Optional[int]], bool] = {}
+        already_set: Dict[Tuple[int, int | None], bool] = {}
 
         # Iterate through all responses and update content
         for response_idx, response in enumerate(responses):
@@ -797,7 +940,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
 
                 if isinstance(content, str):
                     # String content
-                    str_key: Tuple[int, Optional[int]] = (choice_idx_in_response, None)
+                    str_key: Tuple[int, int | None] = (choice_idx_in_response, None)
                     if str_key in guardrail_map:
                         if str_key not in already_set:
                             # First chunk - set the complete guardrailed text
@@ -817,7 +960,7 @@ class OpenAIChatCompletionsHandler(BaseTranslation):
                     # List content - handle each content item
                     for content_idx, content_item in enumerate(content):
                         if "text" in content_item:
-                            list_key: Tuple[int, Optional[int]] = (
+                            list_key: Tuple[int, int | None] = (
                                 choice_idx_in_response,
                                 content_idx,
                             )

litellm/llms/openai/responses/transformation.py

@@ -20,6 +20,8 @@ from litellm.types.utils import LlmProviders
 
 from ..common_utils import OpenAIError
 
+OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS = 16
+
 if TYPE_CHECKING:
     from litellm.litellm_core_utils.litellm_logging import Logging as _LiteLLMLoggingObj
 
@@ -59,6 +61,19 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
             key="supports_none_reasoning_effort",
         )
 
+    @staticmethod
+    def _enforce_min_max_output_tokens(max_output_tokens: "int | None") -> "int | None":
+        """Raise sub-minimum max_output_tokens up to the OpenAI Responses API minimum.
+
+        OpenAI's Responses API rejects max_output_tokens below 16 for every model
+        (not gpt-5 specific), so a client like Claude Code that sends a max_tokens=1
+        warmup probe on model switch would otherwise 400. Values that are None or
+        already at/above the minimum are returned unchanged.
+        """
+        if isinstance(max_output_tokens, int) and max_output_tokens < OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS:
+            return OPENAI_RESPONSES_API_MIN_MAX_OUTPUT_TOKENS
+        return max_output_tokens
+
     def get_supported_openai_params(self, model: str) -> list:
         """
         All OpenAI Responses API params are supported
@@ -92,6 +107,9 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
         """
         params = dict(response_api_optional_params)
 
+        if "max_output_tokens" in params:
+            params["max_output_tokens"] = self._enforce_min_max_output_tokens(params.get("max_output_tokens"))
+
         if self._is_gpt_5_model(model=model):
             temperature = params.get("temperature")
             if temperature is not None and temperature != 1:

litellm/llms/openai_like/dynamic_config.py

@@ -91,7 +91,7 @@ def create_config_class(provider: SimpleProviderConfig):
         def get_supported_openai_params(self, model: str) -> list:
             """Get supported OpenAI params, excluding tool-related params for models
             that don't support function calling."""
-            from litellm.utils import supports_function_calling
+            from litellm.utils import supports_function_calling, supports_reasoning
 
             supported_params = super().get_supported_openai_params(model=model)
 
@@ -113,6 +113,10 @@ def create_config_class(provider: SimpleProviderConfig):
                     f"function calling — removed tool-related params from supported params."
                 )
 
+            _supports_reasoning = supports_reasoning(model=model, custom_llm_provider=provider.slug)
+            if _supports_reasoning and "reasoning_effort" not in supported_params:
+                supported_params.append("reasoning_effort")
+
             return supported_params
 
         def map_openai_params(

litellm/llms/openai_like/messages/transformation.py

@@ -1,8 +1,11 @@
 from typing import Any, Optional
 
+import litellm
 from litellm.llms.anthropic.experimental_pass_through.messages.transformation import (
     AnthropicMessagesConfig,
 )
+from litellm.llms.openai_like.json_loader import SimpleProviderConfig
+from litellm.secret_managers.main import get_secret_str
 
 DEFAULT_ANTHROPIC_API_VERSION = "2023-06-01"
 
@@ -67,3 +70,69 @@ class OpenAILikeAnthropicMessagesConfig(AnthropicMessagesConfig):
         if base.endswith("/v1"):
             base = base[: -len("/v1")]
         return f"{base}/v1/messages"
+
+
+class JSONProviderAnthropicMessagesConfig(OpenAILikeAnthropicMessagesConfig):
+    """
+    Provider-level native Anthropic Messages passthrough for JSON-configured
+    OpenAI-compatible providers whose ``supported_endpoints`` in providers.json
+    includes ``"/v1/messages"``. Resolves the api key and api base from the
+    provider's configured env vars, then forwards the Anthropic payload
+    untranslated like ``OpenAILikeAnthropicMessagesConfig``.
+    """
+
+    def __init__(self, provider: SimpleProviderConfig):
+        super().__init__()
+        self._provider = provider
+
+    @property
+    def custom_llm_provider(self) -> Optional[str]:
+        return self._provider.slug
+
+    def should_strip_billing_metadata(self) -> bool:
+        return True
+
+    def _resolve_api_key(self, api_key: Optional[str]) -> Optional[str]:
+        return api_key or get_secret_str(self._provider.api_key_env) or litellm.api_key
+
+    def _resolve_api_base(self, api_base: Optional[str]) -> str:
+        env_api_base = get_secret_str(self._provider.api_base_env) if self._provider.api_base_env else None
+        return api_base or env_api_base or self._provider.base_url
+
+    def validate_anthropic_messages_environment(
+        self,
+        headers: dict[str, str],
+        model: str,
+        messages: list[Any],
+        optional_params: dict,
+        litellm_params: dict,
+        api_key: Optional[str] = None,
+        api_base: Optional[str] = None,
+    ) -> tuple[dict[str, str], Optional[str]]:
+        return super().validate_anthropic_messages_environment(
+            headers=headers,
+            model=model,
+            messages=messages,
+            optional_params=optional_params,
+            litellm_params=litellm_params,
+            api_key=self._resolve_api_key(api_key),
+            api_base=api_base,
+        )
+
+    def get_complete_url(
+        self,
+        api_base: Optional[str],
+        api_key: Optional[str],
+        model: str,
+        optional_params: dict,
+        litellm_params: dict,
+        stream: Optional[bool] = None,
+    ) -> str:
+        return super().get_complete_url(
+            api_base=self._resolve_api_base(api_base),
+            api_key=api_key,
+            model=model,
+            optional_params=optional_params,
+            litellm_params=litellm_params,
+            stream=stream,
+        )

litellm/llms/openai_like/providers.json

@@ -168,6 +168,13 @@
     },
     "supported_endpoints": ["/v1/chat/completions", "/v1/responses"]
   },
+  "meta": {
+    "base_url": "https://api.meta.ai/v1",
+    "api_key_env": "META_API_KEY",
+    "api_base_env": "META_API_BASE",
+    "base_class": "openai_gpt",
+    "supported_endpoints": ["/v1/chat/completions", "/v1/responses", "/v1/messages"]
+  },
   "pinstripes": {
     "base_url": "https://pinstripes.io/v1",
     "api_key_env": "PINSTRIPES_API_KEY",

litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py

@@ -998,6 +998,8 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
                 response_modalities.append("IMAGE")
             elif modality == "audio":
                 response_modalities.append("AUDIO")
+            elif modality == "video":
+                response_modalities.append("VIDEO")
             else:
                 response_modalities.append("MODALITY_UNSPECIFIED")
         return response_modalities

litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/experimental_pass_through/transformation.py

@@ -17,6 +17,10 @@ from ..output_params_utils import sanitize_vertex_anthropic_output_params
 
 
 class VertexAIPartnerModelsAnthropicMessagesConfig(AnthropicMessagesConfig, VertexBase):
+    @property
+    def custom_llm_provider(self) -> Optional[str]:
+        return "vertex_ai"
+
     def should_strip_billing_metadata(self) -> bool:
         return True
 

litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/output_params_utils.py

@@ -26,7 +26,7 @@ def _model_accepts_output_config_effort(model: str) -> bool:
     """
     from litellm.llms.anthropic.chat.transformation import AnthropicConfig
 
-    return AnthropicConfig._model_supports_effort_param(model)
+    return AnthropicConfig._model_supports_effort_param(model, "vertex_ai")
 
 
 def sanitize_vertex_anthropic_output_params(data: dict, model: str) -> None:

litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/transformation.py

@@ -112,6 +112,7 @@ class VertexAIAnthropicConfig(AnthropicConfig):
             prompt_caching_set=self.is_cache_control_set(messages),
             file_id_used=self.is_file_id_used(messages),
             mcp_server_used=self.is_mcp_server_used(optional_params.get("mcp_servers")),
+            custom_llm_provider="vertex_ai",
         )
 
         beta_set = set(auto_betas)

litellm/main.py

@@ -92,7 +92,10 @@ from litellm.litellm_core_utils.completion_timeout import CompletionTimeout
 from litellm.litellm_core_utils.request_timeout_resolver import (
     get_configured_request_timeout,
 )
-from litellm.litellm_core_utils.get_litellm_params import OPTIONAL_KWARGS_KEYS
+from litellm.litellm_core_utils.get_litellm_params import (
+    AWS_CREDENTIAL_KWARGS_KEYS,
+    OPTIONAL_KWARGS_KEYS,
+)
 from litellm.litellm_core_utils.dd_tracing import tracer
 from litellm.litellm_core_utils.get_provider_specific_headers import (
     ProviderSpecificHeaderUtils,
@@ -5322,7 +5325,7 @@ def completion(  # type: ignore
             tpm=kwargs.get("tpm"),
             rpm=kwargs.get("rpm"),
             use_xai_oauth=kwargs.get("use_xai_oauth", False),
-            aws_bedrock_project_id=kwargs.get("aws_bedrock_project_id"),
+            **{key: kwargs[key] for key in AWS_CREDENTIAL_KWARGS_KEYS if key in kwargs},
         )
         cast(LiteLLMLoggingObj, logging).update_environment_variables(
             model=model,

litellm/models/mcp_server.py

@@ -96,6 +96,7 @@ class LiteLLM_MCPServerTable(LiteLLMPydanticObjectBase):
     available_on_public_internet: bool = True
     delegate_auth_to_upstream: bool = False
     oauth_passthrough: bool = False
+    dcr_bridge: Optional[bool] = None
     is_byok: bool = False
     byok_description: List[str] = Field(default_factory=list)
     byok_api_key_help_url: Optional[str] = None

litellm/models/verification_token.py

@@ -36,6 +36,7 @@ class LiteLLM_VerificationToken(LiteLLMPydanticObjectBase):
     budget_reset_at: Optional[datetime] = None
     allowed_cache_controls: Optional[list] = []
     allowed_routes: Optional[list] = []
+    key_type: str | None = None
     permissions: Dict = {}
     model_spend: Dict = {}
     model_max_budget: Dict = {}

litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py

@@ -1,12 +1,26 @@
 import re
+from datetime import datetime, timezone
 from typing import Dict, List, Optional, Set, Tuple, cast
 
 from fastapi import HTTPException
 from starlette.datastructures import Headers
 from starlette.requests import Request
 from starlette.types import Scope
+from typing_extensions import assert_never
 
+import litellm
 from litellm._logging import verbose_logger
+from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import (
+    BridgeEnvelopeAdmitted,
+    BridgeEnvelopeInvalid,
+    NotBridgeEnvelope,
+    envelope_keys_from_master_key,
+    is_bridge_envelope_shaped,
+    resolve_bridge_envelope,
+)
+from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (
+    EnvelopeIdentity,
+)
 from litellm.proxy._types import (
     UI_TEAM_ID,
     LiteLLM_TeamTable,
@@ -17,12 +31,17 @@ from litellm.proxy._types import (
     UserAPIKeyAuth,
 )
 from litellm.proxy.auth.ip_address_utils import IPAddressUtils
-from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
+from litellm.proxy.auth.user_api_key_auth import (
+    _run_centralized_common_checks,
+    user_api_key_auth,
+)
+from litellm.proxy.common_utils.http_parsing_utils import _read_request_body
 from litellm.proxy.common_utils.user_api_key_cache import get_management_object_ttl
 from litellm.repositories.table_repositories import (
     AgentsRepository,
     MCPServerRepository,
 )
+from litellm.types.mcp_server.mcp_server_manager import MCPServer
 
 
 def _parse_mcp_server_names_from_path(path: str, mcp_servers_header: Optional[List[str]] = None) -> Optional[List[str]]:
@@ -226,6 +245,29 @@ class MCPRequestHandler:
             client_ip=IPAddressUtils.get_mcp_client_ip(request),
         ):
             validated_user_api_key_auth = UserAPIKeyAuth()
+        elif (
+            (
+                bridge_delegate_target := MCPRequestHandler._single_dcr_bridge_delegate_target(
+                    path=request_route,
+                    mcp_servers=mcp_servers,
+                    client_ip=IPAddressUtils.get_mcp_client_ip(request),
+                )
+            )
+            is not None
+            and oauth2_headers
+            and is_bridge_envelope_shaped(oauth2_headers["Authorization"])
+        ):
+            # A single DCR-bridge oauth_delegate target carrying an envelope-shaped
+            # Authorization: open the envelope, admit under its recovered identity, and
+            # inject the inner upstream token for egress. A non-envelope bearer on the same
+            # server is NOT admitted here — it falls through to the oauth2 arm, which 401s.
+            validated_user_api_key_auth, mcp_server_auth_headers = await MCPRequestHandler._admit_dcr_bridge_delegate(
+                server=bridge_delegate_target,
+                authorization_value=oauth2_headers["Authorization"],
+                mcp_server_auth_headers=mcp_server_auth_headers,
+                request=request,
+                route=request_route,
+            )
         elif oauth2_headers:
             # Authorization on a non-delegated server: the bearer must be a real
             # LiteLLM credential, so a failed validation is a genuine 401/403 and
@@ -432,6 +474,334 @@ class MCPRequestHandler:
                 return False
         return True
 
+    @staticmethod
+    def _single_dcr_bridge_delegate_target(
+        path: str, mcp_servers: Optional[List[str]], client_ip: Optional[str]
+    ) -> Optional[MCPServer]:
+        """The one DCR-bridge ``oauth_delegate`` server this request targets, or ``None``.
+
+        Returns the server only when EXACTLY ONE target resolves and it is both
+        ``is_oauth_delegate`` and ``is_dcr_bridge``. Fails closed (``None``) on a
+        multi-target request, an unresolved target, or a non-matching server, so the
+        envelope admission arm never fires for an aggregate scope or a server that did not
+        opt into the bridge. Mirrors :meth:`_target_servers_are_true_passthrough`.
+        """
+        from litellm.proxy._experimental.mcp_server.mcp_server_manager import (
+            global_mcp_server_manager,
+        )
+
+        target_names = MCPRequestHandler._resolve_target_server_names(path=path, mcp_servers_header=mcp_servers)
+        if len(target_names) != 1:
+            return None
+        server = global_mcp_server_manager.get_mcp_server_by_name(target_names[0], client_ip=client_ip)
+        if server is None or not server.is_oauth_delegate or not server.is_dcr_bridge:
+            return None
+        # Egress resolves the injected per-server token only by alias / server_name; a server with
+        # neither cannot receive the forwarded token, so fail closed rather than admit-and-drop.
+        if not (server.server_name or server.alias):
+            return None
+        return server
+
+    @staticmethod
+    async def _admit_dcr_bridge_delegate(
+        server: MCPServer,
+        authorization_value: str,
+        mcp_server_auth_headers: Optional[Dict[str, Dict[str, str]]],
+        request: Request,
+        route: str,
+    ) -> Tuple[UserAPIKeyAuth, Optional[Dict[str, Dict[str, str]]]]:
+        """Open the bridge envelope and admit the caller under the live key it references.
+
+        The envelope's signature proves the user authenticated when it was minted, but
+        authorization is resolved fresh here rather than trusted from the envelope: the
+        sealed ``key_hash`` reloads the current ``UserAPIKeyAuth`` record, and the admitted
+        identity then runs through the standard pipeline's centralized policy gate, so the
+        key's present restrictions and revocation state gate the request instead of a
+        snapshot frozen at mint time. The inner upstream token is injected under the
+        server's per-server auth-header key so egress forwards it via the
+        ``PassthroughConfig`` override; the envelope ``Authorization`` the leak-defense
+        strips never reaches the upstream. A new headers dict is returned rather than
+        mutating the input. Fails closed with a 401 on an invalid or expired envelope, or
+        when the referenced key is missing, blocked, or expired, its owner is
+        SCIM-deactivated, or the centralized policy gate rejects it (blocked team or
+        project, org or budget limits).
+
+        The sealed token is keyed alias-first, matching the order egress resolves
+        (``lookup_mcp_server_auth_in_headers`` tries ``alias`` before ``server_name``). Keying
+        under ``server_name`` would leave a caller-supplied ``x-mcp-{alias}-authorization`` at the
+        higher-priority alias slot, pairing the admitted identity with an attacker's upstream
+        credential; the alias-keyed injection overwrites any such caller value.
+        """
+        from litellm.proxy.proxy_server import master_key
+
+        if not master_key:
+            raise HTTPException(status_code=500, detail="Server misconfigured: master_key is not set")
+
+        await MCPRequestHandler._run_pre_db_read_auth_checks(request=request, route=route)
+
+        keys = envelope_keys_from_master_key(master_key)
+        result = resolve_bridge_envelope(authorization_value, keys, datetime.now(timezone.utc), server.server_id)
+        match result:
+            case BridgeEnvelopeAdmitted():
+                header_key = server.alias or server.server_name
+                if header_key is None:
+                    raise HTTPException(status_code=500, detail="Server misconfigured: MCP server has no routable name")
+                admitted = await MCPRequestHandler._reload_admitted_principal(result.identity)
+                await MCPRequestHandler._enforce_admitted_live_policy(admitted=admitted, request=request, route=route)
+                injected = {header_key: {"Authorization": result.upstream_authorization.get_secret_value()}}
+                new_headers = {**(mcp_server_auth_headers or {}), **injected}
+                return admitted, new_headers
+            case BridgeEnvelopeInvalid() | NotBridgeEnvelope():
+                raise HTTPException(status_code=401, detail="Invalid or expired credential")
+            case _:
+                assert_never(result)
+
+    @staticmethod
+    async def _run_pre_db_read_auth_checks(request: Request, route: str) -> None:
+        """Run the proxy-wide gates ``user_api_key_auth`` applies before any key lookup: the
+        request-size and body-safety limits, the IP allowlist, and the ``general_settings``
+        route allowlist. The envelope arm bypasses ``user_api_key_auth`` (it opens the envelope
+        and reloads the identity itself), so without this a caller blocked by IP or hitting a
+        proxy route the allowlist forbids would be admitted through an envelope where the same
+        principal presented on the normal MCP admission path would be rejected. Runs before the
+        envelope crypto so a disallowed caller is turned away before any work, mirroring the
+        standard pipeline's pre-DB ordering. Violations raise the gate's own status (an IP or
+        route block is a 403, an oversized body its own limit error)."""
+        from litellm.proxy.auth.auth_utils import pre_db_read_auth_checks
+
+        await pre_db_read_auth_checks(
+            request=request,
+            request_data=await _read_request_body(request=request),
+            route=route,
+        )
+
+    @staticmethod
+    async def _reload_admitted_principal(identity: EnvelopeIdentity) -> UserAPIKeyAuth:
+        """Reload the live litellm record the envelope's subject references.
+
+        Dispatches on the sealed subject type: a ``key_hash`` reloads the virtual key that
+        minted the envelope (the scripted two-header client that presents a litellm key at the
+        token endpoint), a ``user_id`` reloads the user that authenticated interactively (the
+        DCR client, whose SSO login at the bridged authorize yields a user, not a key). Both
+        return a ``UserAPIKeyAuth`` the caller runs through the centralized policy gate, so
+        team/project/org/budget/SCIM enforcement is identical to the principal presenting
+        itself directly."""
+        match identity.subject_type:
+            case "key_hash":
+                return await MCPRequestHandler._reload_admitted_key(identity.subject)
+            case "user_id":
+                return await MCPRequestHandler._reload_admitted_user(identity.subject)
+            case _:
+                assert_never(identity.subject_type)
+
+    @staticmethod
+    async def _reload_admitted_user(user_id: str) -> UserAPIKeyAuth:
+        """Reload the live user an interactively-minted envelope references and admit them as
+        themselves.
+
+        The DCR client authenticates via SSO at the bridged authorize, which yields a user
+        subject rather than a virtual key, so the envelope admits under the user's own
+        identity: the reloaded ``user_id`` and the user's own MCP object permission ride on the
+        returned ``UserAPIKeyAuth``, and the SAME ``get_allowed_mcp_servers`` the key path uses then
+        computes which servers the user may reach, so the user's litellm MCP grants and access groups
+        gate the request exactly as a key's do. Only the user's OWN object permission is bound: a
+        ``UserAPIKeyAuth`` carries a single ``team_id`` while a user may belong to many teams, so
+        team-inherited MCP grants for a user are a follow-up (they need a many-teams union
+        ``get_allowed_mcp_servers`` does not do off one auth object). The caller's centralized policy
+        gate enforces the user's live budget and org state, and a SCIM-deactivated owner fails closed.
+
+        Error handling mirrors the key path's retryable-503 contract, but ``get_user_object`` defeats a
+        type-based check: where ``get_key_object`` raises a typed ``ProxyException`` for a missing key
+        and lets a DB outage propagate raw, ``get_user_object`` catches every DB failure and re-raises a
+        bare ``ValueError``, so a missing user and a real outage look identical and the original error
+        survives only as ``__context__``. ``_raise_503_if_db_unavailable`` therefore walks the cause
+        chain: a transient DB outage still surfaces as a retryable 503, while a missing user, or any
+        other non-outage resolution failure, fails closed as a 401 rather than an opaque 500. The
+        object-permission load shares this one boundary, so an outage there is classified the same
+        way (``get_object_permission`` itself swallows a failed load to ``None``, matching how
+        ``get_key_object`` best-effort-loads a key's object permission)."""
+        from litellm.proxy.auth.auth_checks import get_object_permission, get_user_object
+        from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
+
+        if prisma_client is None:
+            raise HTTPException(status_code=500, detail="Server misconfigured: no database connection")
+        try:
+            user_object = await get_user_object(
+                user_id=user_id,
+                prisma_client=prisma_client,
+                user_api_key_cache=user_api_key_cache,
+                user_id_upsert=False,
+            )
+            # Resolve the user's own MCP object permission (get_user_object does not load it) so the shared
+            # get_allowed_mcp_servers can grant the user their litellm-granted servers. Reuses the same
+            # get_object_permission resolver the key and team paths use; no permission logic is duplicated.
+            object_permission = user_object.object_permission if user_object is not None else None
+            if user_object is not None and object_permission is None and user_object.object_permission_id:
+                object_permission = await get_object_permission(
+                    object_permission_id=user_object.object_permission_id,
+                    prisma_client=prisma_client,
+                    user_api_key_cache=user_api_key_cache,
+                )
+        except (ProxyException, HTTPException):
+            raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
+        except Exception as e:  # noqa: BLE001  # a DB outage anywhere in the resolution is a retryable 503, not an opaque 500; anything else fails closed as 401
+            MCPRequestHandler._raise_503_if_db_unavailable(e)
+            raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
+        if user_object is None:
+            raise HTTPException(status_code=401, detail="Invalid or expired credential")
+        if isinstance(user_object.metadata, dict) and user_object.metadata.get("scim_active") is False:
+            raise HTTPException(status_code=401, detail="Invalid or expired credential")
+        return UserAPIKeyAuth(
+            user_id=user_object.user_id,
+            user_role=user_object.user_role,
+            object_permission=object_permission,
+            object_permission_id=user_object.object_permission_id,
+        )
+
+    @staticmethod
+    async def _reload_admitted_key(key_hash: str) -> UserAPIKeyAuth:
+        """Reload the live key record an admitted envelope references and re-check live policy.
+
+        Resolving the current ``UserAPIKeyAuth`` (cache first, then DB) is what stops the
+        envelope from carrying frozen authority: the key's present team/org/object-permission
+        restrictions ride on the returned object, and a key that has since been deleted,
+        blocked, or expired fails closed with a 401 here rather than being admitted as an
+        unrestricted identity. ``get_key_object`` raises for a hash with no key row; a
+        blocked or expired row is rejected explicitly because ``get_key_object`` resolves a
+        row without applying those checks (the main ``user_api_key_auth`` pipeline enforces
+        them downstream, which this admission path bypasses). The owner's SCIM state is the
+        other builder-inline check mirrored here, so IdP offboarding revokes every envelope
+        minted under the user's keys rather than leaving them live until expiry. Team,
+        project, org, and budget state are NOT re-checked here; the caller runs the admitted
+        identity through ``_enforce_admitted_live_policy`` for those.
+        """
+        from litellm.proxy.auth.auth_checks import get_key_object
+        from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
+
+        if prisma_client is None:
+            raise HTTPException(status_code=500, detail="Server misconfigured: no database connection")
+        try:
+            key_object = await get_key_object(
+                hashed_token=key_hash,
+                prisma_client=prisma_client,
+                user_api_key_cache=user_api_key_cache,
+            )
+        except (ProxyException, HTTPException):
+            raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
+        except Exception as e:  # noqa: BLE001  # a DB outage during reload is a retryable 503, not an opaque 500
+            MCPRequestHandler._raise_503_if_db_unavailable(e)
+            raise
+        if not MCPRequestHandler._admitted_key_is_active(key_object):
+            raise HTTPException(status_code=401, detail="Invalid or expired credential")
+        await MCPRequestHandler._reject_if_admitted_owner_scim_deactivated(key_object)
+        return key_object
+
+    @staticmethod
+    def _raise_503_if_db_unavailable(e: Exception) -> None:
+        """Raise a retryable 503 when ``e`` means the auth database is unreachable, else return so the
+        caller applies its own fail-closed mapping. A DB outage must not masquerade as an auth failure
+        (401) or surface as an opaque 500; the caller retries. Mirrors ``UserAPIKeyAuthExceptionHandler``,
+        which renders a service-unavailable database error as 503 on the standard pipeline.
+
+        Classifies across the ``__cause__``/``__context__`` chain, not just ``e`` itself: ``get_user_object``
+        re-raises every DB failure as a bare ``ValueError``, so a type-based check on the top exception
+        would miss a real outage wrapped inside it."""
+        from litellm.proxy.db.exception_handler import PrismaDBExceptionHandler
+
+        if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(e):
+            raise HTTPException(
+                status_code=503,
+                detail="Service Unavailable, the authentication database is temporarily unreachable. Please retry shortly.",
+            ) from None
+
+    @staticmethod
+    async def _reject_if_admitted_owner_scim_deactivated(key_object: UserAPIKeyAuth) -> None:
+        """Fail closed with a 401 when the key's owning user was deactivated via SCIM.
+
+        The standard pipeline enforces this inline in ``_user_api_key_auth_builder`` rather
+        than in ``common_checks``, so the centralized policy gate does not cover it; without
+        this mirror, IdP offboarding would leave the user's already-minted envelopes live
+        until expiry. A failed user lookup skips the gate (fail-open), matching the builder:
+        this is the one deliberately fail-open check in an otherwise fail-closed arm, so a
+        transient DB outage during this lookup admits the request rather than rejecting it,
+        keeping parity with how the standard pipeline treats the same lookup failure."""
+        if key_object.user_id is None:
+            return
+        from litellm.proxy.auth.auth_checks import get_user_object
+        from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
+
+        try:
+            user_object = await get_user_object(
+                user_id=key_object.user_id,
+                prisma_client=prisma_client,
+                user_api_key_cache=user_api_key_cache,
+                user_id_upsert=False,
+            )
+        except Exception as e:  # noqa: BLE001  # mirror the builder's fail-open user lookup; DB errors are of any type
+            verbose_logger.debug(f"bridge admission: user lookup failed, skipping SCIM gate: {e}")
+            user_object = None
+        if user_object is None or not isinstance(user_object.metadata, dict):
+            return
+        if user_object.metadata.get("scim_active") is False:
+            raise HTTPException(status_code=401, detail="Invalid or expired credential")
+
+    @staticmethod
+    async def _enforce_admitted_live_policy(admitted: UserAPIKeyAuth, request: Request, route: str) -> None:
+        """Run the standard pipeline's authorization checks over the admitted identity.
+
+        Mirrors the ``user_api_key_auth`` wrapper between the builder and its return: clear the
+        request-scoped ``budget_reservation`` on the reloaded identity, run the route gate
+        (``RouteChecks.should_call_route``) to enforce the identity's ``allowed_routes`` and any
+        disabled/admin-only route, then run ``_run_centralized_common_checks`` (the same gate every
+        builder path funnels through) for team-block, project-block, org, and budget. The route gate
+        closes a bypass: a key barred from MCP routes could otherwise mint an envelope at the token
+        endpoint (not itself an MCP route) and replay it against MCP, because the centralized checks
+        treat MCP as an inference route and never re-check ``allowed_routes``.
+
+        Failures surface with the status the standard pipeline would give them, mirroring
+        ``UserAPIKeyAuthExceptionHandler``: a disallowed route is the route gate's own 403, an
+        over-budget identity is a 429, a sub-check that raised its own ``HTTPException``/
+        ``ProxyException`` keeps that status, a transient database outage is a retryable 503, and
+        only a genuinely unresolvable failure (a blocked team/project raises a bare ``Exception``,
+        same as the standard pipeline's fallback) becomes the fail-closed 401. Collapsing every
+        failure to 401 was misleading: it told an over-budget but validly-authenticated caller their
+        credential was invalid, which on a DCR client reads as broken auth and can trigger a
+        pointless re-authorize loop that cannot fix a budget problem, and it masked a DB outage as an
+        auth error."""
+        from litellm.proxy.auth.route_checks import RouteChecks
+
+        admitted.budget_reservation = None
+        try:
+            RouteChecks.should_call_route(route=route, valid_token=admitted, request=request)
+            await _run_centralized_common_checks(
+                user_api_key_auth_obj=admitted,
+                request=request,
+                request_data=await _read_request_body(request=request),
+                route=route,
+            )
+        except (HTTPException, ProxyException):
+            raise
+        except litellm.BudgetExceededError as e:
+            raise HTTPException(status_code=getattr(e, "status_code", 429), detail=str(e)) from None
+        except Exception as e:  # noqa: BLE001  # untyped gate failure: retryable 503 for a DB outage, else fail closed 401
+            MCPRequestHandler._raise_503_if_db_unavailable(e)
+            raise HTTPException(status_code=401, detail="Invalid or expired credential") from None
+
+    @staticmethod
+    def _admitted_key_is_active(key_object: UserAPIKeyAuth) -> bool:
+        """False when the referenced key is blocked or past its expiry, so a revoked key
+        cannot be admitted through its still-unexpired envelope. Mirrors the active-key gate
+        the bridge token endpoint applies at mint time."""
+        if key_object.blocked is True:
+            return False
+        expires = key_object.expires
+        if expires is None:
+            return True
+        expiry = expires if isinstance(expires, datetime) else datetime.fromisoformat(expires)
+        if expiry.tzinfo is None or expiry.tzinfo.utcoffset(expiry) is None:
+            expiry = expiry.replace(tzinfo=timezone.utc)
+        return expiry >= datetime.now(timezone.utc)
+
     @staticmethod
     def _resolve_target_server_names(path: str, mcp_servers_header: Optional[List[str]]) -> List[str]:
         """

litellm/proxy/_experimental/mcp_server/bridge_token_flow.py

@@ -0,0 +1,694 @@
+"""Bridge token flow: litellm identity resolution and the DCR-bridge oauth_delegate mint/refresh pipeline."""
+
+import math
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from typing import TYPE_CHECKING, Literal, Optional
+
+from fastapi import HTTPException, Request
+from fastapi.responses import JSONResponse
+from pydantic import SecretStr
+from typing_extensions import assert_never
+
+from litellm._logging import verbose_logger
+from litellm.proxy._experimental.mcp_server.oauth_utils import TOKEN_NO_CACHE_HEADERS
+from litellm.types.mcp_server.mcp_server_manager import MCPServer
+
+if TYPE_CHECKING:
+    from litellm.proxy._experimental.mcp_server.discoverable_endpoints import _BridgeAuthorizationCode
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (
+        EnvelopeIdentity,
+        EnvelopeKeys,
+        RefreshCredential,
+        UpstreamTokenGrant,
+    )
+    from litellm.proxy._types import UserAPIKeyAuth
+
+
+def _litellm_key_from_request(request: Request) -> Optional[str]:
+    """Return the LiteLLM API key presented on the request, or ``None``.
+
+    Accepts the key from ``x-litellm-api-key`` (what MCP clients such as Claude Desktop/Code
+    send) as well as ``Authorization``; either may carry a bare token or ``Bearer <token>``.
+    ``x-litellm-api-key`` wins when both are present, since ``Authorization`` may instead carry
+    an OAuth/upstream bearer.
+    """
+    for header_value in (
+        request.headers.get("x-litellm-api-key"),
+        request.headers.get("Authorization") or request.headers.get("authorization"),
+    ):
+        if not header_value:
+            continue
+        value = header_value.strip()
+        if value.lower().startswith("bearer "):
+            value = value[7:].strip()
+        if value:
+            return value
+    return None
+
+
+def _key_is_active(key_obj: "UserAPIKeyAuth") -> bool:
+    """``True`` when the presented key is neither blocked nor past its expiry.
+
+    The OAuth token endpoint is unauthenticated, so the presented key is validated here before it is
+    trusted; a revoked or expired key must not mint a bridge envelope or write a stored credential.
+    ``get_key_object`` resolves a row without these checks (the main ``user_api_key_auth`` pipeline
+    enforces them downstream, which this endpoint bypasses), so they are applied here. Deleted keys
+    are already rejected upstream, where ``get_key_object`` raises on a row that no longer exists.
+
+    This is an active-state gate only; it deliberately does not require a ``user_id``. A valid
+    team-scoped or service-account key has no ``user_id`` yet is a legitimate credential, so gating
+    on ``user_id`` presence would wrongly reject it. Callers that need the user (the per-user token
+    store) derive it separately via :func:`_active_key_user_id`.
+
+    Total by design: ``expires`` is typed ``str | datetime``, and an unparseable string would make
+    ``datetime.fromisoformat`` raise. Since the callers run this outside their key-resolution
+    ``try``, an uncaught parse error would surface as a 500 instead of the endpoint's fail-closed
+    behavior, so a malformed expiry is treated as inactive (return ``False``) rather than raising.
+    """
+    if key_obj.blocked is True:
+        return False
+    expires = key_obj.expires
+    if expires is not None:
+        if isinstance(expires, datetime):
+            expiry = expires
+        else:
+            try:
+                expiry = datetime.fromisoformat(expires)
+            except (ValueError, TypeError):
+                return False
+        if expiry.tzinfo is None or expiry.tzinfo.utcoffset(expiry) is None:
+            expiry = expiry.replace(tzinfo=timezone.utc)
+        if expiry < datetime.now(timezone.utc):
+            return False
+    return True
+
+
+def _active_key_user_id(key_obj: "UserAPIKeyAuth") -> str | None:
+    """The active key's ``user_id``, or ``None`` when the key is blocked/expired or simply has no
+    ``user_id`` (a team-scoped or service-account key). Used only by the per-user token store, which
+    needs a user to key the stored credential; the bridge mint uses the key hash and does not."""
+    return key_obj.user_id if _key_is_active(key_obj) else None
+
+
+@dataclass(frozen=True, slots=True)
+class _ResolvedKey:
+    """An active litellm key resolved from the token request: its hash (the value ``get_key_object``
+    and the cache/DB layer key the record by) and the live record."""
+
+    key_hash: str
+    key: "UserAPIKeyAuth"
+
+
+_KeyResolutionFailure = Literal["no_active_key", "unavailable", "unresolvable"]
+"""Why a token request yielded no active litellm key, kept distinct so a caller statuses each truthfully
+instead of blaming the client for a gateway problem:
+- ``no_active_key``: none was presented, or the presented key is unknown / blocked / expired (the
+  caller's request is at fault)
+- ``unavailable``: the auth database was transiently unreachable while resolving (retryable)
+- ``unresolvable``: the gateway cannot resolve identity right now (no DB connection, or an unexpected
+  error) -- a gateway fault, not the caller's
+The classification mirrors admission's ``_reload_admitted_key`` so the mint (ingress) and admission
+(egress) never disagree on the status of the same outage."""
+
+
+async def _resolve_active_litellm_key(request: Request) -> "_ResolvedKey | _KeyResolutionFailure":
+    """Resolve the presented litellm key to an active key record, or say precisely why not.
+
+    Single resolution path the OAuth token endpoint reuses, resolving authoritatively via
+    ``get_key_object`` (cache first, then DB). The failure is a value, not a bare ``None``, so a caller
+    can tell "the client sent no usable credential" (a request error) apart from "the gateway could not
+    check" (an infrastructure error) and status each truthfully; collapsing both to ``None`` is what let
+    a DB outage read as a 400. A resolved key is still gated by ``_key_is_active``, so a blocked or
+    expired key is ``no_active_key`` while a valid team-scoped or service-account key (no ``user_id``)
+    resolves. Classification mirrors admission's ``_reload_admitted_key``: no DB connection is a gateway
+    fault, a ``ProxyException`` / ``HTTPException`` from ``get_key_object`` is an unknown or invalid key,
+    a database-service-unavailable error is a retryable outage, and anything else is an unexpected
+    gateway fault."""
+    token = _litellm_key_from_request(request)
+    if not token:
+        return "no_active_key"
+    from litellm.proxy._types import hash_token  # noqa: PLC0415  # inline import avoids a module-load circular import
+
+    return await _reload_active_key_by_hash(hash_token(token))
+
+
+async def _reload_active_key_by_hash(key_hash: str) -> "_ResolvedKey | _KeyResolutionFailure":
+    """Reload the live key record for ``key_hash`` (cache first, then DB) and gate it on active state,
+    returning the resolved key or a precise failure. Shared by the token request's presented-key
+    resolution (:func:`_resolve_active_litellm_key`, which hashes the presented key) and the refresh
+    path (which already holds the hash sealed in the refresh envelope), so both re-validate identity
+    through one active-key gate and one failure classification. Classification mirrors admission's
+    ``_reload_admitted_key``: no DB connection is a gateway fault, a ``ProxyException`` / ``HTTPException``
+    from ``get_key_object`` is an unknown or invalid key, a database-service-unavailable error is a
+    retryable outage, and anything else is an unexpected gateway fault. A blocked or expired key is
+    ``no_active_key``, so a revoked key can neither mint nor refresh a bridge envelope."""
+    from litellm.proxy._types import (
+        ProxyException,  # noqa: PLC0415  # inline import avoids a module-load circular import
+    )
+    from litellm.proxy.auth.auth_checks import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        get_key_object,
+    )
+    from litellm.proxy.db.exception_handler import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        PrismaDBExceptionHandler,
+    )
+    from litellm.proxy.proxy_server import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        prisma_client,
+        user_api_key_cache,
+    )
+
+    if prisma_client is None:
+        return "unresolvable"
+    try:
+        key_obj = await get_key_object(
+            hashed_token=key_hash,
+            prisma_client=prisma_client,
+            user_api_key_cache=user_api_key_cache,
+        )
+    except (ProxyException, HTTPException):
+        return "no_active_key"
+    except Exception as exc:  # noqa: BLE001  # classify: a DB outage is retryable, anything else is an opaque gateway fault
+        if PrismaDBExceptionHandler.is_database_service_unavailable_error(exc):
+            return "unavailable"
+        verbose_logger.debug(
+            "_reload_active_key_by_hash: unexpected key-resolution error (%s)",
+            type(exc).__name__,
+        )
+        return "unresolvable"
+    if not _key_is_active(key_obj):
+        return "no_active_key"
+    return _ResolvedKey(key_hash=key_hash, key=key_obj)
+
+
+async def _reload_active_user_by_id(user_id: str) -> "_KeyResolutionFailure | None":
+    """Re-validate a live litellm user by id, returning ``None`` when the user is active or a precise
+    failure otherwise. The interactive DCR client authenticates via SSO, so its refresh envelope seals a
+    user subject; renewing it must re-check the user is still live (present and not SCIM-deactivated) so a
+    deactivated user cannot keep refreshing, mirroring how admission re-validates the same user subject on
+    the egress side. No DB connection is a gateway fault (``unresolvable``) and a
+    database-service-unavailable error is a retryable outage (``unavailable``). Everything else fails
+    closed as ``no_active_key`` (the caller maps it to invalid_grant): a ``ProxyException`` /
+    ``HTTPException``, a SCIM-deactivated user, and, unlike the key path, a missing user. ``get_user_object``
+    catches every DB failure and re-raises a bare ``ValueError`` (a deleted user and a real outage look
+    identical, the original error surviving only as ``__context__``), so the outage check walks the cause
+    chain, and a missing user falls through to ``no_active_key`` rather than an opaque gateway fault."""
+    from litellm.proxy._types import (
+        ProxyException,  # noqa: PLC0415  # inline import avoids a module-load circular import
+    )
+    from litellm.proxy.auth.auth_checks import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        get_user_object,
+    )
+    from litellm.proxy.db.exception_handler import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        PrismaDBExceptionHandler,
+    )
+    from litellm.proxy.proxy_server import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        prisma_client,
+        user_api_key_cache,
+    )
+
+    if prisma_client is None:
+        return "unresolvable"
+    try:
+        user_object = await get_user_object(
+            user_id=user_id,
+            prisma_client=prisma_client,
+            user_api_key_cache=user_api_key_cache,
+            user_id_upsert=False,
+        )
+    except (ProxyException, HTTPException):
+        return "no_active_key"
+    except Exception as exc:  # noqa: BLE001  # a DB outage is retryable; a missing user (get_user_object's wrapped ValueError) or any other resolution failure fails closed as no_active_key, never a 500
+        if PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain(exc):
+            return "unavailable"
+        verbose_logger.debug("_reload_active_user_by_id: user-resolution error (%s)", type(exc).__name__)
+        return "no_active_key"
+    if user_object is None:
+        return "no_active_key"
+    if isinstance(user_object.metadata, dict) and user_object.metadata.get("scim_active") is False:
+        return "no_active_key"
+    return None
+
+
+async def _key_owner_scim_deactivated(key: "UserAPIKeyAuth") -> bool:
+    """True only when the key's owning user was explicitly SCIM-deactivated, so a refresh revokes an
+    offboarded owner's key exactly as admission does via ``_reject_if_admitted_owner_scim_deactivated``.
+    A key with no owner, a missing owner record, or a failed lookup fails OPEN (returns ``False``),
+    matching admission and the standard builder: a key may outlive its owner record, and a transient DB
+    blip must not revoke a live key. Only an explicit ``scim_active`` of ``False`` gates renewal."""
+    if key.user_id is None:
+        return False
+    from litellm.proxy.auth.auth_checks import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        get_user_object,
+    )
+    from litellm.proxy.proxy_server import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        prisma_client,
+        user_api_key_cache,
+    )
+
+    if prisma_client is None:
+        return False
+    try:
+        owner = await get_user_object(
+            user_id=key.user_id,
+            prisma_client=prisma_client,
+            user_api_key_cache=user_api_key_cache,
+            user_id_upsert=False,
+        )
+    except Exception as exc:  # noqa: BLE001  # fail open: a missing owner (get_user_object's wrapped ValueError) or a DB blip must not revoke a live key
+        verbose_logger.debug("refresh: key-owner SCIM lookup failed, not revoking (%s)", type(exc).__name__)
+        return False
+    return owner is not None and isinstance(owner.metadata, dict) and owner.metadata.get("scim_active") is False
+
+
+async def _revalidate_active_subject(identity: "EnvelopeIdentity") -> "_KeyResolutionFailure | None":
+    """Re-validate that the subject sealed in a refresh envelope is still live, dispatching on its type:
+    a key_hash reloads the virtual key, a user_id reloads the user. Returns ``None`` when the subject is
+    active or a precise failure otherwise, so revocation gates renewal for either identity source the same
+    way admission gates the egress: a blocked or expired key, a SCIM-deactivated key owner (mirroring
+    admission's owner check, so an offboarded user cannot keep renewing a still-active key), and a
+    deactivated or deleted user all fail closed to ``no_active_key``."""
+    match identity.subject_type:
+        case "key_hash":
+            reloaded = await _reload_active_key_by_hash(identity.subject)
+            if not isinstance(reloaded, _ResolvedKey):
+                return reloaded
+            if await _key_owner_scim_deactivated(reloaded.key):
+                return "no_active_key"
+            return None
+        case "user_id":
+            return await _reload_active_user_by_id(identity.subject)
+        case _:
+            assert_never(identity.subject_type)
+
+
+async def _extract_user_id_from_request(request: Request) -> str | None:
+    """The litellm ``user_id`` for the token request, so a per-user token is stored under the same
+    identity the egress later reads it by. Storage is best-effort, so every non-resolved outcome
+    (including a transient DB outage) collapses to ``None`` here and the caller simply skips the store;
+    the bridge mint, which must status those outcomes differently, consumes
+    :func:`_resolve_active_litellm_key` directly."""
+    resolved = await _resolve_active_litellm_key(request)
+    if not isinstance(resolved, _ResolvedKey):
+        return None
+    return _active_key_user_id(resolved.key)
+
+
+_UpstreamGrantRejection = Literal["no_access_token", "expired_lifetime"]
+"""Why an upstream token response cannot back a bridge envelope:
+- ``no_access_token``: the response carries no usable ``access_token``
+- ``expired_lifetime``: the response reports a parseable, non-positive ``expires_in``, i.e. an upstream
+  token that is already dead, so sealing it would forward a bearer the edge cannot use
+An absent or unparseable ``expires_in`` is NOT a rejection; the lifetime is merely unknown and the
+envelope caps it, the by-design behaviour for an upstream that omits the field."""
+
+
+def _classify_upstream_lifetime(raw_expires_in: object) -> "int | Literal['unspecified', 'expired']":
+    """Classify an upstream ``expires_in`` into a positive number of seconds, ``"unspecified"`` (absent
+    or unparseable, so the envelope caps it), or ``"expired"`` (a non-positive value the upstream reports
+    as already elapsed). Telling "we do not know the lifetime" apart from "the upstream says it is
+    already dead" is what stops an explicitly-expired token from silently receiving the envelope's 1h
+    cap. The expired decision is made on the parsed numeric value, not on ``int(...)`` of it, so a
+    positive sub-second lifetime in ``(0, 1)`` is not truncated to ``0`` and misread as elapsed; the
+    envelope works in whole seconds, so such a lifetime clamps up to its 1s floor. ``bool`` is excluded
+    (an ``int`` subclass but never a real lifetime), and the conversions can raise on ``NaN`` /
+    ``Infinity`` / oversized input, which reads as unparseable rather than surfacing as a 500."""
+    if raw_expires_in is None or isinstance(raw_expires_in, bool) or not isinstance(raw_expires_in, (int, float, str)):
+        return "unspecified"
+    try:
+        numeric = float(raw_expires_in)
+        seconds = int(numeric)
+    except (ValueError, TypeError, OverflowError):
+        return "unspecified"
+    if numeric <= 0:
+        return "expired"
+    return max(1, seconds)
+
+
+def _bridge_grant_from_token_response(token_response: object) -> "UpstreamTokenGrant | _UpstreamGrantRejection":
+    """Validate an upstream OAuth token response into a typed grant, or say why it cannot back an
+    envelope. Each field is isinstance-checked so nothing untyped from ``response.json()`` reaches the
+    grant. ``expires_in`` is read three ways (see :func:`_classify_upstream_lifetime`): an unknown
+    lifetime leaves the grant ``expires_in`` ``None`` for the envelope to cap, a positive value is
+    honoured, and an explicit already-elapsed value is a rejection rather than a silent fall-through to
+    the cap."""
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        UpstreamTokenGrant,
+    )
+
+    if not isinstance(token_response, dict):
+        return "no_access_token"
+    access = token_response.get("access_token")
+    if not isinstance(access, str) or not access:
+        return "no_access_token"
+    lifetime = _classify_upstream_lifetime(token_response.get("expires_in"))
+    if lifetime == "expired":
+        return "expired_lifetime"
+    token_type = token_response.get("token_type")
+    scope = token_response.get("scope")
+    return UpstreamTokenGrant(
+        access_token=SecretStr(access),
+        token_type=token_type if isinstance(token_type, str) and token_type else "Bearer",
+        # The upstream refresh_token is deliberately NOT sealed: the edge never consumes it (it forwards
+        # only token_type + access_token), so it would be dead weight embedding a long-lived upstream
+        # credential in the client-held bearer, and it enlarges the envelope. Refresh support is a
+        # follow-up (a dedicated refresh-envelope); the client re-runs authorization_code at the cap.
+        refresh_token=None,
+        scope=scope if isinstance(scope, str) and scope else None,
+        expires_in=lifetime if isinstance(lifetime, int) else None,
+    )
+
+
+# ---------------------------------------------------------------------------
+# DCR-bridge oauth_delegate mint: a three-phase pipeline whose failures are values.
+#
+#   prepare  (before the upstream exchange) -> validate every precondition and resolve identity+keys
+#   exchange (the single-use upstream code is consumed here, in exchange_token_with_server)
+#   finish   (after the exchange)           -> seal the upstream grant into the client-held envelope
+#
+# Every precondition lives in ``prepare``, which runs BEFORE the exchange, so no failure can burn the
+# single-use code or rotate a refresh token, for either grant type -- that whole class of bug is gone
+# by construction rather than guarded case by case. Failures are values mapped to an OAuth-shaped
+# response in one place (``_bridge_mint_error_response``), so status codes and the RFC 6749 §5.2 body
+# shape are uniform. Adding a failure mode is a new literal plus a match arm the type checker forces.
+# ---------------------------------------------------------------------------
+
+_BridgeMintError = Literal[
+    "no_identity",
+    "invalid_refresh",
+    "identity_unavailable",
+    "identity_unresolvable",
+    "not_configured",
+    "no_upstream_token",
+    "upstream_token_expired",
+    "too_large",
+]
+
+
+@dataclass(frozen=True, slots=True)
+class _BridgeMintReady:
+    """Everything the seal needs, resolved once before the exchange: the identity to bind the envelope
+    to and the master-key-derived envelope keys. The identity is a key_hash subject for the scripted
+    two-header client (resolved from the litellm key it presents) or a user_id subject for the
+    interactive SSO client (the user recovered from the gateway authorization code), so one phase-3 seal
+    serves both. Resolving identity here means ``_finish_bridge_mint`` has no preconditions left to
+    fail."""
+
+    identity: "EnvelopeIdentity"
+    keys: "EnvelopeKeys"
+
+
+def _bridge_mint_error_response(error: _BridgeMintError) -> JSONResponse:
+    """Map a bridge-mint failure value to its token-endpoint response: one place, RFC 6749 §5.2 shape
+    (top-level ``error``, no-store headers) for every case, with a status truthful about where the
+    failure is. The caller's request is 400, a transient gateway outage is 503, a gateway
+    misconfiguration is 500, and an upstream problem is 502. The identity-resolution statuses match how
+    admission statuses the same conditions on the egress side, so mint and admit never disagree under
+    one outage."""
+    match error:
+        case "no_identity":
+            status, code, desc = (
+                400,
+                "invalid_request",
+                "this server issues a gateway-bound credential; complete the interactive sign-in, or "
+                "send a litellm credential (x-litellm-api-key or Authorization) on the token request",
+            )
+        case "invalid_refresh":
+            status, code, desc = (
+                400,
+                "invalid_grant",
+                "the refresh credential is not a valid, live refresh envelope for this server; "
+                "re-run authorization_code to obtain a new one",
+            )
+        case "identity_unavailable":
+            status, code, desc = (
+                503,
+                "temporarily_unavailable",
+                "the authentication database is temporarily unreachable; retry shortly",
+            )
+        case "identity_unresolvable":
+            status, code, desc = (
+                500,
+                "server_error",
+                "the gateway could not resolve the litellm identity for this request",
+            )
+        case "not_configured":
+            status, code, desc = (
+                500,
+                "server_error",
+                "the gateway is not configured to mint a gateway-bound credential (master_key is not set)",
+            )
+        case "no_upstream_token":
+            status, code, desc = (
+                502,
+                "server_error",
+                "the upstream token response has no usable access_token",
+            )
+        case "upstream_token_expired":
+            status, code, desc = (
+                502,
+                "server_error",
+                "the upstream token response reports an already-expired lifetime",
+            )
+        case "too_large":
+            status, code, desc = (
+                502,
+                "server_error",
+                "the upstream token is too large to seal into a gateway-bound credential",
+            )
+        case _:
+            assert_never(error)
+    return JSONResponse(
+        status_code=status, content={"error": code, "error_description": desc}, headers=TOKEN_NO_CACHE_HEADERS
+    )
+
+
+def _key_resolution_failure_to_mint_error(failure: _KeyResolutionFailure) -> _BridgeMintError:
+    """Lift an identity-resolution failure into the mint taxonomy, preserving origin so the status stays
+    truthful: the caller's missing credential is 400, a transient DB outage is 503, and a gateway that
+    cannot resolve identity is 500."""
+    match failure:
+        case "no_active_key":
+            return "no_identity"
+        case "unavailable":
+            return "identity_unavailable"
+        case "unresolvable":
+            return "identity_unresolvable"
+        case _:
+            assert_never(failure)
+
+
+def _upstream_rejection_to_mint_error(rejection: _UpstreamGrantRejection) -> _BridgeMintError:
+    """Lift an upstream-response rejection into the mint taxonomy; both are upstream faults (502)."""
+    match rejection:
+        case "no_access_token":
+            return "no_upstream_token"
+        case "expired_lifetime":
+            return "upstream_token_expired"
+        case _:
+            assert_never(rejection)
+
+
+async def _prepare_bridge_mint(
+    request: Request,
+    mcp_server: MCPServer,
+    bridge_identity: "_BridgeAuthorizationCode | None" = None,
+) -> "_BridgeMintReady | _BridgeMintError":
+    """Phase 1 for the authorization_code grant, BEFORE the upstream exchange: confirm the gateway can
+    mint (master_key set), resolve the litellm identity, and derive the envelope keys. Returns a ready
+    context or a precise failure value. Running before the exchange is what makes every failure here fail
+    closed without consuming the single-use code.
+
+    Two identity sources, one envelope. The interactive DCR client authenticates via SSO at the bridged
+    authorize, so its identity arrives as ``bridge_identity`` (the user recovered from the gateway
+    authorization code) and mints a user subject. The scripted two-header client presents a litellm key
+    on the token request instead, so its identity is the active key's hash and mints a key_hash subject.
+    A missing or invalid presented key keeps its resolution origin so the mapper statuses it truthfully;
+    neither source present is ``no_identity``. The refresh_token grant has its own phase-1
+    (:func:`_prepare_bridge_refresh`), which recovers identity from the presented refresh envelope."""
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        envelope_keys_from_master_key,
+    )
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        key_hash_identity,
+        user_identity,
+    )
+    from litellm.proxy.proxy_server import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        master_key,
+    )
+
+    if not master_key:
+        return "not_configured"
+    keys = envelope_keys_from_master_key(master_key)
+    if bridge_identity is not None:
+        identity = user_identity(server_id=mcp_server.server_id, user_id=bridge_identity.litellm_user_id)
+        return _BridgeMintReady(identity=identity, keys=keys)
+    resolved = await _resolve_active_litellm_key(request)
+    if not isinstance(resolved, _ResolvedKey):
+        return _key_resolution_failure_to_mint_error(resolved)
+    identity = key_hash_identity(server_id=mcp_server.server_id, key_hash=resolved.key_hash)
+    return _BridgeMintReady(identity=identity, keys=keys)
+
+
+@dataclass(frozen=True, slots=True)
+class _BridgeRefreshReady:
+    """A validated refresh request: the identity+keys to mint the renewed pair under, the upstream refresh
+    token (unwrapped from the client's refresh envelope) to exchange with the upstream IdP, and the scope
+    sealed alongside it at mint. The upstream refresh token is a ``SecretStr`` like every other credential
+    in this layer, so a repr or a traceback that captures this value never exposes the raw upstream refresh
+    token in plaintext. ``upstream_scope`` carries the originally-granted scope so the renewal re-requests
+    it when the client (a DCR/MCP client that typically omits scope on refresh) sends none, keeping the
+    renewed token's scope stable against an upstream that would otherwise narrow or drop it."""
+
+    ready: "_BridgeMintReady"
+    upstream_refresh_token: SecretStr
+    upstream_scope: str | None = None
+
+
+def _refresh_key_failure_to_mint_error(failure: _KeyResolutionFailure) -> _BridgeMintError:
+    """Lift an identity-resolution failure on the refresh path into the mint taxonomy. Unlike the mint
+    path, a resolved-but-inactive (or unknown) key is ``invalid_grant`` rather than ``invalid_request``:
+    the client did present an identity (sealed in the refresh envelope), but it is no longer live, so the
+    refresh is invalid and the client must re-authenticate. A transient outage is still 503 and a gateway
+    fault still 500, matching the mint path and admission."""
+    match failure:
+        case "no_active_key":
+            return "invalid_refresh"
+        case "unavailable":
+            return "identity_unavailable"
+        case "unresolvable":
+            return "identity_unresolvable"
+        case _:
+            assert_never(failure)
+
+
+async def _prepare_bridge_refresh(
+    mcp_server: MCPServer, refresh_value: str | None
+) -> "_BridgeRefreshReady | _BridgeMintError":
+    """Phase 1 for the refresh_token grant, BEFORE the upstream exchange: open the client's refresh
+    envelope, re-validate the sealed litellm identity so a revoked key cannot keep refreshing, and
+    recover the upstream refresh token to exchange. Identity comes entirely from the sealed envelope, not
+    the HTTP request, so the request object is not needed here. The client presents a refresh envelope,
+    never a raw upstream refresh token, so a missing value, a non-envelope, an unopenable envelope, or one
+    minted for another server is ``invalid_grant``. Running before the exchange means a rejected refresh
+    never consumes or rotates the upstream refresh token."""
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        BridgeRefreshOpened,
+        envelope_keys_from_master_key,
+        open_bridge_refresh_envelope,
+    )
+    from litellm.proxy.proxy_server import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        master_key,
+    )
+
+    if not master_key:
+        return "not_configured"
+    if not refresh_value:
+        return "invalid_refresh"
+    keys = envelope_keys_from_master_key(master_key)
+    opened = open_bridge_refresh_envelope(refresh_value, keys, datetime.now(timezone.utc), mcp_server.server_id)
+    if not isinstance(opened, BridgeRefreshOpened):
+        return "invalid_refresh"
+    failure = await _revalidate_active_subject(opened.identity)
+    if failure is not None:
+        return _refresh_key_failure_to_mint_error(failure)
+    return _BridgeRefreshReady(
+        ready=_BridgeMintReady(identity=opened.identity, keys=keys),
+        upstream_refresh_token=opened.refresh.refresh_token,
+        upstream_scope=opened.refresh.scope,
+    )
+
+
+def _finish_bridge_mint(
+    ready: "_BridgeMintReady", mcp_server: MCPServer, token_response: object, now: datetime
+) -> "JSONResponse | _BridgeMintError":
+    """Phase 3, AFTER the upstream exchange: seal the upstream grant into the client-held access envelope
+    using the pre-resolved identity and keys, and, when the upstream returned a refresh token, seal a
+    long-lived refresh envelope alongside it so the client can renew without re-authenticating. Shared by
+    the authorization_code and refresh_token paths, so a renewal that the upstream rotates re-issues a
+    fresh refresh envelope. The only hard failures here are properties of the upstream access token (no
+    usable token, an already-expired lifetime, or a token too large to seal); a refresh token that cannot
+    be sealed degrades to an access-only response rather than failing the whole exchange."""
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        build_bridge_token_response,
+    )
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        SealedEnvelope,
+        UpstreamTokenGrant,
+    )
+
+    grant = _bridge_grant_from_token_response(token_response)
+    if not isinstance(grant, UpstreamTokenGrant):
+        return _upstream_rejection_to_mint_error(grant)
+    sealed = build_bridge_token_response(ready.identity, grant, ready.keys, now)
+    if not isinstance(sealed, SealedEnvelope):
+        return "too_large"
+    # Report expires_in from the JWT's own second-truncated exp, rounding the elapsed portion up, so the
+    # client is never told the bearer lives past the point admission (which uses that exp) rejects it.
+    expires_in = max(0, int(sealed.expires_at.timestamp()) - math.ceil(now.timestamp()))
+    refresh_envelope = _mint_refresh_envelope_value(ready.identity, token_response, ready.keys, now, mcp_server)
+    body = {
+        "access_token": sealed.token.get_secret_value(),
+        "token_type": "Bearer",
+        "expires_in": expires_in,
+        # A refresh envelope rides along only when the upstream returned a refresh token to seal; when it
+        # rotates on renewal, the client receives the new one and the old envelope's upstream token dies.
+        **({"refresh_token": refresh_envelope} if refresh_envelope is not None else {}),
+    }
+    return JSONResponse(body, headers=TOKEN_NO_CACHE_HEADERS)
+
+
+def _upstream_refresh_credential(token_response: object) -> "RefreshCredential | None":
+    """Extract the upstream refresh grant from a token response, or ``None`` when there is none to seal.
+    Each field is isinstance-checked so nothing untyped reaches the refresh envelope; ``refresh_expires_in``
+    (the refresh token's own lifetime, when the upstream reports it) is classified like ``expires_in`` and
+    bounds the refresh envelope's TTL. An upstream that reports the refresh token itself as already elapsed
+    (``refresh_expires_in`` non-positive) yields ``None`` rather than a refresh envelope: sealing a dead
+    token would hand the client a full-TTL-capped envelope the IdP will reject, so the exchange degrades to
+    an access-only response (the client re-authenticates at access expiry), mirroring how
+    :func:`_bridge_grant_from_token_response` refuses an already-elapsed access token instead of capping it."""
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        RefreshCredential,
+    )
+
+    if not isinstance(token_response, dict):
+        return None
+    refresh = token_response.get("refresh_token")
+    if not isinstance(refresh, str) or not refresh:
+        return None
+    lifetime = _classify_upstream_lifetime(token_response.get("refresh_expires_in"))
+    if lifetime == "expired":
+        return None
+    scope = token_response.get("scope")
+    return RefreshCredential(
+        refresh_token=SecretStr(refresh),
+        scope=scope if isinstance(scope, str) and scope else None,
+        expires_in=lifetime if isinstance(lifetime, int) else None,
+    )
+
+
+def _mint_refresh_envelope_value(
+    identity: "EnvelopeIdentity", token_response: object, keys: "EnvelopeKeys", now: datetime, mcp_server: MCPServer
+) -> str | None:
+    """Seal the upstream refresh grant (if any) into a refresh envelope and return its bearer string, or
+    ``None`` when the upstream returned no refresh token or the refresh token is too large to seal. A
+    too-large refresh token degrades to an access-only response (logged) rather than failing an exchange
+    that already succeeded upstream: the client simply re-authenticates when the access envelope expires."""
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.bridge_credentials import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        build_bridge_refresh_token_response,
+    )
+    from litellm.proxy._experimental.mcp_server.outbound_credentials.envelope import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+        SealedEnvelope,
+    )
+
+    refresh_credential = _upstream_refresh_credential(token_response)
+    if refresh_credential is None:
+        return None
+    sealed = build_bridge_refresh_token_response(identity, refresh_credential, keys, now)
+    if isinstance(sealed, SealedEnvelope):
+        return sealed.token.get_secret_value()
+    verbose_logger.warning(
+        "bridge mint: the upstream refresh token is too large to seal into a refresh envelope for "
+        "server=%s; issuing an access-only response, so the client re-authenticates at access expiry",
+        mcp_server.server_id,
+    )
+    return None

litellm/proxy/_experimental/mcp_server/db.py

@@ -52,6 +52,7 @@ _AUTH_FLOW_SCOPED_FIELDS: frozenset = frozenset(
         "token_url",
         "registration_url",
         "oauth2_flow",
+        "dcr_bridge",
         "token_exchange_endpoint",
         "audience",
         "subject_token_type",
@@ -75,6 +76,34 @@ _TOKEN_EXCHANGE_COLUMN_FIELDS: frozenset = frozenset(
     }
 )
 
+# The client-forwarded token modes share one stored-credential shape: the admin-declared upstream
+# OAuth app (client_id/client_secret) plus the same authorize relay, and neither mints anything the
+# gateway keeps. So a switch WITHIN this class must preserve the stored app, unlike a cross-class
+# switch (e.g. an oauth2 row whose client may be DCR-minted and is not reusable elsewhere).
+_CLIENT_FORWARDED_AUTH_TYPES: frozenset = frozenset({"true_passthrough", "oauth_delegate"})
+
+# Minted token material that must never survive a client rotation on a persisted row.
+_MINTED_TOKEN_CREDENTIAL_FIELDS: frozenset = frozenset({"access_token", "refresh_token", "expires_in"})
+
+
+def _credential_auth_class(auth_type: Optional[str]) -> Optional[str]:
+    """Collapse the client-forwarded modes to one credential class; every other auth_type is its own
+    class. Used so credential handling keys off whether the stored-credential shape actually changed,
+    not off a raw auth_type inequality that treats true_passthrough<->oauth_delegate as a full reset."""
+    if auth_type in _CLIENT_FORWARDED_AUTH_TYPES:
+        return "client_forwarded"
+    return auth_type
+
+
+def _drop_stale_minted_on_client_rotation(merged: Dict[str, Any], new_creds: Dict[str, Any]) -> Dict[str, Any]:
+    """When the update rotates the client, drop stale minted token keys it did not itself set, so an old
+    app's access/refresh token never rides forward under the new client. A no-op when no client key changed."""
+    if "client_id" not in new_creds and "client_secret" not in new_creds:
+        return merged
+    return {
+        key: value for key, value in merged.items() if key not in _MINTED_TOKEN_CREDENTIAL_FIELDS or key in new_creds
+    }
+
 
 def _is_global_env_var_scope(scope: Any) -> bool:
     """``scope="user"`` entries are placeholders the user fills in; everything
@@ -678,7 +707,9 @@ async def update_mcp_server(
         existing = await MCPServerRepository(prisma_client).table.find_unique(where={"server_id": data.server_id})
 
     auth_type_changed = bool(
-        data.auth_type and existing and existing.auth_type is not None and existing.auth_type != data.auth_type
+        data.auth_type
+        and existing
+        and _credential_auth_class(existing.auth_type) != _credential_auth_class(data.auth_type)
     )
 
     # Clear stale credentials when auth_type changes but no new credentials provided
@@ -711,11 +742,12 @@ async def update_mcp_server(
     # would wipe encrypted secrets that the UI cannot display back.
     if "credentials" in data_dict and data_dict["credentials"] is not None:
         if existing and existing.credentials:
-            # Only merge when auth_type is unchanged. Switching auth types
-            # (e.g. oauth2 → api_key) should replace credentials entirely
-            # to avoid stale secrets from the previous auth type lingering.
-            auth_type_unchanged = data.auth_type is None or data.auth_type == existing.auth_type
-            if auth_type_unchanged:
+            # Only merge when the credential CLASS is unchanged. A cross-class switch
+            # (e.g. oauth2 → api_key, or oauth2 → true_passthrough) replaces credentials
+            # entirely to avoid stale secrets from the previous class lingering; a switch
+            # within the client-forwarded class (true_passthrough ↔ oauth_delegate) keeps
+            # the same declared app and so must merge, not replace.
+            if not auth_type_changed:
                 existing_creds = (
                     json.loads(existing.credentials)
                     if isinstance(existing.credentials, str)
@@ -726,8 +758,9 @@ async def update_mcp_server(
                     if isinstance(data_dict["credentials"], str)
                     else dict(data_dict["credentials"])
                 )
-                # New values override existing; existing keys not in update are preserved
-                merged = {**existing_creds, **new_creds}
+                # New values override existing; existing keys not in update are preserved. A client
+                # rotation additionally drops the previous app's stale minted token keys.
+                merged = _drop_stale_minted_on_client_rotation({**existing_creds, **new_creds}, new_creds)
                 # Migrate-on-write for legacy rows: token-exchange settings the
                 # old blob shape carried move to their dedicated columns (unless
                 # the caller set the column this update, or the row already has
@@ -746,6 +779,14 @@ async def update_mcp_server(
     # Add audit fields
     data_dict["updated_by"] = touched_by
 
+    # prisma-python rejects a raw ``None`` for a ``Json?`` field ("value is required but not set"); the
+    # clear paths above use ``None`` as the merge-skip sentinel, so translate it here to ``Json(None)``,
+    # which writes SQL null and reads back as ``None``. Done at the edge so the merge guards stay simple.
+    if "credentials" in data_dict and data_dict["credentials"] is None:
+        from prisma import Json  # noqa: PLC0415  # local import: prisma may be ungenerated at module load in some tools
+
+        data_dict["credentials"] = Json(None)
+
     updated_mcp_server = await MCPServerRepository(prisma_client).table.update(
         where={"server_id": data.server_id},
         data=data_dict,  # type: ignore

litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py

@@ -10,7 +10,7 @@ from urllib.parse import parse_qsl, urlencode, urlparse, urlunparse
 import httpx
 from fastapi import APIRouter, Form, HTTPException, Request
 from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse, Response
-from pydantic import BaseModel, ValidationError
+from pydantic import BaseModel, ConfigDict, Field, SecretStr, ValidationError
 
 from litellm._logging import verbose_logger
 from litellm.llms.custom_httpx.http_handler import (
@@ -21,6 +21,24 @@ from litellm.proxy._experimental.mcp_server.auth.token_endpoint_auth import (
     TokenEndpointAuthConfigError,
     build_token_endpoint_client_auth,
 )
+from litellm.proxy._experimental.mcp_server.bridge_token_flow import (
+    _bridge_mint_error_response,
+    _BridgeMintReady,
+    _BridgeRefreshReady,
+    _extract_user_id_from_request,
+    _finish_bridge_mint,
+    _prepare_bridge_mint,
+    _prepare_bridge_refresh,
+)
+from litellm.proxy._experimental.mcp_server.faults import (
+    CallerRejected,
+    CredentialSource,
+    UpstreamProtocolFault,
+    classify_upstream_dcr_rejection,
+    classify_upstream_token_rejection,
+    dcr_fault_detail,
+    render_token_fault,
+)
 from litellm.proxy._experimental.mcp_server.oauth_utils import (
     TOKEN_NO_CACHE_HEADERS,
     get_request_base_url,
@@ -37,7 +55,7 @@ from litellm.types.mcp import MCPAuth, MCPCredentials
 from litellm.types.mcp_server.mcp_server_manager import MCPServer
 
 if TYPE_CHECKING:
-    from litellm.proxy._types import LiteLLM_MCPServerTable, UserAPIKeyAuth
+    from litellm.proxy._types import LiteLLM_MCPServerTable
 
 # TTL cache for upstream OAuth metadata fetched from pass-through MCP servers.
 # Keeps us from hammering the upstream IdP on each discovery request.
@@ -91,6 +109,8 @@ def encode_state_with_base_url(
     code_challenge: Optional[str] = None,
     code_challenge_method: Optional[str] = None,
     client_redirect_uri: Optional[str] = None,
+    litellm_user_id: str | None = None,
+    mcp_server_id: str | None = None,
 ) -> str:
     """
     Encode the base_url, original state, and PKCE parameters using encryption.
@@ -101,6 +121,11 @@ def encode_state_with_base_url(
         code_challenge: PKCE code challenge from client
         code_challenge_method: PKCE code challenge method from client
         client_redirect_uri: Original redirect_uri from client
+        litellm_user_id: The SSO-authenticated litellm user captured at the bridge authorize
+            (interactive dcr_bridge oauth_delegate only); the callback seals it into the gateway
+            authorization code so the token mint can bind the envelope to this user
+        mcp_server_id: The bridge server the interactive flow targets, sealed alongside
+            litellm_user_id so the gateway code cannot be replayed against another server
 
     Returns:
         An encrypted string that encodes all values
@@ -111,6 +136,8 @@ def encode_state_with_base_url(
         "code_challenge": code_challenge,
         "code_challenge_method": code_challenge_method,
         "client_redirect_uri": client_redirect_uri,
+        "litellm_user_id": litellm_user_id,
+        "mcp_server_id": mcp_server_id,
     }
     state_json = json.dumps(state_data, sort_keys=True)
     encrypted_state = encrypt_value_helper(state_json)
@@ -138,6 +165,68 @@ def decode_state_hash(encrypted_state: str) -> dict:
     return state_data
 
 
+_BRIDGE_AUTH_CODE_PREFIX = "llm_bcode_"
+
+
+class _BridgeAuthorizationCode(BaseModel):
+    """The identity and upstream code the gateway seals into the authorization code it hands a DCR
+    client for an interactive dcr_bridge oauth_delegate sign-in, recovered at the token endpoint."""
+
+    model_config = ConfigDict(frozen=True)
+    upstream_code: str = Field(min_length=1)
+    litellm_user_id: str = Field(min_length=1)
+    mcp_server_id: str = Field(min_length=1)
+
+
+def is_bridge_authorization_code(code: str) -> bool:
+    """Cheap prefix check that ``code`` is a gateway-sealed bridge authorization code rather than a
+    raw upstream code, so the token endpoint can route without decrypting."""
+    return code.startswith(_BRIDGE_AUTH_CODE_PREFIX)
+
+
+def seal_bridge_authorization_code(upstream_code: str, litellm_user_id: str, mcp_server_id: str) -> str:
+    """Seal the upstream authorization code and the SSO-captured litellm user into a gateway
+    authorization code. The DCR client only echoes this opaque value back at the token endpoint; the
+    gateway decrypts it there to recover the user (to bind the envelope) and the upstream code (to
+    exchange with the upstream), so a litellm identity captured in the browser at authorize survives
+    to the back-channel token call with nothing stored server-side. Encrypted with the repo's
+    authenticated symmetric helper (the same family the OAuth state uses), so the client can neither
+    read nor forge it."""
+    payload = json.dumps(
+        {"upstream_code": upstream_code, "litellm_user_id": litellm_user_id, "mcp_server_id": mcp_server_id},
+        sort_keys=True,
+    )
+    return _BRIDGE_AUTH_CODE_PREFIX + encrypt_value_helper(payload)
+
+
+def open_bridge_authorization_code(code: str) -> _BridgeAuthorizationCode | None:
+    """Recover the sealed identity and upstream code, or ``None`` when ``code`` is not a gateway
+    bridge code or does not decrypt / validate. Total over hostile input: a raw upstream code (the
+    scripted two-header path) returns ``None`` and the caller falls through to the existing
+    behavior."""
+    if not is_bridge_authorization_code(code):
+        return None
+    decrypted = decrypt_value_helper(
+        code[len(_BRIDGE_AUTH_CODE_PREFIX) :], "bridge_authorization_code", return_original_value=False
+    )
+    if not isinstance(decrypted, str):
+        return None
+    try:
+        return _BridgeAuthorizationCode.model_validate_json(decrypted)
+    except ValidationError:
+        return None
+
+
+def _redirect_to_litellm_login(request: Request) -> RedirectResponse:
+    """Send an unauthenticated browser through litellm login before the interactive bridge authorize
+    can capture its identity. The bridge oauth_delegate flow seals the SSO user into the gateway code,
+    so a session is required; without one there is nothing to bind. After login the user re-initiates
+    the connection, which then finds the session cookie (the seamless return-to round-trip, which is
+    origin-validated against the control-plane URL, is a follow-up)."""
+    base_url = get_request_base_url(request)
+    return RedirectResponse(f"{base_url}/sso/key/generate")
+
+
 # LIT-4197: some upstream authorization servers reject an over-long ``state``
 # (the encrypted OAuth session blob routinely exceeds their limit). The upstream
 # only needs an opaque value it echoes back on ``/callback``, so we forward a
@@ -304,90 +393,6 @@ def _validate_token_response(
             )
 
 
-def _litellm_key_from_request(request: Request) -> Optional[str]:
-    """Return the LiteLLM API key presented on the request, or ``None``.
-
-    Accepts the key from ``x-litellm-api-key`` (what MCP clients such as Claude Desktop/Code
-    send) as well as ``Authorization``; either may carry a bare token or ``Bearer <token>``.
-    ``x-litellm-api-key`` wins when both are present, since ``Authorization`` may instead carry
-    an OAuth/upstream bearer.
-    """
-    for header_value in (
-        request.headers.get("x-litellm-api-key"),
-        request.headers.get("Authorization") or request.headers.get("authorization"),
-    ):
-        if not header_value:
-            continue
-        value = header_value.strip()
-        if value.lower().startswith("bearer "):
-            value = value[7:].strip()
-        if value:
-            return value
-    return None
-
-
-def _active_key_user_id(key_obj: "UserAPIKeyAuth") -> Optional[str]:
-    """The key's ``user_id``, or ``None`` if the key is blocked or expired.
-
-    The OAuth token endpoint is unauthenticated, so the presented key is validated here before its
-    identity is trusted to key a stored credential; a revoked or expired key must not be able to
-    write or overwrite the per-user OAuth token. ``get_key_object`` resolves a row without these
-    checks (the main ``user_api_key_auth`` pipeline enforces them downstream, which this endpoint
-    bypasses), so they are applied here. Deleted keys are already rejected upstream, where
-    ``get_key_object`` raises on a row that no longer exists.
-    """
-    if key_obj.blocked is True:
-        return None
-    expires = key_obj.expires
-    if expires is not None:
-        expiry = expires if isinstance(expires, datetime) else datetime.fromisoformat(expires)
-        if expiry.tzinfo is None or expiry.tzinfo.utcoffset(expiry) is None:
-            expiry = expiry.replace(tzinfo=timezone.utc)
-        if expiry < datetime.now(timezone.utc):
-            return None
-    return key_obj.user_id
-
-
-async def _extract_user_id_from_request(request: Request) -> Optional[str]:
-    """Resolve the LiteLLM ``user_id`` at the OAuth token endpoint so a per-user token is stored
-    under the same identity the egress later reads it by (``user_api_key_auth.user_id``).
-
-    Resolves authoritatively via ``get_key_object`` (cache first, then DB) instead of a raw cache
-    peek. On a multi-replica gateway the token-exchange request can land on a worker whose in-memory
-    cache never saw the key, and a cross-replica Redis hit deserializes to a plain ``dict`` rather
-    than a ``UserAPIKeyAuth``; the previous code read only ``Authorization`` and did
-    ``getattr(cached, "user_id")`` with no ``model_type`` rehydration and no DB fallback, so it
-    silently returned ``None`` and the token was never persisted, which makes the egress 401 on every
-    reconnect. The resolved key is validated (``_active_key_user_id``) before its identity is trusted,
-    so a blocked or expired key cannot write. Returns ``None`` when no key is present, the key cannot
-    be resolved, or it is blocked/expired.
-    """
-    token = _litellm_key_from_request(request)
-    if not token:
-        return None
-    try:
-        from litellm.proxy._types import hash_token  # noqa: PLC0415
-        from litellm.proxy.auth.auth_checks import get_key_object  # noqa: PLC0415
-        from litellm.proxy.proxy_server import (  # noqa: PLC0415
-            prisma_client,
-            user_api_key_cache,
-        )
-
-        key_obj = await get_key_object(
-            hashed_token=hash_token(token),
-            prisma_client=prisma_client,
-            user_api_key_cache=user_api_key_cache,
-        )
-        return _active_key_user_id(key_obj)
-    except Exception as exc:
-        verbose_logger.debug(
-            "_extract_user_id_from_request: could not resolve a LiteLLM user_id for the presented "
-            "key (%s); per-user token will not be stored server-side.",
-            type(exc).__name__,
-        )
-        return None
-
-
 async def _store_per_user_token_server_side(
     server: MCPServer,
     user_id: str,
@@ -471,7 +476,8 @@ def _raise_if_not_oauth2(mcp_server: MCPServer) -> None:
     through: the caller owns the upstream token, and this relayed flow is how a browser obtains
     one against the upstream IdP (the admin UI's browser-only Authorize uses it). The minted
     token is upstream-audienced and held by the caller; the gateway persists nothing for these
-    modes (DCR persistence is opt-in and never enabled on this path).
+    modes (``_persist_dcr_client_registration`` skips them unconditionally, so even the admin
+    Authorize path with ``persist_credentials`` enabled writes nothing to the server row).
     """
     from litellm.proxy._experimental.mcp_server.mcp_server_manager import (  # noqa: PLC0415  # circular import with mcp_server_manager at module load
         _UPSTREAM_OAUTH_DISCOVERY_AUTH_TYPES,
@@ -498,23 +504,86 @@ def _raise_unless_oauth2_discovery_server(
     mcp_server_name: Optional[str],
     description: str,
 ) -> None:
-    """404 a NAMED discovery request unless it resolves to an oauth2 server.
+    """404 a NAMED discovery request unless it resolves to an oauth2 or DCR-bridge server.
 
     A named server that is unknown (or hidden from the caller) and one that exists
     but is non-oauth2 both return the same 404, so the well-known discovery paths
     cannot be used to enumerate non-OAuth server names. Root discovery (no name) is
     unaffected, and pass-through servers are resolved by the caller before this runs.
+    DCR-bridge servers are admitted because they serve the gateway's own authorization
+    server metadata (the register, authorize, and token relays).
     """
     if mcp_server_name is None:
         return
     if mcp_server is not None and mcp_server.auth_type == MCPAuth.oauth2:
         return
+    if mcp_server is not None and mcp_server.is_dcr_bridge:
+        return
     raise HTTPException(
         status_code=404,
         detail=f"MCP server '{mcp_server_name}' is {description}",
     )
 
 
+def _dcr_bridge_relays_client_registration(mcp_server: MCPServer) -> bool:
+    """True when a DCR-bridge server relays client registration to the upstream authorization
+    server instead of short-circuiting to an admin-configured OAuth client. In the relay arm the
+    upstream holds each client's own registration, so the authorize and token relays pass the
+    client's ``client_id`` and ``redirect_uri`` through verbatim and the authorization code
+    returns directly to the client's redirect URI without transiting the gateway. Gateway-side
+    redirect trust and the ``/callback`` state relay therefore only apply to the short-circuit
+    arm, where the upstream only knows the gateway's own callback."""
+    return mcp_server.is_dcr_bridge and bool(mcp_server.registration_url) and not mcp_server.client_id
+
+
+def _require_s256_pkce(
+    code_challenge: Optional[str],
+    code_challenge_method: Optional[str],
+) -> Tuple[str, str]:
+    """DCR-bridge servers serve unauthenticated public OAuth clients, so the PKCE downgrade
+    paths (no challenge, or a non-S256 method; RFC 7636 defaults a missing method to ``plain``)
+    are rejected at the gateway instead of relying on upstream enforcement. Returns the
+    validated pair so callers get non-optional values."""
+    if code_challenge and code_challenge_method == "S256":
+        return code_challenge, code_challenge_method
+    raise HTTPException(
+        status_code=400,
+        detail=(
+            "This server requires PKCE: send code_challenge with "
+            "code_challenge_method=S256 on the authorization request"
+        ),
+    )
+
+
+def _redirect_to_upstream_authorize(
+    *,
+    mcp_server: MCPServer,
+    client_id: str,
+    redirect_uri: str,
+    state: str,
+    code_challenge: str,
+    code_challenge_method: str,
+    response_type: Optional[str],
+    scope: Optional[str],
+) -> RedirectResponse:
+    """The bridge relay arm's authorize redirect: every client-supplied parameter passes through
+    to the upstream authorize endpoint verbatim, no relay state cookie is set, and the upstream
+    enforces its own registered redirect binding for the client."""
+    scope_value = scope or (" ".join(mcp_server.scopes) if mcp_server.scopes else None)
+    passthrough_params = {
+        "client_id": client_id,
+        "redirect_uri": redirect_uri,
+        "state": state,
+        "response_type": response_type or "code",
+        "code_challenge": code_challenge,
+        "code_challenge_method": code_challenge_method,
+        **({"scope": scope_value} if scope_value else {}),
+    }
+    parsed_auth_url = urlparse(mcp_server.authorization_url or "")
+    merged_params = {**dict(parse_qsl(parsed_auth_url.query)), **passthrough_params}
+    return RedirectResponse(urlunparse(parsed_auth_url._replace(query=urlencode(merged_params))))
+
+
 async def authorize_with_server(
     request: Request,
     mcp_server: MCPServer,
@@ -530,6 +599,24 @@ async def authorize_with_server(
     if mcp_server.authorization_url is None:
         raise HTTPException(status_code=400, detail="MCP server authorization url is not set")
 
+    if mcp_server.is_dcr_bridge:
+        # Enforce S256 PKCE on both bridge arms. The relay arm forwards the validated,
+        # now-non-optional pair to the upstream authorize; the short-circuit arm keeps
+        # calling this for its enforcement side effect, then falls through to the gateway
+        # /callback flow below, which reads the original code_challenge names.
+        bridge_challenge, bridge_method = _require_s256_pkce(code_challenge, code_challenge_method)
+        if _dcr_bridge_relays_client_registration(mcp_server):
+            return _redirect_to_upstream_authorize(
+                mcp_server=mcp_server,
+                client_id=client_id,
+                redirect_uri=redirect_uri,
+                state=state,
+                code_challenge=bridge_challenge,
+                code_challenge_method=bridge_method,
+                response_type=response_type,
+                scope=scope,
+            )
+
     # Trusted redirect_uri: same-origin, loopback, or ops-allowlisted.
     # The URI is encrypted into the OAuth state and decoded on
     # /callback to redirect the user back; a non-trusted URI would be
@@ -538,12 +625,31 @@ async def authorize_with_server(
     parsed = urlparse(redirect_uri)
     base_url = urlunparse(parsed._replace(query=""))
     request_base_url = get_request_base_url(request)
+
+    # Interactive dcr_bridge oauth_delegate sign-in: this arm runs the gateway /callback and /token in
+    # the loop, so the gateway can capture the litellm user here (from the browser's UI session) and
+    # carry it to the back-channel token mint. Seal the SSO user and the target server into the state;
+    # the callback reads them back to mint the gateway authorization code. A DCR client cannot present a
+    # litellm key, so the browser session is the only identity source; without one there is nothing to
+    # bind, so send the user through login first. Every other oauth2 server keeps the identity-less state.
+    litellm_user_id: str | None = None
+    if mcp_server.is_dcr_bridge and mcp_server.is_oauth_delegate:
+        from litellm.proxy._experimental.mcp_server.byok_oauth_endpoints import (  # noqa: PLC0415  # inline import avoids a module-load circular import
+            _user_id_from_session_cookie,
+        )
+
+        litellm_user_id = _user_id_from_session_cookie(request)
+        if litellm_user_id is None:
+            return _redirect_to_litellm_login(request)
+
     encoded_state = encode_state_with_base_url(
         base_url=base_url,
         original_state=state,
         code_challenge=code_challenge,
         code_challenge_method=code_challenge_method,
         client_redirect_uri=redirect_uri,
+        litellm_user_id=litellm_user_id,
+        mcp_server_id=mcp_server.server_id if litellm_user_id else None,
     )
     relay_state = secrets.token_urlsafe(_OAUTH_STATE_HANDLE_BYTES)
 
@@ -572,6 +678,13 @@ async def authorize_with_server(
     return response
 
 
+def _token_credential_source(mcp_server: MCPServer) -> CredentialSource:
+    """Mirrors the resolved-client rule in :func:`exchange_token_with_server`: when the server has a
+    stored client_id the gateway presents its own credentials upstream, so a credential rejection is
+    the operator's fault, not the caller's."""
+    return "gateway_stored" if mcp_server.client_id else "caller_supplied"
+
+
 async def exchange_token_with_server(
     request: Request,
     mcp_server: MCPServer,
@@ -606,50 +719,123 @@ async def exchange_token_with_server(
     except TokenEndpointAuthConfigError as exc:
         raise HTTPException(status_code=400, detail=str(exc)) from exc
 
+    bridge_identity: _BridgeAuthorizationCode | None = None
+    bridge_mint_ready: _BridgeMintReady | None = None
+    bridge_upstream_refresh: SecretStr | None = None
+    bridge_upstream_scope: str | None = None
+    refresh_request_scope: str | None = None
+    is_bridge = mcp_server.is_oauth_delegate and mcp_server.is_dcr_bridge
+
     if grant_type == "refresh_token":
-        if not refresh_token:
+        # Phase 1 for a bridge refresh: open the client's refresh envelope, re-validate the sealed
+        # identity, and unwrap the real upstream refresh token BEFORE building token_data, so the exchange
+        # sends the upstream token and never the envelope. A failure returns without touching the upstream.
+        if is_bridge:
+            prepared_refresh = await _prepare_bridge_refresh(mcp_server, refresh_token)
+            if not isinstance(prepared_refresh, _BridgeRefreshReady):
+                return _bridge_mint_error_response(prepared_refresh)
+            bridge_mint_ready = prepared_refresh.ready
+            bridge_upstream_refresh = prepared_refresh.upstream_refresh_token
+            bridge_upstream_scope = prepared_refresh.upstream_scope
+        # A bridge server sends the unwrapped upstream refresh token recovered from the client's refresh
+        # envelope above; every other server sends the client's own refresh token verbatim.
+        upstream_refresh_token = (
+            bridge_upstream_refresh.get_secret_value() if bridge_upstream_refresh is not None else refresh_token
+        )
+        if not upstream_refresh_token:
             raise HTTPException(
                 status_code=400,
                 detail="refresh_token is required for refresh_token grant",
             )
         token_data: dict = {
             "grant_type": "refresh_token",
-            "refresh_token": refresh_token,
+            "refresh_token": upstream_refresh_token,
             **client_auth.body,
         }
-        if scope:
-            token_data["scope"] = scope
+        refresh_request_scope = scope or bridge_upstream_scope
+        if refresh_request_scope:
+            token_data["scope"] = refresh_request_scope
     else:
         if not code:
             raise HTTPException(
                 status_code=400,
                 detail="code is required for authorization_code grant",
             )
+        # Interactive dcr_bridge oauth_delegate: the client presents the gateway authorization code the
+        # callback sealed. Recover the SSO user and the real upstream code from it; the upstream exchange
+        # below uses the upstream code, and the mint binds the envelope to the recovered user. Bind the
+        # sealed server to this request so a code minted for one bridge server cannot be spent at another.
+        # A raw upstream code (scripted path) opens to None and the code is used as-is.
+        bridge_identity = open_bridge_authorization_code(code)
+        if bridge_identity is not None:
+            if bridge_identity.mcp_server_id != mcp_server.server_id:
+                raise HTTPException(
+                    status_code=400,
+                    detail="Authorization code was issued for a different MCP server",
+                )
+            code = bridge_identity.upstream_code
+        bridge_token_relay = _dcr_bridge_relays_client_registration(mcp_server)
+        if bridge_token_relay and not redirect_uri:
+            raise HTTPException(
+                status_code=400,
+                detail=(
+                    "redirect_uri is required for the authorization_code grant on this server; "
+                    "send the same redirect_uri used on the authorization request"
+                ),
+            )
         proxy_base_url = get_request_base_url(request)
+        resolved_redirect_uri = redirect_uri if bridge_token_relay else f"{proxy_base_url}/callback"
         token_data = {
             "grant_type": "authorization_code",
             "code": code,
-            "redirect_uri": f"{proxy_base_url}/callback",
+            "redirect_uri": resolved_redirect_uri,
             **client_auth.body,
         }
         if code_verifier:
             token_data["code_verifier"] = code_verifier
-
+        # Phase 1 for a bridge authorization_code mint: resolve identity (the SSO user recovered above, or
+        # the presented litellm key) and the envelope keys BEFORE the exchange consumes the single-use code.
+        if is_bridge:
+            prepared = await _prepare_bridge_mint(request, mcp_server, bridge_identity)
+            if not isinstance(prepared, _BridgeMintReady):
+                return _bridge_mint_error_response(prepared)
+            bridge_mint_ready = prepared
     async_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.Oauth2Check)
-    response = await async_client.post(
-        mcp_server.token_url,
-        headers={"Accept": "application/json", **client_auth.headers},
-        data=token_data,
-    )
+    try:
+        response = await async_client.post(
+            mcp_server.token_url,
+            headers={"Accept": "application/json", **client_auth.headers},
+            data=token_data,
+        )
+        if response is not None:
+            response.raise_for_status()
+    except httpx.HTTPStatusError as exc:
+        fault = classify_upstream_token_rejection(
+            exc.response,
+            credential_source=_token_credential_source(mcp_server),
+            log_context=mcp_server.server_id,
+        )
+        upstream_rejected_bridge_refresh = (
+            is_bridge
+            and grant_type == "refresh_token"
+            and isinstance(fault, CallerRejected)
+            and fault.code == "invalid_grant"
+        )
+        if upstream_rejected_bridge_refresh:
+            verbose_logger.info(
+                "bridge refresh: the upstream rejected the sealed refresh token for server=%s with "
+                "invalid_grant (revoked or expired at the IdP); returning invalid_grant so the client "
+                "re-runs authorization_code rather than an opaque upstream error",
+                mcp_server.server_id,
+            )
+            return _bridge_mint_error_response("invalid_refresh")
+        return render_token_fault(fault)
     if response is None:
         raise HTTPException(
             status_code=502,
             detail="MCP upstream token endpoint returned no response",
         )
-
-    response.raise_for_status()
     token_response = response.json()
-    access_token = token_response["access_token"]
 
     # Validate token response against server-configured rules before any storage.
     # This rejects tokens from wrong Slack workspaces, Atlassian orgs, etc.
@@ -689,8 +875,23 @@ async def exchange_token_with_server(
                 mcp_server.server_id,
             )
 
+    # A DCR-bridge oauth_delegate server hands the client a gateway-bound envelope (identity plus the
+    # upstream token) instead of the raw upstream token, so the one bearer both admits the caller and
+    # forwards the upstream credential. Only this mode mints; every other server returns the raw token.
+    if bridge_mint_ready is not None:
+        if refresh_request_scope and isinstance(token_response, dict) and not token_response.get("scope"):
+            token_response = {**token_response, "scope": refresh_request_scope}
+        # Phase 3: seal the upstream grant into the client-held envelope; failures map through the same
+        # OAuth-shaped response as the phase-1 preconditions.
+        minted = _finish_bridge_mint(bridge_mint_ready, mcp_server, token_response, datetime.now(timezone.utc))
+        return minted if isinstance(minted, JSONResponse) else _bridge_mint_error_response(minted)
+
+    raw_access_token = token_response.get("access_token") if isinstance(token_response, dict) else None
+    if not isinstance(raw_access_token, str) or not raw_access_token:
+        return render_token_fault(UpstreamProtocolFault(note="the upstream token response has no usable access_token"))
+
     result = {
-        "access_token": access_token,
+        "access_token": raw_access_token,
         "token_type": token_response.get("token_type", "Bearer"),
     }
 
@@ -718,6 +919,22 @@ class _PersistedDcrCredentials(BaseModel):
     client_id: Optional[str] = None
     client_secret: Optional[str] = None
     token_endpoint_auth_method: Optional[str] = None
+    redirect_uris: Optional[list[str]] = None
+
+
+def _redirect_uri_not_registered(credentials: _PersistedDcrCredentials, current_redirect_uri: str) -> bool:
+    """Whether a persisted DCR client is positively known NOT to cover the current callback.
+
+    A DCR client is bound to the redirect_uris it was registered with; if the proxy's
+    resolved public origin has since changed, every authorize built for it will be
+    rejected by the IdP. Clients persisted before ``redirect_uris`` was recorded (and
+    admin-configured clients, which never get a recording) return False so they are
+    grandfathered rather than re-registered, because re-minting a client_id orphans
+    every user's refresh tokens for that server."""
+    recorded = credentials.redirect_uris
+    if not recorded:
+        return False
+    return current_redirect_uri not in recorded
 
 
 def _get_persisted_dcr_credentials(credentials: object) -> Optional[_PersistedDcrCredentials]:
@@ -784,11 +1001,23 @@ async def _get_persisted_mcp_server_with_dcr_client_id(
     return persisted_mcp_server, credentials
 
 
-async def _reuse_persisted_dcr_client_if_available(mcp_server: MCPServer) -> bool:
+async def _reuse_persisted_dcr_client_if_available(
+    mcp_server: MCPServer, current_redirect_uri: Optional[str] = None
+) -> bool:
     persisted = await _get_persisted_mcp_server_with_dcr_client_id(mcp_server)
     if persisted is None:
         return False
     persisted_mcp_server, credentials = persisted
+    if current_redirect_uri is not None and _redirect_uri_not_registered(credentials, current_redirect_uri):
+        verbose_logger.debug(
+            "register_client_with_server: not reusing persisted DCR client for server_id=%s; its registered "
+            "redirect_uris=%s do not include the current callback %s. The operator-facing warning for this "
+            "re-registration event is emitted once by _persisted_dcr_redirect_uri_is_stale.",
+            mcp_server.server_id,
+            credentials.redirect_uris,
+            current_redirect_uri,
+        )
+        return False
     if not _apply_persisted_dcr_credentials(mcp_server, credentials):
         return False
 
@@ -807,11 +1036,36 @@ async def _reuse_persisted_dcr_client_if_available(mcp_server: MCPServer) -> boo
     return bool(mcp_server.client_id)
 
 
-DcrRegistrationPersistenceResult = Literal["persisted", "reused", "failed"]
+async def _persisted_dcr_redirect_uri_is_stale(mcp_server: MCPServer, current_redirect_uri: str) -> bool:
+    """Whether the server's persisted DCR client is bound to redirect_uris that no longer
+    cover the current proxy callback, meaning authorize is guaranteed to fail IdP-side.
+
+    Consulted when the in-memory server already carries a hydrated client_id, which
+    otherwise short-circuits registration before any redirect check can run. Servers
+    without a persisted DCR recording (admin-configured client_id, or registered before
+    redirect_uris were recorded) are never reported stale."""
+    persisted = await _get_persisted_mcp_server_with_dcr_client_id(mcp_server)
+    if persisted is None:
+        return False
+    _, credentials = persisted
+    if not _redirect_uri_not_registered(credentials, current_redirect_uri):
+        return False
+    verbose_logger.warning(
+        "register_client_with_server: persisted DCR client for server_id=%s is registered with redirect_uris=%s "
+        "which do not include the current callback %s (proxy origin changed); registering a replacement client. "
+        "Users previously signed in to this server will need to re-authenticate.",
+        mcp_server.server_id,
+        credentials.redirect_uris,
+        current_redirect_uri,
+    )
+    return True
+
+
+DcrRegistrationPersistenceResult = Literal["persisted", "reused", "skipped", "failed"]
 
 
 async def _persist_dcr_client_registration(
-    mcp_server: MCPServer, registration_response: object
+    mcp_server: MCPServer, registration_response: object, current_redirect_uri: str
 ) -> DcrRegistrationPersistenceResult:
     """Persist the dynamically registered OAuth client (RFC 7591) onto the MCP server row.
 
@@ -821,7 +1075,23 @@ async def _persist_dcr_client_registration(
     full re-authorization instead of a silent refresh. Mirrors the ``encrypt_credentials``
     write that ``client_credentials`` and token exchange already use. Failures are logged,
     never raised: registration still returns to the caller even when persistence fails.
+
+    The client-forwarded token modes (``true_passthrough`` / ``oauth_delegate``) are skipped
+    unconditionally: the caller holds the upstream token and the gateway must hold no OAuth
+    client identity for these servers. Persisting here would stamp ``oauth2_flow`` and a
+    ``client_id`` onto a server whose mode promises the gateway stores nothing, making a
+    fresh pass-through server read as gateway-authorized.
+
+    ``redirect_uris`` records what the client is bound to so a later origin change can be
+    detected as a positive mismatch and trigger re-registration instead of stranding the
+    server on IdP-side redirect_uri rejections. ``client_secret`` and
+    ``token_endpoint_auth_method`` are written explicitly (None when absent) because
+    ``update_mcp_server`` merges credential blobs: a re-registered public client must not
+    inherit the previous client's secret or auth method.
     """
+    if mcp_server.is_true_passthrough or mcp_server.is_oauth_delegate:
+        return "skipped"
+
     try:
         registration = _DcrClientRegistration.model_validate(registration_response)
     except ValidationError as exc:
@@ -833,17 +1103,16 @@ async def _persist_dcr_client_registration(
         )
         return "failed"
 
-    if await _reuse_persisted_dcr_client_if_available(mcp_server):
+    if await _reuse_persisted_dcr_client_if_available(mcp_server, current_redirect_uri=current_redirect_uri):
         return "reused"
 
     credentials: MCPCredentials = {
         "client_id": registration.client_id,
-        **({"client_secret": registration.client_secret} if registration.client_secret is not None else {}),
-        **(
-            {"token_endpoint_auth_method": "client_secret_basic"}
-            if registration.token_endpoint_auth_method == "client_secret_basic"
-            else {}
+        "client_secret": registration.client_secret,
+        "token_endpoint_auth_method": (
+            "client_secret_basic" if registration.token_endpoint_auth_method == "client_secret_basic" else None
         ),
+        "redirect_uris": [current_redirect_uri],
     }
 
     from litellm.proxy._experimental.mcp_server.db import update_mcp_server  # noqa: PLC0415
@@ -887,19 +1156,28 @@ async def register_client_with_server(
     token_endpoint_auth_method: Optional[str],
     fallback_client_id: Optional[str] = None,
     persist_credentials: bool = False,
+    client_redirect_uris: Optional[list] = None,
 ):
     _raise_if_not_oauth2(mcp_server)
     request_base_url = get_request_base_url(request)
+    current_redirect_uri = f"{request_base_url}/callback"
     dummy_return = {
         "client_id": fallback_client_id or mcp_server.server_name,
         "client_secret": "dummy",
-        "redirect_uris": [f"{request_base_url}/callback"],
+        "redirect_uris": [current_redirect_uri],
     }
 
-    if mcp_server.client_id:
+    if mcp_server.client_id and not (
+        persist_credentials
+        and mcp_server.registration_url
+        and await _persisted_dcr_redirect_uri_is_stale(mcp_server, current_redirect_uri)
+    ):
         return dummy_return
 
-    if await _reuse_persisted_dcr_client_if_available(mcp_server):
+    if await _reuse_persisted_dcr_client_if_available(
+        mcp_server,
+        current_redirect_uri=current_redirect_uri if persist_credentials else None,
+    ):
         return dummy_return
 
     if mcp_server.authorization_url is None:
@@ -908,12 +1186,19 @@ async def register_client_with_server(
     if mcp_server.registration_url is None:
         return dummy_return
 
+    bridge_relay = _dcr_bridge_relays_client_registration(mcp_server)
+    if bridge_relay and not client_redirect_uris:
+        raise HTTPException(
+            status_code=400,
+            detail="redirect_uris is required to register a client with this server",
+        )
+
     register_data = {
         "client_name": client_name,
-        "redirect_uris": [f"{request_base_url}/callback"],
-        "grant_types": grant_types or [],
-        "response_types": response_types or [],
-        "token_endpoint_auth_method": token_endpoint_auth_method or "",
+        "redirect_uris": client_redirect_uris if bridge_relay else [current_redirect_uri],
+        "grant_types": grant_types or (["authorization_code", "refresh_token"] if bridge_relay else []),
+        "response_types": response_types or (["code"] if bridge_relay else []),
+        "token_endpoint_auth_method": token_endpoint_auth_method or ("none" if bridge_relay else ""),
     }
     headers = {
         "Content-Type": "application/json",
@@ -921,22 +1206,29 @@ async def register_client_with_server(
     }
 
     async_client = get_async_httpx_client(llm_provider=httpxSpecialProvider.Oauth2Register)
-    response = await async_client.post(
-        mcp_server.registration_url,
-        headers=headers,
-        json=register_data,
-    )
+    try:
+        response = await async_client.post(
+            mcp_server.registration_url,
+            headers=headers,
+            json=register_data,
+        )
+        if response is not None:
+            response.raise_for_status()
+    except httpx.HTTPStatusError as exc:
+        status_code, detail = dcr_fault_detail(
+            classify_upstream_dcr_rejection(exc.response, log_context=mcp_server.server_id)
+        )
+        raise HTTPException(status_code=status_code, detail=detail) from exc
     if response is None:
         raise HTTPException(
             status_code=502,
             detail="MCP upstream registration endpoint returned no response",
         )
-    response.raise_for_status()
 
     token_response = response.json()
 
-    if persist_credentials:
-        persistence_result = await _persist_dcr_client_registration(mcp_server, token_response)
+    if persist_credentials and not bridge_relay:
+        persistence_result = await _persist_dcr_client_registration(mcp_server, token_response, current_redirect_uri)
         if persistence_result == "reused":
             return dummy_return
 
@@ -1159,7 +1451,20 @@ async def callback(
         # states while permitting same-origin / allowlisted clients.
         redirect_uri = _get_validated_client_redirect_uri(request, state_data)
 
-        params = {"code": code, "state": original_state}
+        # Interactive dcr_bridge oauth_delegate: the state carries the litellm user the authorize step
+        # captured. Instead of forwarding the raw upstream code (which the client would present at the
+        # token endpoint with no way to prove who signed in), seal the user and the upstream code into a
+        # gateway authorization code and forward THAT. The token endpoint decrypts it to bind the
+        # envelope to this user. Every other flow forwards the raw code unchanged.
+        litellm_user_id = state_data.get("litellm_user_id")
+        mcp_server_id = state_data.get("mcp_server_id")
+        forwarded_code = code
+        if isinstance(litellm_user_id, str) and litellm_user_id and isinstance(mcp_server_id, str) and mcp_server_id:
+            forwarded_code = seal_bridge_authorization_code(
+                upstream_code=code, litellm_user_id=litellm_user_id, mcp_server_id=mcp_server_id
+            )
+
+        params = {"code": forwarded_code, "state": original_state}
         complete_returned_url = _append_query_params(redirect_uri, params)
         response = RedirectResponse(url=complete_returned_url, status_code=302)
         _clear_oauth_state_cookie(response, request, state)
@@ -1359,6 +1664,13 @@ async def _build_oauth_protected_resource_response(
     else:
         resource_url = f"{request_base_url}/mcp"
 
+    if mcp_server is not None and mcp_server_name and mcp_server.is_dcr_bridge:
+        return {
+            "authorization_servers": [f"{request_base_url}/{mcp_server_name}"],
+            "resource": resource_url,
+            "scopes_supported": (mcp_server.scopes if mcp_server.scopes else []),
+        }
+
     # Pass-through branch: proxy the upstream's own metadata so discovery
     # directs the client at the real IdP (Okta, Keycloak, …) instead of us.
     if mcp_server is not None and (
@@ -1688,6 +2000,7 @@ async def register_client(request: Request, mcp_server_name: Optional[str] = Non
                 response_types=data.get("response_types", []),
                 token_endpoint_auth_method=data.get("token_endpoint_auth_method", ""),
                 fallback_client_id=resolved.server_name or resolved.name,
+                client_redirect_uris=data.get("redirect_uris"),
             )
         return dummy_return
 
@@ -1702,4 +2015,5 @@ async def register_client(request: Request, mcp_server_name: Optional[str] = Non
         response_types=data.get("response_types", []),
         token_endpoint_auth_method=data.get("token_endpoint_auth_method", ""),
         fallback_client_id=mcp_server_name,
+        client_redirect_uris=data.get("redirect_uris"),
     )

litellm/proxy/_experimental/mcp_server/faults/__init__.py

@@ -0,0 +1,38 @@
+"""Typed fault values for upstream OAuth/DCR failures (phase 1 of the MCP error-handling framework).
+
+The invariant this package exists to enforce: an upstream failure is classified ONCE into a single
+fault value, and the response status, wire error code, and prose are all derived from that value.
+Deriving all three from one classification makes contradictory pairings (a caller-fault error code on
+a server-fault status) unrepresentable, and gives the trust-boundary rule one enforcement point:
+spec-defined machine fields may cross to callers, upstream prose and raw bodies go to server logs.
+"""
+
+from litellm.proxy._experimental.mcp_server.faults.classify import (
+    classify_upstream_dcr_rejection,
+    classify_upstream_token_rejection,
+)
+from litellm.proxy._experimental.mcp_server.faults.render_oauth import (
+    dcr_fault_detail,
+    render_token_fault,
+)
+from litellm.proxy._experimental.mcp_server.faults.types import (
+    CallerRejected,
+    CredentialSource,
+    GatewayRejected,
+    UpstreamOAuthFault,
+    UpstreamProtocolFault,
+    UpstreamReportedFault,
+)
+
+__all__ = [
+    "CallerRejected",
+    "CredentialSource",
+    "GatewayRejected",
+    "UpstreamOAuthFault",
+    "UpstreamProtocolFault",
+    "UpstreamReportedFault",
+    "classify_upstream_dcr_rejection",
+    "classify_upstream_token_rejection",
+    "dcr_fault_detail",
+    "render_token_fault",
+]