The semantic tool filter builds its index by listing every MCP server
without per-user credentials, so servers needing per-user auth
(interactive OAuth tokens, user-scoped env vars) reject the anonymous
tools/list and contribute zero routes. Request-time expansion resolves
that auth, so filter_tools received tools the router could never
select: an empty index failed open N->N (past 128 tools OpenAI rejects
the request outright) and a partial index matched only unavailable
tools, stripping every tool from the request.
filter_tools now syncs missing tools into the router before matching
(building the router when absent) behind an asyncio lock so each tool
embeds once, and falls back to the full tool list when matches map to
no available tool, consistent with the zero-match fallback.
Context-window overflows keep failing closed.
An empty upstream delta (e.g. Bedrock Converse's empty reasoning delta
mid-thinking-block) falls through the translate fallback as
text_delta {"text": ""} at the open thinking block's index, crashing
Anthropic SDK clients like Claude Code with "Content block is not a
text block". Payload-less deltas carry no information, so never emit
them.
* feat(guardrails): support streaming text transformation in generic_guardrail_api
* chore(guardrails): address PR review feedback
* fix(guardrails): fail closed on tool-call and prefix-rewrite leaks in streaming transform
* fix(guardrails): address Bugbot review on streaming transform correctness
* fix(guardrails): coerce holdback in handler for in-process guardrails
* fix(guardrails): harden streaming transform (holdback coercion, tool-call passthrough, n>1 finish_reason)
* test(guardrails): targeted _mode_matches coverage for all guardrail_mode shapes
* fix(guardrails): inspect streamed tool calls and harden incremental_diff edge cases
* test: move ComplianceChecker mode tests to the compliance PR
* fix(guardrails): strip content from tool-call passthrough so streamed text can't bypass the transform
* fix(guardrails): four correctness fixes for incremental_diff streaming path
Four bug fixes on top of the OSS PR's incremental_diff streaming text
transformation, all inside the incremental_diff code paths only. No
existing block_only, non-streaming, or pre_call behavior is touched.
Fix#1 — Mixed content+tool_call finish_reason ordering
_tool_call_passthrough_chunk now takes an optional finish_reason_per_choice
map. For a choice carrying both delta.content and delta.tool_calls,
finish_reason is stripped from the passthrough and recorded on the map so
the final synthetic text chunk delivers it. Without this, SSE-compliant
clients stopping at finish_reason drop the guardrailed text — defeating
the redaction the whole feature exists for. (Greptile P1 twice, Veria.)
Fix#2 — Choice index sort in _process_streaming_transform
indices/texts_to_check were derived from dict insertion order. For n>1
streams where choice 1 emits before choice 0, guardrail-returned texts
aligned to the input order mapped back to the wrong choice indices on
write-back — wrong text goes to wrong choice. Sort raw_by_index.keys()
up front so realignment is deterministic. (Bugbot Medium.)
Fix#3 — Cross-chunk pre-tool-call text flush
With default streaming_sampling_rate=5, text chunks followed by a pure
tool-call chunk carrying finish_reason='tool_calls' would emit the
passthrough with finish_reason before any transformed text delta had
fired. Same failure mode as fix#1 but cross-chunk. Now we flush any
accumulated text via _round(is_final=False) BEFORE yielding the
tool-call passthrough. (Greptile P1.)
Fix#4 — Terminator chunk for deferred finish_reason on empty mutated_text
_build_transform_chunk returned None early when mutated_text_per_choice
was empty. If a mixed content+tool_call chunk had deferred its
finish_reason (via fix#1) and the guardrail then suppressed the text
(empty return), the deferred finish_reason was never delivered. Now on
is_final=True with empty mutated_text_per_choice, we emit a terminator
carrying finish_reason per choice from finish_reason_per_choice.
(Bugbot High.)
Also normalized Optional[X] → X | None across the OSS PR's added surface
via ruff UP045 autofix to keep the strict-rule gate within budget. Pure
mechanical typing style change, no semantic effect.
Regression tests for all four fixes:
- test_mixed_chunk_finish_reason_arrives_after_transformed_text (#1)
- test_text_flush_precedes_tool_call_passthrough (#3)
- test_final_finish_reason_flushed_when_guardrail_suppresses_text (#4)
- test_transform_sends_texts_sorted_by_choice_index (#2)
All fixes reachable only when streaming_transform_mode == 'incremental_diff'
is configured (via _run_incremental_transform_stream) or when a
StreamTransformSink is present (via _process_streaming_transform). Verified
scope-clean: no changes to block_only, non-streaming, pre_call, moderation,
or sibling guardrails.
---------
Co-authored-by: Marton Schneider <marton@schneider.co.nl>
With enable_jwt_auth enabled but no enterprise license (premium_user
False), the JWT premium check fired on every request before the token
was inspected, so the master key, sk- virtual keys, and the encrypted
CLI/UI SSO session token that `lite login` issues all 401'd with "JWT
Auth is an enterprise only feature" and were never decoded. That broke
`lite login`, `lite claude`, and the proxy master key on any deployment
that turned JWT auth on without a license.
Move the premium check inside the is_jwt branch so it gates only real
JWTs. Non-JWT credentials fall through to their own auth paths
regardless of license; actual JWTs still require premium, so the
enterprise gate is unchanged for the feature it protects.
grok-4.3 is a third-party frontier model on Bedrock Mantle, served on the
/openai/v1 base (like gpt-5.x and gemma-4), not the standard /v1 path used by
open-weights models such as gpt-oss. #31916 added the model without
use_openai_responses_path, so mantle_base_segment() routed it to /v1, where
Bedrock rejects the call with "Berm is not enabled for this account"
(access_denied) — the model is only reachable on the frontier /openai/v1 path.
Add use_openai_responses_path=true to the bedrock_mantle/xai.grok-4.3 price-map
entry (both model_prices_and_context_window.json and the backup) so
mantle_base_segment() returns "openai/v1", and update the registry test to
assert use_openai_path is True.
* fix: enforce user budget on team keys
User budget was skipped when the key belonged to a team, letting
users exceed their personal budget by going through a team key.
Remove the team_object guard in _user_max_budget_check so user
budgets are always enforced. Add skip_user_budget_on_team_key
general_settings flag to opt back into the old behavior.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: update test to expect user budget enforcement on team keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): enforce user budget on team keys in reservation path and expose skip flag in UI
Extends the read-time fix so the optimistic budget reservation also reserves the user spend counter for team-scoped keys, register skip_user_budget_on_team_key in ConfigGeneralSettings so /config/field/update accepts it, and surface it as a Boolean toggle on the Admin UI General Settings table via allowed_args in /config/list.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: assert budget_exceeded ProxyException in personal budget test
Tighten the broad pytest.raises(Exception) so the test only passes when
the auth flow rejects with a budget_exceeded ProxyException, and switch
the new ConfigGeneralSettings field to Optional[bool] to match the
surrounding annotation style
* fix: revert to bool | None to stay under UP045 strict budget
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models
* test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping
* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models
Narrow the fix to the temperature reconciliation; the reasoning_effort
budget cap is reverted because the live translation grid relies on
budget_tokens >= max_tokens to reject unsupported effort tiers
(xhigh/max) on budget-mode models, so capping turned those 400s into
200s.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
discoverable_endpoints.py had grown to 2695 lines mixing FastAPI route handlers with the dcr_bridge token-flow logic, against the no-monster-files convention. This moves the bridge token flow (the litellm-key/user resolution, the SCIM revalidation gate, and the mint/refresh envelope logic with their types and error mappers) into a dedicated bridge_token_flow.py, leaving the route handlers and the shared exchange_token_with_server orchestrator in discoverable_endpoints.py importing from it
Pure relocation, zero behavior change. The moved code is byte-verbatim except one type annotation quoted as a forward reference (_BridgeAuthorizationCode is used only for typing and imported under TYPE_CHECKING to avoid a cycle), and the new module imports nothing from discoverable_endpoints at runtime. 275 tests pass unchanged; the test patch targets for moved internals were repointed to the new module and verified to still apply
get_group_ids_from_service_principal only read the first page of the
Graph API appRoleAssignedTo response, so tenants with more than 100
groups assigned to the enterprise application silently lost group
memberships during SSO login. Loop over @odata.nextLink with the same
MAX_GRAPH_API_PAGES cap that get_user_groups_from_graph_api already
uses, and warn when the cap is hit.
Ported from #32792 by @saisurya237 so CI can run.
Fixes#32790
Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com>
* feat(router): opt-in session affinity for complexity router
Complexity router reclassified every turn, which could flip the routed
model group mid-session and break provider-side prompt caching. Add a
session_affinity config flag: when a session_id is resolvable, pin the
model chosen on the first turn and reuse it for the rest of the session,
skipping reclassification. Pinned turns still stamp the adaptive
bandit's chosen-model metadata so reward feedback keeps working when
adaptive=True.
* fix(router): refresh session-affinity TTL on hit, scope pin by API key
Two issues from review: the TTL was only set on the first classification,
so an active session outliving session_affinity_ttl_seconds silently lost
its pin instead of refreshing as documented. And the cache key was scoped
only by session_id, which is client-supplied and unauthenticated, so two
different callers reusing the same session_id could poison each other's
routing pin. Refresh the TTL on every cache hit, and namespace the cache
key by the proxy-derived API key hash when available.
* fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models'
key_type is not persisted on a key (the proxy maps it to allowed_routes and
drops it), so the keys tables only inspected the models list and rendered
'All Proxy Models' for any key with an empty models array, including SCIM,
Management and Read-only keys that cannot call a single model.
Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a
scope tooltip for those recognized scopes; unrestricted, AI-API and custom
keys keep the existing model-list rendering.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): move key_scope helper to components root
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(keys): persist key_type on virtual keys so the UI reads scope directly
Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy,
and proxy-extras schemas plus an additive migration) and stop dropping the
value in handle_key_type, so management/read_only/llm_api/default keys store
their type alongside the derived allowed_routes. Surface it on the key read
and create response models. The dashboard now prefers the persisted key_type
for the no-inference buckets and keeps the allowed_routes derivation as the
fallback for keys created before the column existed (key_type null), so no
backfill is required.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): add key_type column to LiteLLM_DeletedVerificationToken
The deleted-token archive model inherits key_type from the verification
token, so regenerate/delete flows write key_type into
LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration)
so the archive insert does not fail with FieldNotFoundError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The refresh-envelope helpers merge cleanly alongside the faults package. The upstream invalid_grant
special case for bridge refreshes moves inside the post-call except branch (its old location after a
second raise_for_status would be unreachable under the call-time-raise structure this branch
introduced) and now keys off the classified fault; _upstream_oauth_error is dropped since the
classifier already parses the RFC 6749 error field with total accessors
Extends the fault matrix per review: server_error and temporarily_unavailable are codes by which the
upstream blames itself, so they classify as a new UpstreamReportedFault arm rendering 502/503 with a
matching wire code instead of a 400 that blames the caller; invalid_target is a gateway capability
gap (RFC 8707 resource indicators, LIT-4339) and is gateway-blamed regardless of whose credentials
were presented; the DCR classifier shares the same blame assignment. The gateway-fault arm is renamed
GatewayRejected since it now covers capability gaps as well as stored-credential rejections
The bridge refresh path decided whether an upstream token-endpoint rejection was invalid_grant by substring-matching the raw response body, so a rejection whose actual error is something else but whose error_description merely contains the string invalid_grant would false-match, map to invalid_grant, and trigger a needless authorization_code re-run
Parse the RFC 6749 section 5.2 error object and compare the error field. A non-JSON body, or an error that is not invalid_grant, now propagates as the upstream error rather than being reinterpreted. The regression test drives an invalid_client rejection whose description contains the string invalid_grant and asserts it is not mapped, mutation-checked against the substring match
Replaces the accreted relay helpers with a faults package (types, classify, render_oauth): every
upstream token/DCR rejection is classified into exactly one fault value and the response status,
wire error code, and prose are all derived from that value, so a caller-fault code can never ship
on a server-fault status (the bugbot finding on invalid_grant over a 500). Classification takes the
credential source into account: invalid_client and friends against the server's stored credentials
are the operator's fault and render as 502 server_error with gateway-authored prose while the IdP's
prose stays in server logs; the same codes against caller-supplied credentials relay on the status
the code implies. Classifiers are total, so an unreadable rejection body (lying content-encoding,
unconsumed stream) yields the same 502 fault instead of resurrecting the opaque 500 (the second
bugbot finding); DCR rejections normalize to 400 per RFC 7591 regardless of the upstream's status
The prior fix sent the sealed scope on a refresh, but the re-minted refresh envelope re-seals scope from the upstream response, and RFC 6749 section 5.1 lets an upstream omit scope when it is unchanged. So after one refresh whose response omitted scope, the new envelope sealed scope=None and every subsequent refresh dropped it, letting a stricter upstream narrow the renewed token
When the upstream omits scope on a bridge refresh, seal the scope we requested (which RFC 6749 section 5.1 defines as the granted scope when omitted) into the renewed access and refresh envelopes, so the scope survives the whole refresh chain. The regression test refreshes against an upstream that omits scope, asserts the new refresh envelope still carries it, and refreshes again off that envelope to prove the chain does not lose it, mutation-checked
The refresh envelope seals the upstream scope as the scope to re-request (RefreshCredential), but _prepare_bridge_refresh dropped it, unwrapping only the refresh token, and the exchange added scope to the upstream request only from the client's HTTP form. A DCR/MCP client typically omits scope on refresh, so the sealed scope was never sent and a stricter upstream could narrow or drop the renewed token's scope
Thread the sealed scope through _BridgeRefreshReady.upstream_scope and fall back to it when the client sends none; a client-supplied scope still wins, which RFC 6749 section 6 bounds to the original grant. The regression test drives a refresh where the client omits scope and asserts the upstream POST carries the sealed scope, mutation-checked against both the drop and the fallback
The token and DCR relays serve unauthenticated OAuth clients, so only the RFC 6749/7591 error fields may cross the trust boundary. A rejection body outside those contracts (HTML error page, proxy banner, stack trace) is now logged server-side, bounded, and the client response names only the upstream status. Addresses the Veria information-exposure finding
* fix(anthropic-messages): send bare-string tool_choice to Responses API, propagate router-alias litellm_params
The Anthropic /v1/messages -> Responses API adapter always wrapped
tool_choice in an object ({"type": "auto"}, {"type": "required"}), but
the Responses API's tool_choice schema for these cases is a bare
string ("auto"/"required"/"none"). Sending the object shape to an
OpenAI-compatible backend (e.g. vLLM) fails Pydantic validation with a
400. The "none" case also fell through to "auto" instead of mapping to
"none".
Separately, litellm_params configured directly on a router-alias
deployment (auto_router/complexity_router, adaptive_router,
quality_router, or semantic auto_router) - e.g.
cache_control_injection_points, drop_params - were silently dropped
for every request through that alias. async_pre_routing_hook swaps
`model` from the alias name to the selected tier/route's model before
the deployment lookup runs, so the outbound call only ever merged in
the tier deployment's own litellm_params, never the alias's. Register
non-routing-config litellm_params from the alias deployment and apply
them to the request whenever a pre-routing hook substitutes the model.
* fix: satisfy ruff-strict-budget UP006 and router coverage checker
Use builtin dict[...] generics instead of typing.Dict for the two new
annotations introduced in the previous commit, since they pushed
UP006 over the codebase ceiling in ruff-strict-budget.json. Add a
direct unit test for _register_pre_routing_alias_overrides so the
text-based router_code_coverage.py checker sees it exercised by name.
* fix(router): replace alias-param denylist with a tight allowlist
_PRE_ROUTING_ALIAS_RESERVED_PARAMS excluded router-init-only keys from
the alias's litellm_params before forwarding the rest as request
kwargs, but GenericLiteLLMParams also holds deployment-management
fields (tpm, rpm, weight, tags, max_budget, budget_duration,
use_in_pass_through, litellm_credential_name, ...) on the same object.
Any of those left off the denylist would get silently forwarded as if
they were request kwargs.
Replace the denylist with a tight allowlist of exactly the two
request-shaping params this feature exists for - drop_params and
cache_control_injection_points - so unrelated management fields never
reach the outbound call regardless of what else GenericLiteLLMParams
grows to hold.
* fix(router): re-register adaptive-alias overrides on set_model_list reload
set_model_list() unconditionally clears pre_routing_alias_overrides on
every call (e.g. /config/reload), but _finalize_adaptive_router_if_configured()
skips rebuilding an AdaptiveRouter whose model_name already exists in
self.adaptive_routers - so _register_pre_routing_alias_overrides() never
ran again for an auto_router/adaptive_router alias after a reload,
silently dropping its drop_params/cache_control_injection_points.
Build the Deployment unconditionally and re-register its overrides even
on the skip-existing-router path; only the (expensive) AdaptiveRouter
construction itself stays skipped.
* style: ruff format after merging litellm_internal_staging
* fix(router): drop the alias-param allowlist, exclude only model
Per review discussion: instead of a router.py-local allowlist of exactly
which litellm_params an alias (auto_router/complexity_router,
adaptive_router, quality_router, semantic auto_router) can forward to
the request it routes, _register_pre_routing_alias_overrides now
forwards everything except `model` (the alias marker itself, e.g.
auto_router/complexity_router, never a real provider model).
Router-init-only fields (complexity_router_config,
complexity_router_default_model, auto_router_config,
auto_router_config_path, auto_router_default_model,
auto_router_embedding_model, adaptive_router_config,
adaptive_router_default_model, quality_router_config,
quality_router_default_model) now flow into request_kwargs unfiltered
too. That's safe because litellm.completion()/acompletion() already
strips anything in litellm.types.utils.all_litellm_params before
building the provider request - added these 10 keys there, alongside
the deployment-management fields (tpm, rpm, weight, ...) already listed.
Verified live: without that addition, complexity_router_config lands in
extra_body and ships raw to the provider; with it, it's stripped.
This moves the "which fields aren't real LLM params" list from a
router.py-local allowlist to the single existing global list every
completion() call already depends on, instead of maintaining two.
* refactor(router): look up alias litellm_params on demand instead of caching them
_register_pre_routing_alias_overrides cached each alias's litellm_params
into self.pre_routing_alias_overrides at deployment-init time, which
required keeping that cache in sync with set_model_list() reloads - the
exact bug the previous adaptive-router-reload fix was patching around
(AdaptiveRouter survives a reload, but the cache didn't always get
refreshed to match).
Delete the cache and the registration method entirely. async_pre_routing_hook
now looks up the alias's own litellm_params directly from self.model_list
via self.model_name_to_deployment_indices at request time, the same
model_list that's already correctly rebuilt on every set_model_list()
call. No second piece of state to invalidate, so the reload staleness
bug class isn't possible anymore, and it's less code than before.
Mutation testing surfaced branches in cost_tracking_settings and common_utils that the suite executed but never asserted on. Pin those behaviors with targeted tests: the returned (model, provider) from _resolve_model_for_cost_lookup for deployments carrying a custom_llm_provider and for deployments missing the litellm_params / model_info keys, plus the exact error-response bodies, the caller-identity lookup arguments, and the member and guard branches in common_utils.
Four fixes to the refresh_token grant for dcr_bridge oauth_delegate, surfaced by an adversarial pass over the exchange path
Route the user-subject re-validation's outage check through the chain-aware classifier, so a transient DB outage (which get_user_object wraps in a bare ValueError) reports as unavailable (a retryable 503) rather than collapsing to no_active_key and an invalid_grant, matching how admission now handles the same wrapper
When the upstream reports its own refresh token as already elapsed (refresh_expires_in non-positive), do not seal it into a full-TTL refresh envelope; return no refresh so the exchange degrades to an access-only response, mirroring how the access grant refuses an already-elapsed access token instead of capping it
When the upstream rejects the sealed refresh token with 400 invalid_grant (revoked or expired at the IdP), return an RFC 6749 invalid_grant response so the OAuth client re-runs authorization_code, rather than surfacing the opaque upstream error it cannot act on
Gate key-subject renewal on the owner's SCIM state, mirroring admission's _reject_if_admitted_owner_scim_deactivated, so an offboarded user cannot keep refreshing a still-active key; the check fails open on a missing owner or a DB blip so a key that outlives its owner record does not get wrongly revoked
Each fix has a mutation-checked regression test
Three review findings on the refresh path, addressed at the root:
_BridgeRefreshReady.upstream_refresh_token was a plain str, the one credential in the envelope/bridge
layer that escaped the SecretStr discipline every other one follows (RefreshCredential.refresh_token,
UpstreamTokenGrant.access_token, EnvelopeKeys.signing_key). A repr or a traceback capturing a local
_BridgeRefreshReady would have logged the raw upstream refresh token. It is now a SecretStr, carried as
the SecretStr open_bridge_refresh_envelope already returns and unwrapped only at the point the exchange
builds the upstream request body.
_prepare_bridge_refresh took a request it never read; on the refresh path identity comes entirely from
the sealed envelope, not the HTTP request, so the parameter was dead and misleadingly implied it read
from the request the way the authorization_code prepare does. Removed, and the caller updated.
_reload_active_user_by_id misclassified a missing user as unresolvable (500). This is the same root
cause as the admission user-reload fix: get_user_object raises a bare Exception for a deleted user
rather than a ProxyException, so its except-Exception arm must fail closed to no_active_key (which the
refresh path maps to invalid_grant) for anything that is not a database-service-unavailable outage,
rather than treating a missing user as an opaque gateway fault. Regression tests cover the missing-user
and DB-outage classifications directly.
The live proof showed a refresh envelope presented at the MCP tool-call edge was rejected, but through
the generic oauth2 arm ("expected a virtual key starting with sk-") rather than the bridge arm, because
the admission routing gate is_bridge_envelope_shaped matched only the access prefix. The rejection was
already fail-closed and never forwarded anything upstream, but the path was imprecise and the unit test
modelled a route the real router did not take.
Match either envelope kind in is_bridge_envelope_shaped so the bridge arm engages for a refresh envelope
too, and have resolve_bridge_envelope return BridgeEnvelopeInvalid for it: a refresh envelope is a valid
gateway credential but only ever presented back to the token endpoint, never usable to authenticate a
tool call. Admission now fails it closed with the bridge arm's own 401 ("Invalid or expired
credential"), live-verified, with the upstream never touched. is_bridge_envelope_shaped has a single
caller (the admission routing gate), so the change is contained.
A dcr_bridge oauth_delegate access envelope is capped at one hour, and until now the mode had no refresh
at all: when the envelope expired the client had to re-run the interactive authorization_code flow. This
adds a second client-held credential, the refresh envelope, so the client renews on a back channel and
only re-authenticates when the refresh envelope expires or the upstream refresh token dies.
The refresh envelope is a distinct llm_refresh_ credential that seals only the upstream refresh token
(never the access token) bound to the same litellm identity and MCP server as the access envelope, under
the same master-key-derived keys, with nothing stored server-side. Both envelopes now carry a signed
kind claim ("access" or "refresh") that open() requires to match, so a refresh envelope can never open as
an access credential even if its wire prefix is swapped (the prefix is not signed; the claim is). A
refresh envelope presented at the MCP tool-call edge is not an access envelope, so admission fails it
closed the same way it already fails any non-access bearer.
At the token endpoint the authorization_code mint now returns a refresh envelope alongside the access
envelope whenever the upstream returned a refresh token, and the refresh_token grant is supported for
bridge servers: the client presents its refresh envelope, the endpoint opens it, re-validates the sealed
litellm key so a revoked key cannot keep refreshing, unwraps the real upstream refresh token, exchanges
it with the upstream IdP, and returns a fresh access envelope. Because the endpoint re-seals a refresh
envelope only when the upstream returns a new refresh token, the design mirrors the upstream's own
rotation policy rather than reinventing it: with a rotating upstream the client rotates and reuse is
detected upstream; with a non-rotating upstream the original refresh envelope stands until its bounded
14-day TTL. Both preconditions and the unwrap run before the exchange, so a rejected refresh never
consumes or rotates an upstream token.
The pure envelope and credential layers stay side-effect free: mint/open share one signing, size, and
kind gate across both envelope kinds, and every failure is a value. Tests cover the refresh round-trip,
the kind-claim and server-id bindings, the revoked-key gate, upstream rotation carried through, the
unwrap sending the real upstream token upstream, and edge rejection of a refresh envelope; the three
security bindings are mutation-checked. Limitation documented in the PR: gateway-enforced refresh
rotation with reuse detection would require server-side state, which this zero-custody mode omits by
design, so the refresh envelope inherits the upstream's rotation posture plus gateway identity binding
and a bounded TTL.
* fix(openai/responses): clamp max_output_tokens below API minimum
Claude Code sends a max_tokens=1 warmup probe when running /model, which
the Anthropic Messages -> Responses adapter forwards as max_output_tokens=1.
OpenAI's Responses API rejects values below 16, so the probe failed with a
400. Clamp anything below the minimum up to 16 in map_openai_params so all
Responses API entrypoints (direct, chat->responses, anthropic->responses)
are covered.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(openai/responses): extract _enforce_min_max_output_tokens helper
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix response not being redacted for custom callbacks with streaming enabled
* reduce code duplication
* add unit test
* fix: resolve lint violations in adopted redaction fix
* fix: scope streaming response redaction to the opted-out custom logger
---------
Co-authored-by: Moritz Müller <moritz.mueller2@tu-dresden.de>
get_user_object catches every DB failure in a broad except and re-raises a bare ValueError (litellm/proxy/auth/auth_checks.py), so a real outage and a missing user look identical and the original error survives only as __context__. The dcr_bridge admission path keyed its 503-vs-401 decision on the exception type, so a transient outage during a user-subject reload surfaced as a 401 rather than a retryable 503, and the regression test injected a raw ConnectionError, a shape get_user_object never produces, so it passed on a fiction
Add PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain, which walks __cause__/__context__ (bounded and cycle-safe) the PEP 3134 way, and route _raise_503_if_db_unavailable through it. Move the user's object_permission resolution inside the single classified try so an outage there is a 503 too, never an opaque 500. Pin get_user_object's wrapping with a contract test that drives the real function, and drive the reload tests with that same faithful shape so a chain-blind regression fails them
An upstream token endpoint rejection (e.g. Google requiring client_secret even for PKCE web clients) escaped exchange_token_with_server as a raw httpx.HTTPStatusError, which the global exception handler turned into an opaque 500 Internal server error in the create-flow UI. The RFC 6749 section 5.2 error body the IdP sent (error, error_description, error_uri) is now relayed with the upstream's own 400/401 status; rejections outside the section 5.2 contract map to 502 so a broken upstream is not misattributed to the caller. The same relay covers the non-bridge DCR registration arm, and a 200 token response without a usable access_token now answers 502 instead of a KeyError 500. The catch wraps the post call itself because litellm's AsyncHTTPHandler raises MaskedHTTPStatusError at call time, which also made the pre-existing bridge-relay status check unreachable in production. The dashboard's token exchange error message now composes error and error_description so the form shows the IdP's reason
_reload_admitted_user returned a bare UserAPIKeyAuth(user_id=...), so the shared
get_allowed_mcp_servers found no key/team/object-permission grants and an interactive SSO client could
admit successfully yet see zero tools on a normal (allow_all_keys=False) server. The key path returns
the full key record whose object permission drives that computation; the user path dropped it.
Resolve the user's own MCP object permission and put it on the returned auth, so the same
get_allowed_mcp_servers the key path uses grants the user their litellm-granted servers and access
groups. This reuses get_object_permission (the id-to-grants resolver keys and teams already use) and
does not duplicate any permission logic; get_user_object does not load object_permission, so it is
resolved from the user's object_permission_id the same way the key and team paths do.
Only the user's own object permission is bound. A UserAPIKeyAuth carries a single team_id while a user
may belong to many teams, so team-inherited MCP grants for a user are a follow-up: they need a
many-teams union get_allowed_mcp_servers does not do off one auth object, and faking one here would be
the kind of half-measure that spawns more bugs. Tests cover the user's object permission riding onto the
admitted auth, and the existing admit/SCIM/missing-user/503 cases still hold.
