* fix(streaming): surface in-body error payloads on OpenAI-compatible streams
vLLM and sglang return HTTP 200 streams whose SSE body carries the error,
e.g. data: {"error": {"message": "...", "code": 400}}. The OpenAI-compatible
chunk parser had no detection for this shape: since #23931 the payload parsed
into an empty chunk (choices=[]) and the stream ended silently with 200,
losing the provider's error and never attempting configured fallbacks.
Detect the payload in OpenAIChatCompletionStreamingHandler.chunk_parser and
raise OpenAIError with the upstream message and status code. The existing
mid-stream gate then applies: 4xx surface directly to the client, 5xx wrap
into MidStreamFallbackError so the router can run configured fallbacks.
Fixes#25492
* fix(streaming): serialize messageless error payloads as JSON
Address review feedback: an error dict without a message field now
serializes via json.dumps instead of Python dict repr
* feat(router): add separate ITPM/OTPM deployment rate limits
Support input/output tokens per minute on deployments via enforce_model_rate_limits, with reservation, reconciliation, refund on failure, and rate-limit headers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(router): keep ITPM/OTPM diff minimal in router.py
Drop unrelated Black reformatting from router.py and types/router.py so the PR only contains functional ITPM/OTPM changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): make ITPM/OTPM limits separate and atomic
Address Greptile review on separate ITPM/OTPM deployment rate limits.
- OTPM is now reserved atomically pre-call with rollback, matching the ITPM
path, so concurrent requests can no longer overshoot the configured output
limit before reconciliation
- ITPM counts input tokens only; it no longer accumulates completion tokens,
so the input-token limit and x-ratelimit-limit-input-tokens header describe
input usage as their names imply
- _read_reservation_from_kwargs only falls back to litellm_params.metadata when
the top-level metadata channel is absent, so production requests carrying a
litellm_params.metadata dict still reconcile and refund their reservation
Adds regression tests for OTPM atomicity under concurrency, input-only ITPM
enforcement, and reservation lookup when litellm_params.metadata is present.
* fix(router): subtract input tokens only from remaining-input-tokens header
The in-flight replay for x-ratelimit-remaining-input-tokens subtracted total
tokens (input + output) instead of input tokens only, so clients saw remaining
input quota understated by the completion token count on every response. Now
consistent with the input-only ITPM counter.
* fix(router): make itpm/otpm vs tpm/rpm precedence explicit
When a deployment configures itpm/otpm alongside tpm/rpm, the io-token path
takes over and the tpm/rpm limits are not enforced. Log a warning the first
time such a conflicting deployment is seen so the supersession is not silent,
and document the mutual exclusivity.
Post-call reconciliation now only trues up a counter that was actually
reserved against, so the itpm/otpm keys are no longer incremented for
deployments that never configured that limit.
* fix(router): track actual io-token usage on the reservation-minute key
Post-call reconciliation now keys off the exact cache key stashed at pre-call
time rather than one recomputed from the response-time minute. This fixes two
issues: a request whose pre-call estimate was 0 now still writes its actual
billable input to the ITPM counter (previously it was skipped, leaving the
limit unenforceable for that request), and a call that finishes in a later
minute reconciles against the minute it reserved against instead of pushing a
negative delta into the next minute. Counters are only touched when their
limit is configured.
* fix(router): run io-token reconciliation before the model_id guard
async_log_success_event gated IO reconciliation behind the model_id guard that
only the TPM tracking path needs. Since reconciliation works entirely from the
cache keys stashed in kwargs, a success event whose standard_logging_object
lacks model_id would skip reconciliation and leave the reservation on the
counter until the TTL expired, wasting quota. Route the IO path first.
* fix(router): don't replay in-flight delta for itpm/otpm headers
For ITPM/OTPM model groups the counter is incremented at reservation time
(pre-call), so the remaining values returned by get_remaining_model_group_usage
already account for the current request. Replaying the in-flight delta on top
double-counted it and understated x-ratelimit-remaining-input/output-tokens by
up to max_tokens on every response. Skip the delta for io-token groups; the
legacy TPM/RPM replay path is unchanged.
* fix(router): clear io-token reservation after reconcile/refund
async_io_token_refund_failure and async_io_token_reconcile_success now clear
the stashed reservation keys from the request metadata once done. Otherwise, on
a model group mixing IO-limited and non-IO deployments, a failed IO call that
retries on a non-IO fallback left the stale sentinel in the shared request
metadata; the fallback's success handler would divert into IO reconciliation
against the already-refunded key, driving the ITPM counter negative and
skipping the non-IO deployment's TPM tracking.
* fix(router): tidy reservation channel lookup and header guard
Consolidate the reservation channel lookup into a single ordered helper shared
by read and clear, so top-level metadata always wins over litellm_params
metadata without the tangled per-iteration fallback.
Also stop gating the router rate-limit header block on the presence of
x-ratelimit-remaining-input/output-tokens. That block only emits those headers
for ITPM/OTPM groups; for a non-IO group backed by a provider that natively
returns input/output token headers, the extra conditions suppressed the
router's own remaining-tokens/requests headers.
* fix(router): strip client-supplied io-token reservation keys
The reservation sentinels (_litellm_itpm_reserved, _litellm_itpm_cache_key,
and the otpm equivalents) are server-only, but metadata is caller-controlled on
proxy requests. An authenticated caller could forge these fields with an
arbitrary cache key so the post-call reconcile/refund path would decrement any
deployment's ITPM/OTPM counter and let it exceed the configured limit. Strip
the reserved keys from the request metadata in set_io_token_rate_limit_request_kwargs,
which runs before the router stashes its own reservation, so only a genuine
server-side reservation is ever read post-call.
* fix(router): track TPM routing load for io-limited deployments
deployment_callback_on_success early-returned for any deployment with itpm/otpm
set, so its total-token usage never landed in the router's TPM routing counter.
TPM-aware routing strategies then saw 0 load for IO deployments and over-routed
to them in mixed model groups. Only skip tracking when neither tpm/rpm nor
itpm/otpm are configured; itpm/otpm enforcement still runs separately in
ModelRateLimitingCheck, so the routing counter and the enforcement counters
stay independent.
* fix(router): expose standard tpm/rpm headers for io-limited groups
get_remaining_model_group_usage returned early for ITPM/OTPM groups, so a group
that also set tpm/rpm never emitted x-ratelimit-remaining-tokens / -requests;
clients and prometheus gauges reading those saw no data. Build both header sets
instead of returning early.
Also simplify the in-flight header replay: only the tpm/rpm counters are
incremented post-response, so the delta now adjusts just those. The itpm/otpm
counters are incremented at reservation time (pre-call), so the input/output
token headers already reflect the request and are left untouched - which
removes the need for the separate io-group special case.
* fix(router): roll back ITPM on any OTPM reservation error; dedup warning per instance
Two follow-ups from review. The pre-call OTPM reservation only rolled back the
ITPM reservation on a RateLimitError, so a transient cache error while reserving
OTPM left the ITPM counter inflated until the TTL expired; catch any exception,
release the ITPM reservation, then re-raise.
Replace the module-level lru_cache warn-once (caching a logging side effect,
which never re-warns in a long-lived process) with an instance-scoped set of
already-warned deployment ids on ModelRateLimitingCheck.
* fix(router): always clear reservation stash on reconcile; don't collapse id-less warning dedup
Clear the reservation in a finally block so a mid-reconciliation cache error
still removes the stash and a duplicate success event can't re-process it.
Dedup the itpm/otpm-vs-tpm/rpm conflict warning per real deployment id; a
deployment with no id no longer collapses every id-less deployment onto the
str(None) key (which would suppress all but the first warning).
* fix(router): skip io reservation when deployment can't be keyed
_get_cache_keys returned a shared 'global_router:None:None:...' key when a
deployment was missing model_info.id or litellm_params.model, so misconfigured
deployments could share one rate-limit bucket. Return None in that case and
skip io reservation for the request.
* fix(router): honor explicit max_tokens=0 in io reservation
_resolve_max_tokens used 'max_tokens or max_completion_tokens', so an explicit
max_tokens=0 fell through to the model default. Only fall back to
max_completion_tokens when max_tokens is absent.
* fix(ci): satisfy lint budget, router coverage, and dashboard schema sync
- Modernize the new itpm/otpm module's type hints to PEP 585 lowercase
generics (Dict/Tuple/List -> dict/tuple/list) to clear the added UP006
violations; ratchet ruff-strict-budget.json's UP006 ceiling down to match.
- Replace three try/except Exception blocks that must stay broad by design
(token_counter and litellm.get_model_info raise untyped exceptions, and an
io-token refund failure must never break the logging pipeline) with
contextlib.suppress(Exception), matching the codebase's existing resolution
for this exact BLE001 pattern.
- Add direct unit tests for get_model_group_io_token_usage (multi-deployment
aggregation and the empty-model-list case) in test_router_helper_utils.py,
satisfying the router function-coverage check.
- Regenerate the dashboard's schema.d.ts so the new itpm/otpm fields on
GenericLiteLLMParams and ModelGroupInfo are reflected in the OpenAPI types.
* fix: enforce io token rate limits consistently
* fix: honor zero max tokens in otpm reservation
* fix(lint): fix UP007 violation and resync ruff-strict-budget.json to base
Convert Union[_Span, Any] to _Span | Any (safe on this repo's Python >=3.10
floor) to clear the new UP007 violation from the TYPE_CHECKING-gated Span
alias.
The previously committed ruff-strict-budget.json ratcheted UP006 down from a
stale base; litellm_internal_staging has since tightened that same ceiling
further on its own. Reset the file to the current base's committed values and
re-ratchet from there so the budget only ever moves down relative to the
actual merge-base, never against a stale snapshot.
* fix(router): attach ITPM/OTPM headers on dict responses and harden reservation
Strip itpm/otpm from provider kwargs, ensure messages are available for ITPM
estimation, honor max_output_tokens on /v1/responses, and propagate rate-limit
headers through /v1/messages dict responses via _hidden_params.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): attach ITPM/OTPM headers to streaming /v1/messages responses
Wrap bare async iterators in HiddenParamsAsyncIteratorWrapper so
set_response_headers can attach rate-limit headers to streaming Anthropic
messages responses that lack a _hidden_params slot.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: ruff format add_retry_fallback_headers.py
Fix CI ruff format check failure on get_hidden_params_dict call site.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(router): extract set_response_headers helpers to fix C901 budget
Move header-attachment logic into add_retry_fallback_headers helpers so
set_response_headers stays under the strict complexity ceiling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: keep IO token reservation when response usage is missing
Missing usage was reconciled as zero and fully refunded the pre-call
reservation, allowing limit bypass on repeated successful calls. Only
adjust counters when usage is resolved from the response or standard
logging fields; otherwise keep the reservation until TTL expires.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: enforce RPM/TPM alongside IO-token limits on mixed deployments
Deployments with both itpm/otpm and tpm/rpm previously returned after the
IO reservation and skipped RPM/TPM checks. Run both paths and refund the
IO reservation only when RPM/TPM rejects after a successful reservation.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: track TPM usage on success for mixed IO+TPM deployments
The early return after IO-token reconciliation in log_success_event and
async_log_success_event skipped the TPM counter increment, so the tpm_key
the pre-call check reads was never written and tpm_limit was never
actually enforced on deployments that also configure itpm/otpm.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: treat total-only usage as unresolved in IO-token reconcile
usage/standard_logging_object entries carrying only total_tokens (no
prompt/completion or input/output breakdown) were treated as resolved
usage, resolving to (0, 0) and refunding the full reservation. Both
_usage_is_present and the standard_logging_object fallback now require an
actual input/output breakdown before reconciling, keeping the reservation
otherwise.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: reserve minimal token when input/output estimation fails
_reservation_value(0, limit) reserved the entire limit whenever token
estimation failed (empty/unsupported input, tokenizer error), letting one
such request claim the whole bucket and 429 every concurrent request to
the deployment until it completed. Reserve 1 token instead so estimation
failures no longer serialize traffic.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: refund IO reservation synchronously before retry deployment pick
On retry, set_io_token_rate_limit_request_kwargs clears reservation
sentinels from the shared kwargs dict before a background failure handler
can refund them, stranding the counter until TTL. Refund and clear any
stale reservation in _update_kwargs_with_deployment before stripping
sentinels for the next attempt.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(io_token_rate_limit_check): use model-specific tokenizer for ITPM estimate; document sync-refund Redis ceiling
Pass the deployment litellm_params.model to token_counter so it uses the
model's native tokenizer instead of the generic fallback, narrowing the
reservation over/under-estimate window between pre-call and post-call
reconcile.
Add a ponytail: comment to refund_stale_reservation_before_retry explaining
the known ceiling: the synchronous DualCache.increment_cache issues a
blocking Redis INCR when a Redis backend is configured. This only fires on
streaming mid-stream retries (non-streaming failures await their failure
handler before the retry picks a new deployment, leaving no sentinels to
refund). Upgrade path: make _update_kwargs_with_deployment async.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The group_by=team branch queried spend for every team in the date range
regardless of team_id; team_id was only honored in the separate branch that
also required a customer_id. Route the team report through a new
get_spend_by_team helper (sibling of get_spend_by_team_and_customer) that binds
team_id as an optional $3 predicate, so a provided team_id narrows the result
to that team and a null team_id still returns every team.
* ci: gate CircleCI jobs on changed paths
Every CircleCI job used to run on every PR. Now each job starts with a
lightweight `skip_if_unrelated_changes` step that inspects the PR diff and
halts the job as successful when nothing relevant changed. Docs-only PRs
(*.md, *.mdx, docs/) run nothing, UI-only PRs (ui/) run just the frontend
jobs, and any backend change still runs both the backend and frontend jobs.
The decision logic lives in .circleci/scripts/classify_changes.sh (pure,
reads the changed-file list on stdin) so it can be unit tested, while
path_filter.sh handles the git plumbing and fails open (runs the job) on
any uncertainty such as a missing merge base or a non-PR pipeline. Halting
via `circleci-agent step halt` keeps the job green, so required status
checks are never left pending. The Windows smoke job is intentionally left
ungated to avoid cross-platform shell fragility
* fix(ci): keep path filter fail-open when classifier errors
Guard the classify_changes.sh invocation with `|| run_full` so a broken or
non-zero classifier runs the job instead of falling through to a silent
halt, and mark the advisory logging pipe best-effort with `|| true`. Add
path_filter.sh regression tests covering the docs-only halt, backend run,
non-PR fail-open, and classifier-failure fail-open paths
The test posted /config/update, slept a fixed 20s, then fired a single chat request with no readiness check or retry. When the single-process proxy was momentarily not accepting connections in that window, the request failed with a bare openai.APIConnectionError and took the whole job down, since the suite runs against one shared container with pytest -x
Gate the chat request behind a /health/liveliness poll, retry it on connection errors only so real HTTP errors and the Langfuse assertion still fail the test, close the previously leaked aiohttp session, and target 127.0.0.1 instead of the 0.0.0.0 bind address. In CI, give the proxy container --restart on-failure so an intermittent crash recovers instead of leaving the port dead for the rest of the run
* chore: add latest model rule to CLAUDE.md
* chore: correct grammar mistake
* chore: make the rule more concise
* chore: replace rule instead
* chore: revise wording to override memories, etc.
* chore: slightly adjust wording to be more precise
* fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects
spend_tracking/spend_e2e_client.py imported SpendTagsResponse and
TagSpend from models, but neither was ever defined, so importing the
client raised ImportError and pytest aborted collection for the whole
e2e session. The tag-spend tests had never run.
Model /spend/tags as it actually answers: a bare array of per-tag
aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the
existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field
that also wouldn't match the array shape; it now reads .root, matching
how spend_logs consumes its RootModel.
* test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction
Adds regression nets and gap-surfacing tests:
A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the
DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document
that reasoning_effort='none' and thinking type='disabled' are silently dropped
(LIT-3686 / GH #27453)
A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py):
parametrized regression net asserting real completion content, not just a 200,
across the configured providers for /chat/completions and /responses (GH #28991)
A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is
honored and prompt-cache read tokens grow on a repeated cacheable prefix
A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call
rate limiter runs, then asserts no unattributed spend row is left behind by the
internal input-file retrieval (LIT-3266)
A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct
key_alias and asserts each alias gets its own labeled series on /metrics
A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict)
regression proving eviction must not close an httpx client still held by an
in-flight caller (LIT-3221 / GH #13034)
Extends tests/e2e/models.py with the typed request and response fields these
tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage
fields, spend-log api_key)
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(e2e): drop unused litellm-regression-tests submodule
The e2e suite migrated the regression cases into this repo; nothing
imports the submodule at runtime (only a provenance comment references
it), so the .gitmodules entry and gitlink pointing at a personal repo
would just make upstream CI init a submodule it never uses. Remove both
to keep the change test-only.
* test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage
The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client
close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread
cleanup vs shared client teardown) and belongs in its own PR, not this e2e
coverage PR, so revert the file to its base state.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(e2e): migrate access-control and inference-endpoint regression tests
Move the access-control and non-chat inference-endpoint cases from litellm-regression-tests onto the shared e2e harness so a regression in either fails here first
access_control/ asserts the gateway's authorization and error-shape contract: a key limited to one model is denied 403 (key_model_access_denied) when it calls another, a key scoped to allowed_routes=["llm_api_routes"] is forbidden 403 from a management route, and an unknown model is rejected 400 before any provider is called. The source asserted 401 for the disallowed-model case against an older proxy; the live contract is now a 403, so the guard tracks current behavior
llm_translation/ gains one file per non-chat inference endpoint (/v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, /v1/images/generations). Each test registers the deployment it needs through /model/new, drives real provider traffic, asserts the parsed body carries real content instead of just a 200, then deletes the model on teardown, so nothing is hardcoded into the gateway config
* Update endpoints_client.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(bedrock): emit SSE error event when invoke Messages stream ends without message_stop
* fix(bedrock): tighten stream-terminal detection to avoid false positives and double errors
The bytes branch of _is_message_stop_chunk used a plain substring match,
so a content_block_delta whose partial_json contained the literal text
message_stop would look like a real terminal event and suppress the
synthetic incomplete-stream error. Match the SSE event header line
instead.
Also treat a provider-emitted error event as terminal so a stream that
ends with an upstream error is not followed by a second, contradictory
synthetic incomplete-stream error.
* test(bedrock): lock in that the synthetic truncation error event is excluded from logged chunks
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(anthropic_messages): forward provider response headers on streaming /v1/messages responses
* fix(anthropic_messages): forward aclose to inner streaming iterator
* fix(anthropic_messages): forward aclose through the streaming response wrapper
The proxy's streaming cleanup closes the handler's return value via
hasattr(response, "aclose"); the new wrapper hid the upstream
generator's aclose, so provider connections could linger on client
disconnect. The wrapper now delegates aclose to the wrapped stream and
AgenticAnthropicStreamingIterator closes its inner and follow-up
streams. Also adds test coverage for the agentic streaming branch
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(headroom guardrail): log real token/compression stats instead of "allow"
The headroom guardrail fetched tokens_before/tokens_after/compression_ratio
from Headroom's /v1/compress response but only surfaced them via a debug-level
log line, so spend_logs.guardrail_information showed guardrail_response:
"allow" with no way to tell whether compression actually ran or by how much.
_call_compress now returns the token/compression stats alongside the
compressed messages and success flag, and apply_guardrail logs them via
add_standard_logging_guardrail_information_to_request_data when compression
succeeds. Raw message content is intentionally excluded from what's logged -
only token counts, compression ratio, and applied transform names.
* fix(ci): apply ruff format to headroom.py
* fix(review): remove comment per repo's no-comments-unless-asked convention
Addresses codex review feedback - CLAUDE.md says not to add comments
unless explicitly asked; the sensitive-logging guarantee is already
expressed by the stats dict only pulling specific keys, not messages.
* fix: zero out crash-class basedpyright rules across litellm/
* feat(lint): add LIT009 banning inert type: ignore comments
* docs: require bracketed rule and reason on every suppression
* chore(lint): ratchet budgets down and zero crash-class pyright limits
* fix: narrow auto router routelayer through a local before calling
* test: add regression tests for crash-class fixes
* fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex
* feat(mcp): add entra_obo profile to the token_exchange (OBO) arm
Microsoft Entra On-Behalf-Of uses the RFC 7523 jwt-bearer grant rather than RFC 8693, so the existing token_exchange arm cannot mint tokens against Entra. This adds an entra_obo profile on TokenExchangeConfig that switches the request form to Entra's jwt-bearer OBO dialect (the inbound token as assertion, the target resource carried in scope, and requested_token_use=on_behalf_of), while reusing the shared caching, single-flight, TTL, and fail-closed machinery. The exchanger is renamed from Rfc8693TokenExchanger to OboTokenExchanger since it now serves both dialects
The profile is threaded through the config-load path, the DB credentials blob, and the MCPCredentials request schema, so an operator can select it from YAML or the management API. It rides in the existing credentials JSON blob, so there is no new auth_type and no DB migration
Resolves LIT-4163
* feat(mcp): propagate the Entra Conditional Access step-up challenge on the OBO 401
An entra_obo exchange that the IdP rejects for Conditional Access returns a 4xx with
error=interaction_required and a claims blob the client must satisfy to step up. The arm
dropped both and emitted a static RFC 9728 challenge, so a CA-protected Entra upstream was
unreachable through the gateway. The provider now reads the RFC 6749 error code and the claims
string off the rejection body (error_description is still never carried; it can leak IdP
internals), threads them through SubjectTokenRejected -> CredError.unauthorized, and the
challenge builder folds them into WWW-Authenticate: the machine error only when it is a plain
OAuth token (guards against header injection from a hostile body) and the claims base64-encoded
in a claims parameter, the convention MSAL-family clients decode. With neither field the header
is byte-identical to the static challenge. The multi-server aggregate still absorbs a
step-up 401 to an empty listing; only single-server routes surface it
* fix(mcp): use error=insufficient_claims for the Entra step-up challenge
Per Microsoft's claims-challenge format, a WWW-Authenticate carrying a claims challenge must set
error=insufficient_claims (the value MSAL-family clients key on to recognize the challenge and
replay the claims), not the raw token-endpoint code. The challenge now sets insufficient_claims
whenever a claims blob is present and keeps invalid_token otherwise, with a step-up-accurate
error_description in the claims case. The presence of claims now drives the error value, so the
raw oauth_error no longer needs threading from the provider through CredError to the edge; that
plumbing is removed (the provider still reads the error code for its gateway-fault classification).
Both the error value and the base64 claims are fixed-alphabet, so nothing from the IdP body reaches
the header unescaped. Cross-checked field-by-field against Microsoft Learn; a bogus jwt-bearer OBO
POST to the real login.microsoftonline.com/common endpoint confirmed Entra recognizes the grant and
returns the error shape the parser reads. Follow-up: also emit authorization_uri alongside
resource_metadata for strict non-MCP MSAL clients (RFC 9728 resource_metadata already serves MCP)
* fix(mcp): filter blank scopes on the config-load path so entra_obo fails closed
The DB-build path already runs YAML/DB scopes through _extract_scopes (which drops blanks), but the
config-load path read server_config["scopes"] raw. A YAML `scopes: [""]` therefore reached the
exchanger as a ("",) tuple: non-empty, so the entra_obo `not config.scopes` precondition skipped its
fail-closed misconfigured path and the form builder POSTed an empty scope to the IdP instead of
failing before any network call. Config-load now filters blanks the same way, so an all-blank list
normalizes to None and the precondition fails closed. Regression test loads a blank-scope entra_obo
server and asserts the exchange returns misconfigured without POSTing
* feat(prometheus): add api_provider label to token, latency, request and cache metrics
The token (input/output/total), latency (llm_api, time_to_first_token,
request_total, request_queue_time), proxy request (total/failed) and cache
metrics were emitted from the same call sites as litellm_spend_metric and
litellm_requests_metric, which already carry api_provider, yet these were
missing it. That left no way to break tokens, latency, request counts or cache
hits down by upstream provider even though the provider is already on the
payload as custom_llm_provider.
Add api_provider to each metric's label allow-list. The success path already
populates enum_values.api_provider from standard_logging_payload, so those
metrics emit it with no further plumbing. The cache label is added to the
shared _cache_metric_labels list, so alongside litellm_cache_hits_metric and
litellm_cache_misses_metric it also covers litellm_cached_tokens_metric and the
provider prompt-cache read/creation token metrics; the label-presence test
asserts all of them. For the client-side failure path, where a deployment may
not have been resolved, derive it best-effort from
litellm_params.custom_llm_provider, a partial standard_logging_object, or
inference from the requested model name via litellm.get_llm_provider, falling
back to empty rather than guessing.
Resolves LIT-4178
* fix(prometheus): satisfy ruff BLE001 budget and update enterprise label assertions
- Suppress the strict-rule BLE001 budget breach with a justified noqa;
the broad except in the failure-path provider extraction is
intentional defense-in-depth (covered by
test_extract_api_provider_swallows_unknown_model_but_logs_unexpected_errors),
not dead code to delete
- Update tests/enterprise assertions for litellm_tokens_metric,
litellm_input_tokens_metric, litellm_output_tokens_metric, the three
latency metrics, and the proxy request counters to expect the new
api_provider label, matching what litellm_mapped_enterprise_tests
caught in CI
---------
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
The CircleCI proxy containers passed --detailed_debug, and litellm's log
level defaults to DEBUG when LITELLM_LOG is unset, so CI produced very
verbose debug output for no reason. Drop --detailed_debug and set
LITELLM_LOG=ERROR on the proxy containers so real failures still surface
without the debug noise
chunk_processor now reads response.status_code to gate end-of-stream
success logging. These mocks used AsyncMock(spec=httpx.Response), which
spec's against the class and doesn't expose status_code since it's an
instance attribute, not a class attribute, so accessing it raised
AttributeError. Sets status_code=200 explicitly on the success-path mocks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): keep serving reads from the read replica when the primary DB is down at startup
RoutingPrismaWrapper.connect() connected the writer first and let a writer
failure propagate, so a proxy that started during a primary outage ended up
with no Prisma client at all (startup swallows the error under
allow_requests_on_db_unavailable): DB-stored models never loaded and every
inference request failed with 400 Invalid model name, even with a healthy
DATABASE_URL_READ_REPLICA. Workers recycled via MAX_REQUESTS_BEFORE_RESTART
hit this mid-outage and stayed broken for the rest of the outage.
connect() now degrades on a writer-only failure: reads (key auth, DB-stored
model loads) are served by the reader, writes fail at call time, and the DB
health watchdog keeps retrying the writer reconnect, which clears the
degraded flag once the primary recovers. A full outage (both sides down)
still raises as before.
Resolves LIT-4159
* fix(proxy): clear degraded-writer flag when the reconnect probe finds the writer already healthy
The direct-reconnect path returns early when the writer probe succeeds
(engine already reconnected by another path, e.g. an IAM token refresh),
skipping recreate_prisma_client, which was the only runtime path clearing
_writer_unavailable. The stale flag made the watchdog fire reconnect
attempts against a healthy writer on every cooldown cycle until restart.
Clear the flag in the early-return branch and cover it with a regression
test that fails without the change
Two bugs from the upstream-error fixes: the success handler has no
status-code awareness, so removing raise_for_status() left it firing for
every upstream 4xx/5xx too, meaning the new failure hook and the existing
success handler both logged the same request (corrupting SpendLogs/cost
tracking). Separately, the failure hook was passed the raw
httpx.HTTPStatusError, which ProxyLogging's alerting only excludes
HTTPException/ProxyException from, so a normal upstream 403 would trigger a
"High" severity llm_exceptions alert. Gates the success handler (both
non-streaming and end-of-stream) to status_code < 400, and reports upstream
failures to post_call_failure_hook as an HTTPException instead of the raw
httpx error, matching how auth/rate-limit errors are already excluded from
alerting.
Co-authored-by: Cursor <cursoragent@cursor.com>
Pass transcription_cost through additional_costs so cost_breakdown's
input_cost + output_cost + additional_costs sums to total_cost instead
of silently folding it into total_cost only.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool
The advisor_20260301 interceptor honored a caller-supplied api_base once
allow_client_side_credentials was enabled, even without a caller-supplied
api_key. AnthropicModelInfo.get_auth_header() then fell back to the proxy's
own ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, so the server's real credentials
plus the conversation history got sent to a caller-chosen destination
_resolve_advisor_credentials() now only honors api_base alongside a
non-empty caller-supplied api_key, requires the https scheme, and validates
api_base via validate_url() before use, mirroring check_complete_credentials
in auth_utils.py. https is required because validate_url only DNS-pins the
connection for http; for https with TLS verification on it returns the URL
unchanged and relies on certificate validation to block DNS rebinding
* fix(anthropic): also reject advisor api_base when ssl_verify is disabled
validate_url only DNS-pins the connection for http, or for https with
litellm.ssl_verify disabled; the previous https-only check missed the
ssl_verify=False case, where validate_url's rewritten URL was still being
discarded, per Greptile's review of this PR. Reject api_base outright when
ssl_verify is False so the discarded rewrite can no longer matter
* fix(policies): reject non-existent team/key/model scope entries on attachment create
Creating a policy attachment accepted arbitrary team, key, and model values with
no validation, so a typo'd or non-existent team was silently persisted (LIT-4199).
The create endpoint now rejects a concrete (non-wildcard) team, key, or model that
does not resolve to a real entity, wiring the previously-dead PolicyValidator
existence checks and reusing RouteChecks._is_wildcard_pattern so validation agrees
with request-time matching, where only a trailing "*" is a wildcard. Wildcard
patterns are still allowed through since they may match zero entities today and
more later, and tags stay free-form. The Admin UI's Teams field validates the same
rule for immediate feedback when its team list has loaded, deferring to the backend
otherwise.
* style(policies): use builtin list generics and | None in scope validator
Keeps the new find_invalid_scope_entries signature off the UP006/UP045 strict
ruff budgets instead of copying the surrounding legacy typing.List/Optional idiom.
* fix(policies): separate multiple attachment scope errors with ' | '
Addresses Greptile review: joining per-entry validation messages with a bare
space read as one run-on sentence; ' | ' makes the multi-error 400 detail easier
to parse for users and programmatically.
Follow-up to 8c9878025e: returning upstream 4xx/5xx bodies unchanged also
skipped post_call_failure_hook entirely, so spend-tracking and alerting
callbacks never fired for upstream errors, and response_body was hardcoded
to None in the log payload so the actual upstream error body never reached
logging integrations. Adds a small helper that calls post_call_failure_hook
for upstream errors without altering the client-facing response, and parses
response_body unconditionally for logging while still scoping guardrails
and managed-id rewriting to status_code < 400.
Co-authored-by: Cursor <cursoragent@cursor.com>
Generic pass-through endpoints called raise_for_status() on upstream 4xx/5xx
responses and re-raised as HTTPException, which the outer handler reshaped
into a ProxyException with the upstream body stringified into error.message.
Success responses were already forwarded as-is, so failures were the only
case where passthrough wasn't actually transparent. Removes the
raise_for_status() calls for both streaming and non-streaming passthrough so
upstream status, body, and headers reach the client unchanged, while keeping
guardrails/managed-id rewriting scoped to successful responses and leaving
internal proxy failures (auth, config, network errors before any upstream
response) on the existing ProxyException path.
Co-authored-by: Cursor <cursoragent@cursor.com>
Input didn't wrap its function component in React.forwardRef, so the ref
ConversationList passes for rename auto-focus/select silently never attached
under React 18 (function components need forwardRef to receive a ref; that
requirement is dropped in React 19, but this app is on 18.3.1).
MCPEnhancedStreamingIterator only auto-executed one round of MCP tool calls.
When a model retried a tool (e.g. after an error) in its follow-up turn, that
second tool call was streamed but never executed, and the response ended with
no final text. Route follow-up calls back through the same completion-check
phase as the initial response, so further tool-call rounds are handled the
same way, capped at MAX_MCP_TOOL_CALL_ROUNDS to avoid an unbounded loop.
Re-apply the maturin build backend (reverting #31470, which had temporarily
restored the pure-Python uv_build backend). maturin packages the Rust bridge
(litellm.rust_bridge._native) into the wheel; the loader already falls back
gracefully when the native module is absent, so pure-Python installs are
unaffected.
The earlier revert was needed because the release pipeline emitted a bare
cp312 linux_x86_64 wheel that PyPI rejects. That is resolved on the pipeline
side: it now branches on the build backend and, for maturin, builds proper
manylinux_2_28 wheels (x86_64 + aarch64) and validates each wheel carries the
native module.
The [tool.maturin] include for litellm/proxy/_experimental/out/** is sdist
coverage for the Admin UI bundle. maturin's include overrides .gitignore for
the sdist but not the wheel; the committed bundle stays tracked and un-ignored,
so it flows into both the wheel (maturin's source walk) and the sdist as-is.
Coordinates with the release pipeline change that builds the Admin UI from
source and gates the built wheel/sdist on the bundle being present; that should
land first so the pipeline can build and verify a maturin UI wheel.
* feat(mcp): discover the OBO token endpoint via RFC 9728 to RFC 8414 (no IdP guessing)
An oauth2_token_exchange server can now have its token endpoint discovered the
same way the oauth2 (authorization_code) flow already does, instead of always
requiring token_exchange_endpoint/token_url to be configured by hand. The
existing _descovery_metadata chain (RFC 9728 protected-resource metadata ->
RFC 8414 authorization-server metadata -> token_endpoint, SSRF-guarded via
async_safe_get) is reused; both the config-load and DB-build paths gate on a new
_obo_needs_endpoint_discovery so discovery runs only when no endpoint is
configured, and an explicitly configured endpoint still wins and skips the
round-trip. The discovered token endpoint lands on token_url, which
_token_exchange_spec already reads, so no resolver change is needed.
_resolve_oauth2_flow returns None for any non-oauth2 auth_type, so a discovered
token_url on an OBO server is never mis-inferred as the M2M client_credentials
flow.
Discovery for OBO is authoritative only: the resolution order is explicitly
configured endpoint, then RFC 9728 -> RFC 8414 advertisement, then fail closed
(412, on the parent commit). The gateway never guesses the IdP. _descovery_metadata
grows an allow_origin_fallback flag, kept True for the browser oauth2 flow (a
human sees the redirect) but set False for token_exchange so the last-resort
guess that treats the resource server's own origin as its authorization server
is skipped; a subject token is never exchanged against an inferred endpoint.
* fix(mcp): surface a failed OBO exchange at connect instead of an empty tool list
A token_exchange server whose exchange fails with a subject present used to open the MCP
session anyway and mask the failure as an empty tools/list. Single-server routes now run
the exchange preemptively at the transport edge, where a rejected subject raises the RFC
9728 challenge and a gateway fault its public status; the multi-server aggregate keeps
absorbing per-server auth failures. The exchanger caches the preflight result, so the
session's list/call reuses it with no extra IdP round-trip. Discovery now also debug-logs
the authorization server's advertised issuer, grant types, and client auth methods
* fix(mcp): persist the discovered OBO token endpoint to the DB row
A DB-backed oauth2_token_exchange server with no configured endpoint had its token_url
resolved via RFC 9728 -> RFC 8414 only on the in-memory object returned from
build_mcp_server_from_table; the row kept token_url=None, so every rebuild re-ran discovery
and a transient upstream outage during a rebuild left the server with no endpoint until the
next successful discovery. Write the discovered token_url back onto the row so the guard sees
it on the next build. Best-effort and scoped to DB servers: config servers already persist
in-memory, and the write-back never fires from a user connect (only from add/update/reload,
all admin or system driven). Adds DB-path coverage for discovery firing when unset, skipping
when the credentials endpoint is configured, the write-back, and its negative guards
* Revert "fix(auth): deny model access for teamless keys with all-team-models (#32022)"
This reverts commit dfbbda4f19.
* revert: undo teamless all-team-models denial from PR #29746
Reverts the team_id guard in _resolve_key_models_for_auth_check and
get_key_models so teamless keys with all-team-models resolve to []
(unrestricted = all proxy models) rather than being denied.
Adds hardened regression tests across listing (get_key_models), inference
(_enforce_key_and_fallback_model_access, can_key_call_model,
can_key_call_resolved_model), and batch (_enforce_batch_file_model_access)
paths that enforce teamless all-team-models == all-proxy-models and will
fail if anyone re-introduces a team_id guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: retrigger checks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The Router's async fallback orchestrator appended fallback structures
(fallback_model_group, fallbacks, context_window_fallbacks,
content_policy_fallbacks) and the inner fallback exception onto
original_exception.message before re-raising. That message is forwarded
verbatim by the proxy as ProxyException.message. When fallbacks are
configured as inline deployment dicts, the raw provider api_key /
aws_secret_access_key inside those dicts reached any authenticated
caller in the response body.
Route the fallback structures through a new mask_sensitive_structure
helper (reuses the existing SensitiveDataMasker), and wrap the inner
fallback exception string in the existing redact_string. Topology names
still render for debugging under the existing expose_router_debug_in_errors
opt-in; only credential values inside inline-dict fallbacks are masked.
The router's own verbose_router_logger calls that embedded the same
structures are updated alongside, so log output stays consistent with the
exception message.
Verified end-to-end against a real proxy hitting OpenAI: before, the
client response body contained the raw fallback api_key; after, with the
flag on, the api_key value is masked to a 4-char prefix while topology
names are still visible for the operator
* chore(ui): remove debug console.log statements from dashboard
Delete 463 leftover console.log/console.debug calls across 87 files in the
Admin dashboard. These logged form payloads, API responses, and render
traces into every user's browser console.
The ESLint policy already encodes the intent (no-console allows only warn
and error), so those are kept, along with the console.log = function(){}
suppression reassignments and the console.log calls that live inside
string/template literals rendered as example code snippets.
Removal used an AST codemod so only standalone console.log/console.debug
expression statements were dropped; non-statement uses (no-op chart
onValueChange props, a placeholder onClick, and a sequence-expression in
TopKeyView) were handled by hand. Ratchets the no-console lint metric from
484 to 15.
* chore(ui): drop empty blocks left after console.log removal
Greptile flagged three empty control-flow blocks (an if and an else in
chat_completion.tsx, an else in networking.tsx) left behind when their
only content was a deleted console.log. Removes those plus one more empty
if in chat_completion.tsx's catch that the review missed.
* test(ui): drop provider_info_helpers test asserting debug log
The getProviderModels debug console.log calls were removed in this PR, so
the test asserting they fire no longer applies. Remove that test and its
now-unused console.log spy; the remaining 57 tests still cover the
function's actual return-value behavior.
Establishes a real design.md/AGENTS.md for the chat UI (tokens,
component patterns, decision trees) after several rounds of hand-rolled
Tailwind shipping invisible or broken states, then audits every
component in the directory against it: raw <button>s replaced with
shadcn Button throughout, spinners replaced with Skeleton for list/table
loading states, dark-mode contrast bugs fixed (MCPAppsPanel cards were
bg-background instead of bg-card, identical to the page background in
dark mode), Badge variants and status colors aligned with the documented
semantics, and the sidebar's active-nav-item styling switched to the
purpose-built sidebar-* tokens instead of the generic accent/secondary
tokens that collapse to the same value in this theme.
Also: disables model comparison mode and multi-select in favor of a
single active model, moves the model picker from a standalone top bar
into the composer, removes the sidebar collapse toggle and the
non-functional "Search chats" entry, and renames the conversation list's
"Today" group to "Recents".
Fixes a real scroll bug: the model picker's dropdown list was
unscrollable because its container used max-height instead of an
explicit height, which doesn't count as a definite size for the
percentage-height Radix ScrollArea viewport to resolve against — so the
viewport silently expanded to full content height instead of clipping,
and scroll events fell through to the page behind it. Same latent bug
fixed in the sidebar's conversation list.
* fix(anthropic): preserve 1h cache-creation TTL breakdown across streaming usage chunks
Anthropic emits the cache-creation TTL breakdown (ephemeral 5m/1h split) only on
the message_start SSE event; the later message_delta carries the flat
cache_creation_input_tokens count but drops the nested cache_creation object.
ChunkProcessor aggregates prompt_tokens_details last-wins, so message_delta's
details (with cache_creation_token_details=None) clobbered the breakdown captured
from message_start. Cost calc then fell into the flat-rate branch of
calculate_cache_writing_cost and billed 1-hour cache writes at the 5-minute rate,
undercounting the cache-creation cost component by ~37.5% on streaming requests.
Track cache_creation_token_details with the same non-null-wins semantics already
used for the flat cache counts and stitch it back onto the final
prompt_tokens_details when the last chunk lacks it. Non-streaming was unaffected
because its usage is parsed once from the full response body.
* refactor(streaming): extract cache-creation breakdown helpers to stay within strict complexity budget
* test(streaming): cover final-chunk cache-creation breakdown path
---------
Co-authored-by: Richard Warburton <Richard.Warburton@theaccessgroup.com>
* feat(mcp): thread the caller token into tools/list discovery for token_exchange
A token_exchange (OBO) server's tools could not be discovered through the aggregator: the list path
never threaded the caller's token, so every tools/list hit the no-subject branch. v1 masked this with
its client_credentials fallback (discovery used a service token); v2 dropped that fallback, so listing
had no credential and the OBO server's tools never appeared - and an MCP client lists before it calls.
Thread the inbound subject_token into the list path the same way the call path does, gated on
auth_type oauth2_token_exchange so the caller's bearer never leaks into other modes:
_get_tools_from_server takes an oauth2_headers param, extracts the token via _extract_bearer_token, and
passes it to _create_mcp_client; server.py forwards oauth2_headers at the list call site.
authorization_code (resolves off identity plus stored token), the static/config modes, and the
background registry refresh are unaffected, and the list path's existing graceful degradation
(catch -> empty list) is preserved.
* fix(mcp): harden token_exchange OBO from the audit (strip, TTL/expires_in, subject_token_type)
- _should_strip_caller_authorization returns True for oauth2_token_exchange, so the inbound subject
token is never forwarded upstream raw - only the IdP-exchanged token is (matches authorization_code).
- _parse_expires_in accepts a JSON float / numeric-string expires_in, and _ttl_seconds caps the cache
TTL at the token's real remaining lifetime so a short-lived exchanged token is never served stale.
- to_server_spec normalizes a falsy subject_token_type to the default URN, parity with v1.
The subject/key disambiguation (never exchange the LiteLLM key; Authorization: Bearer <litellm-key>
support for /mcp) is intentionally a separate cross-cutting PR off staging, not part of this OBO work.
* fix(mcp): stop caller header bypassing OBO exchange; thread subject into prompts/resources
The per-server x-mcp-* override guard in _create_mcp_client only kept the v2 spec
for authorization_code, so a caller-supplied header silently disabled the RFC 8693
exchange on a token_exchange server and forwarded the raw bearer upstream. Extend
the guard to token_exchange so the exchange always runs and the caller cannot
substitute an arbitrary upstream credential.
prompts/list+get, resources/list+read, and resource-templates/list never threaded
the OBO subject token, so those operations failed closed (401 / empty) on a
token_exchange server. Thread the caller's bearer as the subject for those paths
too, gated on the token_exchange mode via a shared _obo_subject_token helper.
* fix(mcp): keep the OBO/authz_code resolver credential authoritative; centralize OpenAPI strip
A guardrail (e.g. MCPJWTSigner), static_headers, or any other injected Authorization could
shadow the resolver-owned credential for token_exchange / authorization_code servers, so the
upstream would receive e.g. the signer's JWT instead of the exchanged token and reject it. In
_create_mcp_client the resolver-owned credential now wins: a conflicting header is dropped and
the minted/stored token reaches upstream. No behavior change for none/passthrough/static modes,
where an injected Authorization still wins as before.
The OpenAPI/local _request_extra_headers forwarder gated its Authorization strip on
has_client_credentials only, so an OpenAPI-backed token_exchange server with
extra_headers:[Authorization] forwarded the raw subject token upstream and never exchanged. It
now uses the centralized _should_strip_caller_authorization so it matches the managed paths.
* feat(mcp): RFC 9728 challenge for token_exchange (OBO) unauthorized
OBO previously returned an opaque 401 (Bearer error="invalid_request") with no discovery
info, and any IdP exchange failure collapsed to a retryable 503. Now an OBO server behaves like
a standards-compliant OAuth resource server:
- A missing/rejected subject token returns the RFC 9728 / RFC 6750 challenge: 401 +
WWW-Authenticate: Bearer resource_metadata="...", error="invalid_token", so a spec-compliant
MCP client can discover the IdP, SSO, and retry with a fresh subject token.
- The protected-resource metadata for a token_exchange server advertises the JWT-auth issuer(s)
(JWT_ISSUER / litellm_jwtauth.issuers) as authorization_servers -- the IdP that issues and
validates the subject -- instead of the gateway.
- An IdP 4xx (subject rejected) is now a non-retryable 401 (the challenge) instead of a 503, so a
caller with a dead token re-authenticates rather than looping; 5xx/transport stays retryable 503.
* fix(mcp): emit the OBO RFC 9728 challenge preemptively so a no-subject client can discover the IdP
A token_exchange server's tools are not discoverable without a subject token (list is lenient ->
empty), and a tool-call-time 401 is wrapped into a JSON-RPC error so the WWW-Authenticate header is
lost. So a cold-start client never saw the challenge and could not start discovery. Add a
token_exchange branch to the preemptive-401: a no-subject connect to an OBO server now returns
401 + WWW-Authenticate: Bearer resource_metadata=..., error="invalid_token" at the transport level,
so a spec-compliant client discovers the IdP (the PRM advertises the JWT-auth issuer), SSOs, and
retries with a subject token. Verified live on the per-server endpoint; the with-subject connect
still proceeds (no challenge).
(Also formats two lines from earlier commits in this stack.)
* refactor(mcp): inject root_path into the OBO/OAuth challenge edge
The adapter's raise_user_oauth_challenge and raise_token_exchange_challenge
reached into os.getenv("SERVER_ROOT_PATH") via get_server_root_path(), a
hidden ambient read in a module that is meant to be a pure edge. That coupling
made the preemptive-challenge test order-dependent under xdist: a sibling test
sets SERVER_ROOT_PATH at import without cleanup, leaking the prefix into the
challenge URL and failing the exact-match assertion.
Resolve the root path at the imperative-shell call sites and pass it in
keyword-only, so both challenge builders become pure functions of their inputs.
Extract the shared resource_metadata path construction into a single
oauth_protected_resource_path helper, collapsing the duplicated prefix/name
logic the two functions carried.
Also reduce _create_mcp_client below the strict complexity ceiling by extracting
the v2 credential resolution into _resolve_v2_auth, and extract the OBO
protected-resource-metadata branch into _obo_protected_resource_response (which
shipped without coverage) so discovery can be unit-tested directly.
Tests are now hermetic: the adapter tests pass root_path as a real input rather
than monkeypatching the environment, the stale-session preemptive test asserts
structural invariants instead of the exact prefixed URL, and five new tests
cover the OBO PRM issuer branch end to end.
* feat(mcp): OBO cache-key tenant isolation, reactive 401 retry, v1-parity logs
From a pass over the OBO behavior contract. Three changes to the
token_exchange arm, none of which alters any other auth mode.
The exchanged-token cache key now folds in the caller's tenant alongside
the subject token and exchange config, so two tenants presenting the same
opaque token can never share a cache entry; cross-tenant isolation is
structural rather than incidental to subject-token uniqueness. tenant_id is
threaded from the resolver's Subject; it is keyword-only with an empty
default so the no-tenant case and the existing call sites are unchanged.
The tool-call path gains one reactive retry. When an upstream rejects the
injected token with a 401/403, the gateway invalidates the cached exchange,
re-mints once through the IdP by rebuilding the client, and retries the call
exactly once before surfacing the upstream error, so a token revoked or
rotated upstream mid-TTL self-heals without an infinite loop. It is gated
strictly to oauth2_token_exchange; passthrough, authorization_code,
client_credentials, api_key, and none keep their single-call behavior.
MCPClient.call_tool gains a raise_on_error flag (mirroring list_tools) so
the path can tell an upstream 401 apart from an ordinary tool error and
avoid re-running a non-idempotent tool on a non-auth failure.
The exchanger also emits the v1-parity log lines it had dropped (attempt
with server, endpoint and audience; success; cache hit), while never
logging the form, subject token, secret, or minted token.
* fix(mcp): fail closed with 412 when a token_exchange server has no endpoint
A true token_exchange (OBO) server must use only an explicitly configured
token endpoint; it must never guess an IdP or silently fall back to a weaker
source. Previously an OBO server with client credentials but no
token_exchange_endpoint/token_url deferred to v1, which no-op'd and let the
request connect to the upstream with no credential (an upstream 401 rather
than a clear gateway error).
Now such a server is owned by the v2 arm: _token_exchange_spec builds the spec
even when the endpoint is absent, and the exchanger fails closed with a
precondition_required error that maps to HTTP 412 before any upstream or IdP
call, with the caller's subject token never sent anywhere. A missing
client_id/secret still maps to misconfigured (500); a present-but-rejected
subject still maps to 401; an unreachable IdP still maps to 503. The no-subject
case keeps its existing 401 RFC 9728 challenge.
* feat(mcp): log a refused non-Bearer token_type in the OBO exchange
* fix(mcp): surface OBO/authorization_code list-time 401 as a challenge instead of masking it
* feat(mcp): classify RFC 6749 gateway-fault token-exchange errors as 500, not a caller 401
* test(mcp): absorb fixture uses 500 now that 401/403 are challenge-class at list time
* style(mcp): PEP 604 union in the OBO retry signature to keep the UP007 budget flat
