Commit graph

43876 commits

Author SHA1 Message Date
yuneng-jiang
4da00b88ac
Merge branch 'litellm_internal_staging' into litellm_yj_july8 2026-07-08 11:26:17 -07:00
devin-ai-integration[bot]
93c047d52e
feat(proxy): make Microsoft Graph endpoint configurable for GCC High (LIT-4282) (#32517)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-07-08 11:07:58 -07:00
ishaan-berri
ad69d6f3f9
test: emit e2e coverage lines for loki (#32513) 2026-07-08 11:06:56 -07:00
Ishaan Jaff
33aaea363c
ci: add OSS daily branch workflow 2026-07-08 10:54:13 -07:00
David Katz
c0327cded4 fix(mcp): pair token-endpoint client_secret with the same source as client_id
On re-auth against a server with a persisted DCR client, register_client_with_server
short-circuits and returns a placeholder client_secret ("dummy") that the browser
echoes back to /token. exchange_token_with_server overrode the caller's client_id
with the persisted one but still fell back to the caller's secret when the server
had none stored, so a persisted public PKCE client (which has no secret) was paired
with the literal string "dummy" and the IdP rejected the exchange with 401 on every
re-authorization; the proxy surfaced that as a 500. First connects and brand-new
servers worked because a real DCR registration ran and no placeholder existed.
Resolve the secret from the server whenever the server's client_id wins, so a
secretless public client sends no client_secret at all
2026-07-08 10:43:19 -07:00
yucheng-berri
f982b67d78
fix(proxy): harden secret name validation for external secret manager integrations (LIT-4201) (#32092)
key_alias can become the secret name used by external secret manager
integrations (HashiCorp Vault, CyberArk Conjur) when store_virtual_keys is
enabled. Add raise_if_unsafe_secret_name, a shared validation check applied
unconditionally before a secret name reaches either integration or the
/key/generate, /key/update, and /key/regenerate API boundary, independent
of the existing enable_key_alias_format_validation opt-in flag.

Also hardens the Vault URL builder to percent-encode reserved characters
in secret_name (preserving "/" and "@"), and switches the Conjur policy
body to a real YAML serializer instead of raw string interpolation.
2026-07-08 10:36:00 -07:00
Mateo Wang
c2d8a17692
test(responses): replace perma-skip azure shell e2e with offline coverage (#32444) 2026-07-08 10:01:41 -07:00
yuneng-jiang
cbc6a79972
Merge pull request #32432 from thibault-linktree/litellm_ui_session_sidebar_sort_toggle
feat(ui): sort session sidebar calls by duration or start time
2026-07-08 09:22:41 -07:00
Yassin Kortam
bfff5e8d86
fix(mcp): log MCP tool calls returning isError=true as failures (#32238)
An MCP tool call that completes with CallToolResult.isError=true correctly
returns HTTP 200 per the MCP spec, but the shared post-call logging helper
always fired async_success_handler, so the standard logging payload carried
status=success and OTel (whose _parse_error only marks ERROR on
status=failure) showed green spans for failed tools.

The helper now checks the result after async_post_mcp_tool_call_hook runs
(guardrails may flip isError there) and routes error results to the failure
path: success gates are consumed so the @client wrapper cannot enqueue a
success log, failure_handler and async_failure_handler fire with a new
MCPToolResultError carrying the tool's first text content, and
post_call_failure_hook records the failure the same way raised exceptions
already do. Raised exceptions never reach the helper, so no double failure
logging. HTTP wire behavior is unchanged

Resolves LIT-4081
2026-07-08 09:02:48 -07:00
Chenlu Ji
c62e1238d6 feat(tinyfish): surface response headers + top-level response extras
Follow-up to #31411 (superseded and merged as #31997). Two related fixes
so LiteLLM callers see what TinyFish actually returns, plus small
correctness cleanups.

## Response headers surfaced on _hidden_params

TinyFish sets useful response headers (x-request-id on every response,
retry-after and x-ratelimit-limit on 429s). Previously these were only
accessible via BaseLLMException.headers on error paths; on the success
path they were dropped entirely.

Fix: stash headers on both LiteLLM-conventional channels, matching the
pattern used by Gemini / Volcengine / Manus / ChatGPT / OpenAI-responses
providers.

- `_hidden_params["headers"]` -- raw dict from httpx, all keys lowercased.
- `_hidden_params["additional_headers"]` -- passed through
  process_response_headers, which prefixes any x-litellm-* provider
  header with `llm_provider-` so downstream LiteLLM code that trusts
  bare x-litellm-* markers can't be spoofed (values still survive
  under the prefixed key for observability).

## Top-level response extras (query, total_results, page, future fields)

transform_search_response was building a fresh SearchResponse from just
`results`, silently dropping every top-level field TinyFish's response
carries beyond `results` / `object`.

Fix: mutate parsed.results to its truncated slice and return the same
SearchResponse instance rather than reconstructing. Every field pydantic
populated during model_validate -- declared attributes AND extras
(query, total_results, page, parameter_warnings, and any future TinyFish
additions) -- survives regardless of which storage bucket holds it.
Robust against upstream schema evolution: if LiteLLM later promotes a
field from extras to declared, this code needs no change.

## Code cleanup

- List-valued custom params JSON-encoded on the wire (matching the
  existing dict handling), so callers can pass a natural Python list
  for JSON-array wire params.
- URL-encodable-params adapter accepts float in addition to
  str / int / bool; server-side rejection of a wrong-typed float now
  surfaces cleanly with `TinyFish Search:` attribution + docs link.
- Assorted comment / docstring / test-fixture hygiene (no logic changes).

## Tests

70 unit + integration tests pass locally. Live-tested against
production TinyFish with 6 diverse queries (basic / max_results /
country=US / language=ja / domain filter / fetch={"format":"html"}) --
all 6 pass every expected-behavior check.
2026-07-08 01:06:17 -07:00
Mateo Wang
cd6e8cdf23
test(realtime): record and replay websocket traffic in redis vcr cassettes (#32390)
* test(realtime): record and replay websocket traffic in redis vcr cassettes

* style(realtime): ruff-format ws-vcr harness

* fix(realtime): warn instead of silently disabling ws-vcr when the redis client cannot be built
2026-07-08 00:19:06 -07:00
Mateo Wang
684e3e1c2e
test(vertex_ai): bump local_testing vertex tests from gemini-2.5-flash to gemini-3.5-flash (#32439) 2026-07-08 00:17:55 -07:00
Thibault Serot
6d2090a21b fix(ui): reset session sort mode when drawer closes 2026-07-08 17:17:44 +10:00
Thibault Serot
5d89be551b fix(ui): fit session sort toggle inside sidebar column 2026-07-08 17:06:38 +10:00
Yassin Kortam
6f6bd45681
perf(auth): negative-cache missing user/key lookups on the request hot path (#32368) 2026-07-08 09:59:57 +03:00
tin-berri
1fb2b4aef4
fix(mcp): drop the cached per-user OAuth token when the credential row changes (#32302)
* fix(mcp): drop the cached per-user OAuth token when the credential row changes

The v2 authorization_code chain Cached(Refreshing(V2PerUserTokenStore)) caches a positive token
until its expires_at (or 300s without one), and CachedOAuthTokenStore.invalidate had no callers,
so a re-authorization or revocation wrote the DB while egress kept serving the replaced token
from the in-process cache until its TTL. LazyPerUserOAuthTokenStore now exposes invalidate,
MCPServerManager threads it to the write side, and the three credential write sites (the OAuth
callback, the Tools-tab persist endpoint, and the revoke endpoint) drop the cache entry after
the row changes. The v2 refresher's own persist stays untouched; RefreshingTokenStore already
feeds the rotated token back into the cache in the same fetch

* test(mcp): pin cache invalidation on the revoke already-gone branch

Greptile's review flagged that only the happy-path delete asserted the invalidate; a refactor
moving the call inside the try block would silently skip the cache drop when the row was
already deleted by a concurrent request while the cache still held the revoked token. The new
test fails on exactly that mutation

* test(mcp): cover invalidate on the redis-backed lazy store path

Codecov flagged the redis fast path of LazyPerUserOAuthTokenStore.invalidate as unexercised;
the existing invalidate tests only ran the no-redis chain. The new test builds the redis chain
via a fetch and asserts a subsequent invalidate reaches the same store instance without a
rebuild
2026-07-07 23:59:35 -07:00
Thibault Serot
df2d44bab1 feat(ui): sort session sidebar by duration or start time 2026-07-08 16:54:39 +10:00
Yassin Kortam
bcd52754de
feat(rate_limit): support per-tag rpm limiting on a single key (#31502)
Add a tag_rpm_limit field to virtual keys so each request tag gets its own independent RPM counter on the v3 rate limiter. A key configured with per-tag limits tracks each tag/group separately, and requests whose tag has no configured limit fall back to the key-level limit. Includes the dashboard UI to manage per-tag limits on key create and edit.

Resolves LIT-3147
2026-07-08 09:43:47 +03:00
Thibault Serot
34db5f4813 feat(ui): add start time sort toggle to session logs sidebar 2026-07-08 16:26:47 +10:00
tin-berri
d6cbf6e7e3
feat(ui): expose MCP max_concurrent_requests in server create and edit forms (#32397)
* feat(ui): expose MCP max_concurrent_requests in server create and edit forms

The proxy has enforced a per-server outbound tool-call concurrency cap
(max_concurrent_requests) across every MCP egress path since #31641, and the
management API has accepted the field on create and update all along, but the
dashboard offered no way to set it. Add an optional Max Concurrent Requests
input to the MCP server create and edit forms; it applies to every auth type
and transport, so it renders unconditionally rather than gated on auth mode.
Clearing the field on edit sends null so the stored limit is unset.

Also rebuild the per-server semaphore when the configured limit changes.
Previously the semaphore was created once per server_id and never resized, so
an edited limit only took effect after a proxy restart even though the new
value was persisted and reloaded into the registry.

* feat(ui): mark MCP max concurrent requests field label as optional

* test(ui): stop OBO create-form tests from timing out on CI

The token-exchange payload test and the Entra scope-required test filled five
text fields with user.type, which dispatches a full keystroke sequence per
character; every input event runs the antd form onValuesChange handler and
re-renders the whole CreateMCPServer tree, roughly 120 renders per test. As
the form grew the two tests reached 8s and 18s locally, which crosses the 30s
vitest timeout on slower CI containers; ui_unit_tests failed twice this way.
Switch the plain text fields to fireEvent.change (one input event per field),
matching the existing stdio test pattern. Both tests assert form output, not
keystroke behavior, and now run in about 3s each.
2026-07-07 22:47:03 -07:00
Mateo Wang
6df5e1b263
ci: skip unit test workflows when only ui or markdown files change (#32422)
* ci: skip unit test workflows when only docs or ui files change

Mirror the CircleCI backend path filter (.circleci/scripts/classify_changes.sh)
in the GitHub Actions unit test workflows by adding paths-ignore for ui/**,
docs/**, *.md and *.mdx to every test-unit-*.yml pull_request trigger

* ci: drop docs/** from unit test paths-ignore since the folder no longer exists
2026-07-07 22:10:58 -07:00
Mateo Wang
06a43d11c4
test(responses): bound azure shell tool live call at 90s and skip on provider timeout (#32424)
* ci(responses): bound azure shell tool e2e call and enforce per-test timeout

The azure variant of test_responses_api_shell_tool always makes a live
Azure call (its skip outcome means no VCR cassette is ever persisted).
When Azure held the connection instead of answering, the call sat on
litellm's 6000s responses deadline until CircleCI killed the whole job
via no_output_timeout after 15m of silence (job 2013288).

Bound the e2e call at 90s and skip on litellm.Timeout, matching the
existing InternalServerError and BadRequestError skips, and give the
llm_responses_api_testing job the same pytest-timeout guard the
llm_translation_testing job already uses so no single hung test can
consume the 15m no-output window again.

* test(responses): drop job-level pytest timeout, keep shell tool 90s bound
2026-07-07 21:58:14 -07:00
Mateo Wang
404ec7fc2e
ci(llm_responses_api_testing): bound live re-record calls and rerun timeout-only failures to stop 15m no-output kills (#32420) 2026-07-07 21:57:46 -07:00
Mateo Wang
c212c16852
ci: ratchet LIT003 budget down to current count to remove suppression slack (#32423) 2026-07-07 21:56:05 -07:00
yuneng-jiang
ec4f324482
Merge pull request #32405 from BerriAI/litellm_kraken-remove-envref-gates
fix(proxy): resolve os.environ/ refs universally in DB-sourced models
2026-07-07 21:53:56 -07:00
tin-berri
f922be32f0
fix(mcp): accept integer progressToken in host progress capture (#32402) 2026-07-07 21:46:36 -07:00
tin-berri
7cc660866a
fix(ui/mcp): do not reset in-flight OAuth resume when create modal mounts closed (#32416) 2026-07-07 21:42:08 -07:00
Mateo Wang
b2e2a38bc0
fix(passthrough): stream non-sse passthrough responses instead of buffering in memory (#32386)
* fix(passthrough): stream non-sse passthrough responses instead of buffering in memory

Non-SSE passthrough responses were fully read into proxy memory (content = await response.aread()) before the first byte reached the client. For large non-JSON bodies such as Anthropic batch results jsonl files this ballooned proxy RSS to a multiple of the file size and produced near-total TTFB dead air, letting intermediaries kill the silent connection and truncate the download.

The upstream request is now sent with httpx stream semantics and the buffering decision is made from the response headers: application/json (and +json) bodies plus upstream errors keep the buffered behavior since spend logging, guardrails and managed-id rewriting inspect them, while every other 2xx body is relayed as a StreamingResponse that iterates upstream bytes without accumulating them, preserving status code and headers (including x-litellm-*) and firing the success-handler logging with response_body=None once the stream completes.

* fix(passthrough): log client disconnects mid-stream and derive test client cache key from production code

* test(passthrough): intercept AsyncClient.send in legacy passthrough tests and assert final wire params

* test(passthrough): fail with a clear assert when the passthrough client cache scan misses
2026-07-07 20:51:15 -07:00
Mateo Wang
734fd29e00
fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) (#32389)
* fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056)

* test(register_model): use a triple provider prefix as the unresolvable-key fixture

get_model_info now resolves bedrock/bedrock/... like a routing prefix, so the
double-prefix fixture stopped exercising the register_model fallback path.
Lock the new double-prefix resolution in as a model-info regression test
2026-07-07 20:49:03 -07:00
thibault-linktree
4b0ac8b352
fix(responses): make response-id encoding idempotent to prevent MCP gateway double-encoding previous_response_id (#32034) 2026-07-07 19:47:11 -07:00
Tin
732832d342 fix(mcp): bind client-forwarded Authorization to a single upstream
Some checks failed
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
In a listing fan-out over a scope containing more than one server that
consumes the caller's Authorization (true_passthrough, oauth_delegate,
or the legacy delegate/passthrough shapes), the request-wide bearer is
now withheld from the new modes instead of being replayed against every
upstream (RFC 9700 cross-resource replay). Explicitly-addressed
operations (tool call, get_prompt, read_resource, single-server routes)
keep forwarding it.

Multi-server aggregates use the per-server x-mcp-{alias}-authorization
header instead: its value now feeds the passthrough resolver arm as the
inbound token and wins over the request-wide header, binding one token
to one server.
2026-07-07 19:46:23 -07:00
ishaan-berri
3ea27bd64c
test: add e2e coverage module metrics (#32403)
* Split LLM e2e coverage modules

* Add e2e coverage dashboard metrics

* Remove dashboard brief from e2e coverage PR
2026-07-07 19:38:56 -07:00
Mateo Wang
07aeaa17a0
fix(passthrough): stop request params from clobbering merged target query params (#32404)
* fix(passthrough): stop request params from clobbering merged target query params

* fix(passthrough): rewrite managed ids in query params before folding them into the URL
2026-07-07 18:56:56 -07:00
Mateo Wang
ae0d84116a
ci(server-root-path): retry npm/playwright installs and disable matrix fail-fast (#32406) 2026-07-07 18:56:12 -07:00
Mateo Wang
bd6cabee83
fix(model_prices): add gpt-realtime-2.1 models with regional processing uplift (#32387)
* fix(model_prices): add gpt-realtime-2.1 models with regional processing uplift

* fix(model_prices): add cache_read_input_audio_token_cost to gpt-realtime-2.1
2026-07-07 18:56:00 -07:00
mubashir1osmani
a05a1eef94
fix(ui): scope key models dropdown options to the key's team (#32382)
* fix(ui): scope key models dropdown options to the key's team

A teamless key no longer offers the all-team-models option in the create and
edit forms; the backend expands that sentinel to the full proxy model list when
no team is attached, which is rarely what the user intended. A team key no
longer surfaces the all-proxy-models sentinel that leaks in verbatim when the
team's own model list carries it; the dropdown keeps All Team Models plus the
team's individual models.

Adds browser coverage to the management e2e suite: playwright (an optional
dependency behind importorskip) drives the proxy-served dashboard at /ui,
asserts the dropdown options a real user sees for teamless and team keys on
both create and edit, and walks the create modal end to end, reading the
persisted key back through /key/info.

* fix(ui): offer all-proxy-models on teamless keys in the models dropdown

A teamless key has no team allowlist to inherit, so the dropdown now offers All
Proxy Models in place of All Team Models on both the create and edit forms, with
the same exclusive-selection handling. Component and browser e2e tests updated to
pin the swapped option pair; the teamless create case now also walks the modal end
to end and reads the persisted key back through /key/info.

* test(ui): update no-team key creation spec to pick All Proxy Models

The create modal no longer offers All Team Models without a team; the teamless
path now offers All Proxy Models, which is what this spec exercises

* fix(ui): gate All Team Models on the team object being loaded

When a key has a team_id but the teams prop does not yet include the matching team, availableModels stays empty and the models dropdown rendered All Team Models on its own with nothing to compare against. Gate the option on the team object being present so it only appears once team models are known, and add a regression test for the loading state

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): filter all-proxy-models from teamless model fetch in key edit form

The teamless fetch path stored modelAvailableCall results without excludeProxyWideSentinel, so an all-proxy-models entry in the response rendered a second option colliding with the hardcoded All Proxy Models sentinel. Apply the same filter used on the team path and add a regression test asserting the sentinel option is not duplicated

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-07 18:54:19 -07:00
yucheng-berri
5862be3e79 fix(proxy): resolve os.environ/ refs universally in DB-sourced models
Root cause: PR #30867 removed request-time os.environ/ expansion in
BaseAWSLLM.get_credentials. That is only safe if config-load pre-resolves
os.environ/ refs so the value reaching get_credentials is already the real
secret. The YAML config path has always done this. The DB-load path
(ProxyConfig._resolve_db_litellm_param) only re-expanded keys in a hardcoded
whitelist (_DB_LITELLM_PARAM_ENV_REF_KEYS) plus short-circuited env-ref
resolution entirely for team-scoped rows. PR #32256 extended that whitelist
to 18 keys to unblock a customer whose Bedrock model with aws_role_name:
os.environ/BEDROCK_ASSUME_ROLE_ARN broke on v1.90+, but the whitelist is
structurally fragile: every future auth field breaks the same way until
someone remembers to add it

Fix: remove the whitelist and the team-scope short-circuit. The DB-load
resolver now expands os.environ/ on every string field, matching the YAML
path. Trust boundary stays on the write side: only PROXY_ADMIN can create
team_id=None rows, only team admins of a team can create rows scoped to
that team, and the request-body vector is still blocked by
_BANNED_REQUEST_BODY_PARAMS. Team-scoped rows now resolve env refs — this
is a deliberate LIT-3831 threat-model expansion trusting team admins for
env-var reads

Regression tests in tests/test_litellm/proxy/proxy_server/test_proxy_config.py:
- test_ProxyConfig__add_deployment_resolves_env_refs_after_db_decrypt pins
  admin-scoped rows resolve every field (previously api_base stayed literal)
- test_ProxyConfig__add_deployment_resolves_team_env_refs pins team rows
  resolve env refs (previously stayed literal)
- test_ProxyConfig__add_deployment_resolves_env_refs_on_arbitrary_field pins
  the no-whitelist invariant against a made-up field name
- test_ProxyConfig__add_deployment_resolves_env_refs_for_aws_bedrock_auth_params
  (from #32256) still passes
- Path B counterparts (decrypt_model_list_from_db) mirror the above

Left as followups (not fixed here):
- /model/info and /v2/model/info still echo resolved values for fields not
  in the current pop-list (aws_role_name, aws_sts_endpoint, api_base, etc.).
  Fix is to extend remove_sensitive_info_from_deployment; separate PR
- Master-key rotation reads DB rows via decrypt_model_list_from_db which
  now resolves universally, so rotation collapses env-refs into hardcoded
  values. Pre-existing bug for the 6 previously-whitelisted fields; wider
  surface after this PR. Separate PR
2026-07-07 18:31:44 -07:00
Tin Chi Lo
9bf3907de5 fix(mcp): bind oauth_delegate discovery resource to the upstream
oauth_delegate forwards the caller's token to the upstream, which validates its
audience, so the protected-resource metadata must keep resource pointing at the
upstream (returned verbatim, like true_passthrough) rather than rewriting it to
the gateway. Rewriting to the gateway asks the client to mint a token bound to
the gateway audience, which a strict IdP (Entra) refuses to issue for an
unregistered resource and a spec-compliant upstream rejects on receipt. The
legacy is_oauth_passthrough opt-in keeps the gateway rewrite unchanged.
2026-07-07 17:32:13 -07:00
Tin Chi Lo
446a6e8cdd fix(mcp): route true_passthrough and oauth_delegate through upstream OAuth discovery
Both modes advertised LiteLLM as the authorization server and answered initialize locally, so a client with no token connected empty and was never driven into the upstream OAuth flow. The protected-resource discovery now proxies the upstream metadata for both modes (verbatim for true_passthrough, resource rewritten to the gateway for oauth_delegate), and the preemptive 401 emits the matching challenge: oauth_delegate uses the gateway-proxied resource_metadata once admission passes, true_passthrough probes the upstream anonymously and surfaces its WWW-Authenticate verbatim so the client authorizes directly against the upstream
2026-07-07 17:32:13 -07:00
Tin Chi Lo
50c90281f4 feat(mcp): add true_passthrough and oauth_delegate auth modes
Introduce two first-class MCP server auth_type values that make LiteLLM's
role in upstream authentication explicit, added alongside the existing
delegate_auth_to_upstream / oauth_passthrough flags without changing their
behavior.

true_passthrough is a transparent proxy: LiteLLM performs no admission auth,
requires no x-litellm-api-key, mints/stores/refreshes nothing, and forwards the
client's Authorization to the upstream exactly as received. oauth_delegate keeps
normal LiteLLM admission (x-litellm-api-key / SSO / JWT) and then forwards the
client's separate upstream Authorization unchanged; the admission credential is
never forwarded upstream.

Both modes forward the caller's token via the existing extra_headers path and
defer egress credential resolution to v1 (the v2 to_server_spec returns None for
them). Upstream 401/403 responses are surfaced rather than swallowed so upstream
OAuth challenges are preserved. Servers in either mode require per-user auth, so
userless health checks are skipped.
2026-07-07 17:32:13 -07:00
yuneng-jiang
999637883c
Merge pull request #32400 from BerriAI/litellm_internal_staging
Some checks failed
CodeQL / Analyze (python) (push) Waiting to run
CodeQL / Analyze (actions) (push) Waiting to run
CodeQL / Analyze (javascript-typescript) (push) Waiting to run
CodSpeed Benchmarks / benchmarks (push) Waiting to run
Helm unit test / unit-test (push) Waiting to run
Scorecard supply-chain security / Scorecard analysis (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
chore(ci): promote internal staging to main
2026-07-07 17:08:35 -07:00
tin-berri
db2402754a
feat(mcp): let users select the entra_obo token_exchange profile in the UI and API (#32144)
* feat(mcp): let users select the entra_obo token_exchange profile in the UI and API

The backend token_exchange arm supports two wire dialects via token_exchange_profile
("rfc8693" default, or "entra_obo" for Microsoft Entra's On-Behalf-Of, the RFC 7523
jwt-bearer grant), but it could only be set through config.yaml. This surfaces it to the
create/update REST API and the dashboard so an admin can create an entra_obo server there,
completing the parity started in the parent PR for the other token-exchange fields.

token_exchange_profile becomes a dedicated column on LiteLLM_MCPServerTable, mirroring the
sibling fields: it is added to the request models, read column-first in
build_mcp_server_from_table with the credentials-blob as a back-compat fallback and a
default of rfc8693, and carried through both runtime-to-table builders so registry
round-trips preserve it. It is a non-secret dialect selector, so it is not scrubbed from
non-admin or virtual-key responses.

In the dashboard a Profile dropdown (RFC 8693 vs Microsoft Entra OBO) is added to the
token-exchange section. Entra OBO carries the target resource in the scope, so selecting it
makes the scope required and hints the api://<app-id>/.default form, while audience and
subject_token_type (which that dialect ignores) are hidden.

* fix(mcp): extend the blob-to-column lift and non-admin scrubbing to token_exchange_profile

token_exchange_profile gets the same storage contract as the other three
token-exchange settings: the column is authoritative, a blob copy is the legacy
shape — lifted into the column on every write and stripped from the stored
blob — and switching auth_type away from token exchange clears it
(_AUTH_FLOW_SCOPED_FIELDS). Both restricted-view sanitizers scrub it for
uniformity, and the edit form's auth-switch payload nulling includes it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(mcp): assert every token-exchange setting is configurable via config.yaml

Pins the config surface: token_exchange_endpoint, audience, subject_token_type
and token_exchange_profile load from top-level config keys onto the built
server and through to the resolver spec; omitted keys resolve to their
documented defaults (RFC 8693 subject token type, rfc8693 profile), and
token_exchange servers need no oauth2_flow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 16:39:20 -07:00
tin-berri
10c7be239a
test(ui): pin token-exchange field visibility to the oauth2_token_exchange auth type (#32385)
* test(ui): pin that the token-exchange fields render only for the oauth2_token_exchange auth type

No form section asserted the visibility contract: the token-exchange fields
(Token Exchange Endpoint, Audience, Subject Token Type) must appear when
'OAuth Token Exchange (OBO)' is selected and for no other auth type. Assert
hidden under plain OAuth, shown under token exchange, hidden again after
switching to API Key.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(ui): assert the stdio transport switch unmounts the token-exchange fields

The create form gates the whole Authentication section on non-stdio transport,
so selecting OAuth Token Exchange (OBO) and then switching to stdio removes the
token-exchange fields (and their required-credential rules, which antd does not
validate while unmounted). Pin that sequence so the section-level gate cannot
regress silently.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 16:33:23 -07:00
tin-berri
9652509e46
fix(mcp): apply outbound concurrency limit to OBO tool calls (#32071)
The token_exchange (OBO) branch of _call_regular_mcp_tool built its coroutine
by calling _obo_call_tool_with_retry directly, outside the
_limit_outbound_concurrency context manager that the regular branch uses. OBO
tool calls (and the internal re-mint retry, which issues a second upstream
call_tool) therefore bypassed the per-server max_concurrent_requests semaphore,
so an authenticated caller could run unlimited concurrent tool calls against an
OBO MCP server despite an admin-configured limit.

Wrap the OBO coroutine in _limit_outbound_concurrency the same way the regular
path does, holding one permit across the initial call, the on-401 re-mint, and
the retry, so OBO calls honor the configured cap.
2026-07-07 16:02:30 -07:00
Mateo Wang
c8b78d49dd
fix(bedrock): stop stale SigV4 headers clobbering fresh signature on strip-and-retry re-sign (#32371)
* fix(bedrock): stop stale SigV4 headers clobbering fresh signature on re-sign

When the Anthropic /v1/messages strip-thinking-and-retry path re-signs a
Bedrock request, _sign_request received attempt 1's already-signed headers
and copied the old Authorization and X-Amz-Date back over the freshly
computed SigV4 signature, so the retry POSTed the stripped body with a
signature for the original body and AWS returned 403 SignatureDoesNotMatch.

Skip SigV4-computed headers (authorization, x-amz-date,
x-amz-security-token, date) when restoring caller headers after signing,
and only preserve a caller-supplied Authorization that is not itself a
SigV4 header so bearer-token setups keep working.

* fix(bedrock): apply the same stale-header guard to get_request_headers
2026-07-07 16:01:46 -07:00
Mateo Wang
06a97c83bd
test(realtime): assert guardrail block on backend wire traffic instead of model refusal wording (#32388)
test_text_message_blocked_by_guardrail_no_ai_response classified the
model's reply against a safe_markers keyword list to decide whether the
guardrail had blocked the message. gpt-realtime words its refusal of the
guardrail's "say exactly" voice prompt nondeterministically, so any new
phrasing outside the list turned CI red on unrelated PRs; the list had
already been extended in #28191, #28200 and #29477, and drifted again to
"Sorry, I can't comply with that request" (11 of the 13 failed
realtime_translation_testing runs since 2026-06-24, e.g. CircleCI job
2009316 on #32380).

Record every frame the proxy sends to the backend through a
RecordingBackendWebSocket wrapper and assert the invariant the product
actually guarantees: the blocked phrase never reaches OpenAI, only the
guardrail's own conversation.item.create and response.create are
forwarded (the client's reflexive response.create is dropped), and the
blocked phrase never appears in AI output. Replace the fixed
0.3s/3.0s sleeps with an event-driven wait for response.done; client
frames are processed sequentially so no inter-message sleep is needed.

Verified by mutation: disabling the response.create drop fails the
response.create count assertion, and disabling the guardrail fails the
guardrail_violation assertion.
2026-07-07 15:47:05 -07:00
tin-berri
ff6dc33291
feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard (#31772)
* feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard

OAuth 2.0 Token Exchange (RFC 8693, a.k.a. OBO) for MCP servers could previously only be
configured through config.yaml; the create/update REST API and the dashboard had no way to
express it. This wires token_exchange_endpoint, audience, and subject_token_type end to end.

These three are persisted as dedicated columns on LiteLLM_MCPServerTable, mirroring how token_url
and oauth2_flow are stored, so the edit form prefills them and they are unaffected by the
credentials-blob clearing on auth_type change. build_mcp_server_from_table reads the columns first
and falls back to the credentials blob so servers persisted before the columns existed still load.
client_id and client_secret continue to ride the existing encrypted credentials path.

On the dashboard, "OAuth Token Exchange (OBO)" is a distinct auth-type option with its own field
section. The McpOAuthMode classifier gains a token_exchange arm keyed off auth_type; the previous
catch-all oauth2 mode was renamed from "obo" to "authorization_code" so the two on-behalf-of
mechanisms are no longer conflated. The token-exchange IdP endpoint and audience are scrubbed from
non-admin and virtual-key responses, matching how token_url is treated.

* fix(ui): gate the MCP list-401 re-auth on authorization_code, not token_exchange

The renamed 401 gate keyed off isTokenExchange, but that condition exists for authorization_code:
when a stored per-user credential is present yet the tools/list 401s (the backend's refresh could
not mint a token), the user must re-authorize via the browser flow. token_exchange has no
gateway-side authorize step, so the Authorize gate never applied to it. The prior isObo flag was
undefined (a compile error) and, per this file's convention and its tests, meant authorization_code;
renaming it to isTokenExchange changed the behavior and broke the mcp_tools auth-gate test for an
authorization_code server whose token expired with no usable refresh. Gate on isAuthorizationCode
instead and drop the now-unused isTokenExchange

* fix(mcp): clear flow-scoped endpoint config when a server's auth_type changes

Switching an existing oauth2 server to oauth2_token_exchange left the old flow's
token_url on the row. The OBO resolver treats token_exchange_endpoint or token_url
as the configured exchange endpoint, so the stale value both suppressed the RFC
9728/8414 discovery this PR adds and sent the exchange grant (client credentials
plus the user's subject token) to the previous flow's token endpoint

update_mcp_server now mirrors its existing stale-credentials rule for the
flow-scoped columns (authorization_url, token_url, registration_url, oauth2_flow,
token_exchange_endpoint, audience, subject_token_type): when auth_type changes,
each one is cleared unless the same request explicitly provides it, so a
deliberate override in the switch request still wins. Updates that keep the
auth_type never touch these columns, which keeps legacy OBO rows that use
token_url as their exchange endpoint working

The edit form sends explicit nulls for the previous flow's fields on an auth type
switch; antd preserves unmounted field values by default, so without this the old
token_url would be re-sent verbatim and read as an explicit override. Transitions
are detected against the persisted auth_type, so saves that keep the auth type
send nothing extra

Reported by Cursor Bugbot on the PR

* fix(mcp): lift legacy blob token-exchange settings into their columns on every write

The three token-exchange settings live in dedicated columns but also exist on
MCPCredentials as the pre-column REST shape. Writes now lift incoming blob
values into the columns (an explicit top-level value wins, including an
explicit null) and strip them from the stored blob; the same-auth credentials
merge migrates legacy rows the same way. The read-time column-or-blob fallback
then only ever serves rows current code has never written, so clearing a column
to re-enable RFC 9728/8414 discovery can no longer be silently undone by a
stale blob copy.

Also asserts the auth-switch clearing fires on the external fields_set path
(PUT /v1/mcp/server).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(mcp): single source for the RFC 8693 default subject_token_type

The default was applied at four egress build sites plus two model defaults,
each with its own copy of the literal. All sites now share
DEFAULT_SUBJECT_TOKEN_TYPE from litellm.types.mcp. A DB-level DEFAULT is
deliberately not used: Prisma writes explicit values on insert, so a column
default would rarely apply, and NULL-means-RFC-default keeps existing rows
correct.

Also documents two review decisions in place: the audience column keeps the
RFC 8693 parameter name (RFC 8707 resource indicators are already a separate
concept named resource in the v2 egress types), and the migration's
out-of-order timestamp is safe under prisma migrate deploy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: fix import sort order in outbound_credentials/types.py (I001 strict budget)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(mcp): purge legacy blob copies when a token-exchange column is written without credentials

The migrate-on-write in the credentials merge lifts blob values into null
columns, which is correct for legacy rows but could repopulate a column an
admin had cleared in an earlier no-credentials update (that path never touched
the blob, so the stale copy survived to be lifted later). An explicit
token-exchange column write (set or clear) now migrates the row even when the
update carries no credentials: untouched null columns are lifted, every blob
copy is stripped, and unrelated blob keys stay as-is. A cleared column can then
never be resurrected, because no write path leaves a blob copy behind.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(mcp): state the blob-to-column lift contract on the legacy credential keys

The three token-exchange keys on MCPCredentials are the pre-column REST shape
(the only REST shape from 2026-05 until this PR). Document on both the blob
type and the request models that the dedicated columns are authoritative and
that writes lift blob values into them and strip the stored copy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(mcp): scrub subject_token_type in the non-admin and virtual-key sanitizers

The other two token-exchange fields were cleared while subject_token_type was
left visible. It is a public RFC 8693 URN with no disclosure value, but the
sanitizers' rule is that these views receive no token-exchange config at all —
cleared for uniformity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 15:26:12 -07:00
Mateo Wang
46d9742950
fix(vertex_ai): return create_vertex_url result directly for openai-path partner models with custom api_base (#32380) 2026-07-07 14:49:21 -07:00
Mateo Wang
ee69a62304
docs(CLAUDE.md): warn that harness-injected PR template copies strip HTML comments (#32373) 2026-07-07 14:24:09 -07:00
Mateo Wang
b8248a21d2
fix(vertex_ai): build full request path when custom api_base has no path (#32367)
Some checks failed
CodSpeed Benchmarks / benchmarks (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
2026-07-07 13:17:07 -07:00