Step 6b: override _get_tools_from_server to resolve auth via resolve() and list tools through the v2
UpstreamConnection (none / api_key / client_credentials / authorization_code). Inherits v1's
namespacing (_create_prefixed_tools) and static-header resolution. super() is reserved for
non-egress concerns that migrate as their own subsystems: OpenAPI tools (registry, S1.8), the
per-request mcp_auth_header override path, the JWT-signer guardrail, and any not-yet-mapped mode.
List errors map per the scheme: unauthorized -> MCPUpstreamAuthError (401 + WWW-Authenticate, the
LIT-3795 behavior); anything else degrades to an empty list (logged).
Also:
- Make the bridge's v1->v2 adapters public (to_subject / to_server_spec / provider); the egress
manager reuses them (was reportPrivateUsage).
- Make global_mcp_server_manager a PEP 562 lazy singleton, fixing a v1<->v2 import cycle (eager
module-load instantiation imported the v2 subclass mid-load; import-order dependent).
Live-validated: the m2m mode through the override against the harness lists echo via v2 (real token
fetch + Bearer-authed connection); the none mode via an in-process server integration test.
v2 is the egress manager, not a flag-gated opt-in. The composition root constructs
MCPServerManagerV2 directly (lazy import for the subclass cycle); there is no LITELLM_USE_V2_MCP_EGRESS
gate. The migration is the override progression (v2 inherits v1's per-server methods via super() for
modes not yet overridden), not a runtime toggle. Removed v2_egress_enabled and the now-redundant
MCPEgressManager Protocol (the gated-swap contract; v2 is a direct subclass) plus their imports.
Parity stays the validation method for the wiring, but v2's behaviors are the v2 improvements, not
v1 copies. The skeleton still has no overrides, so behavior is identical to v1 today; the per-server
egress overrides land next.
Step 6a of the egress cutover. MCPServerManagerV2 subclasses v1's MCPServerManager with no overrides
yet, so behavior is identical. The composition root selects it via _make_global_mcp_server_manager()
when LITELLM_USE_V2_MCP_EGRESS is set (lazy import avoids the subclass import cycle); flag-off stays
v1 exactly. Verified at startup in both flag states. The per-server egress overrides
(UpstreamConnection + resolve()) land in 6b/6c.
Adds transport selection: a _session_streams context manager opens the SDK client over the
server's transport (streamable-http, sse, or stdio) and normalizes to a (read, write) stream pair.
Auth/headers ride the httpx client per SDK 1.26 (not the transport kwargs); stdio takes
command/args/env with no auth; only the http client needs explicit cleanup. Tested live against
in-process http + sse FastMCP servers and a stdio subprocess. Cross-server aggregation/namespacing
lands with the v2 manager (step 6).
Completes the per-server read surface on UpstreamConnection: list_prompts, get_prompt,
list_resources, read_resource, mirroring list_tools/call_tool (the _run pattern, errors-as-values).
This is the operation surface the v2 manager's handler-facing methods map onto. Tested live against
an in-process FastMCP server with a prompt + resource. sse/stdio transports come next; cross-server
aggregation/namespacing lands with the v2 manager (step 6).
V1OAuthTokenStore backs the authorization_code arm's TokenStore by reading the user's
currently-valid upstream token via v1's resolve_valid_user_oauth_token (read + refresh-on-read).
Because v1 refreshes on every read, the returned token is always valid, so expires_at is set beyond
the resolver's refresh buffer to keep v2's proactive-refresh path inert; the OAuth dance and refresh
stay on v1 until it retires (token_refresher remains unwired). A missing/invalid token is Ok(None)
(the arm 401s to start the OAuth flow); a DB outage is upstream_unavailable. The reader is injected
for unit tests.
Wired into _provider (replaces the in-memory stub). With the _to_server_spec authorization_code
mapping, an interactive oauth2 (delegate=false) server resolves to a stored per-user token or a 401,
never forwarding the caller JWT (the v2 form of LIT-3795). Fires live at the egress cutover.
Adds the interactive-oauth2 split so v2 routes these modes instead of deferring to v1:
- interactive oauth2 (needs_user_oauth_token) + delegate_auth_to_upstream=false -> AuthorizationCodeConfig
(gateway-stored per-user token; the caller JWT is never forwarded). This is the v2 form of the
LIT-3795 fix: a non-delegated interactive oauth2 server must not forward the proxy-auth JWT.
- interactive oauth2 + delegate_auth_to_upstream=true -> PassthroughConfig (forward the caller token).
- auth_type=none + is_oauth_passthrough -> PassthroughConfig (forward), else NoneConfig.
aws_sigv4 stays on the aws_auth seam; misconfigured modes still defer to v1. These per-user modes
fire live at the egress cutover (they ride the extra_headers/mcp_auth_header seam, not
resolve_mcp_auth); the token bridges backing authorization_code land next in this step.
Per-user env-vars are templated static headers (resolved alongside auth via the static-headers
path), not an api_key credential mode; the PerUserEnvVar key_source was unreachable (no server
config maps to it) and contradicted that model. ApiKeySource is now SharedKey | Byok and the
api_key arm matches the two real sources. Static/env-var header resolution reuses v1's
_resolve_static_headers_with_env_vars at the v2 manager (step 6); the clean v2 rewrite lands at v1
retirement alongside the caching model.
Live-testing UpstreamConnection against the Bearer-protected harness surfaced that an upstream 401
mapped to upstream_unavailable (503) instead of unauthorized: the SDK wraps the httpx 401 in an
anyio ExceptionGroup, so the top-level .response check never saw it.
Reuse v1's extract_upstream_auth_failure (renamed from the private _extract_upstream_auth_failure)
instead of a hand-rolled flattener: it already walks the ExceptionGroup and __cause__/__context__
chains portably (BaseExceptionGroup is not a builtin on the supported 3.10) and handles 401/403.
ConnError drops the now-unused protocol_error tag.
Adds a regression test (Bearer-protected in-process server: no auth -> unauthorized; correct bearer
-> Ok) that fails with the old classifier.
Step 2 of the v2 egress transport. UpstreamConnection opens the SDK streamable-http ClientSession
to one upstream MCP server per request and runs an operation, attaching the resolved httpx.Auth
(plus any static/env-var headers) on litellm's httpx client (get_ssl_configuration for SSL/proxy),
returning typed results via ConnError (errors-as-values: a 401 is surfaced distinctly for the
upstream OAuth flow; transport failures map to upstream_unavailable). list_tools + call_tool land
here; sse/stdio and the prompt/resource ops come in later steps.
Still additive: not wired into the manager. Tests cover list_tools + call_tool against an
in-process FastMCP server and the unreachable -> upstream_unavailable mapping.
Step 1 of the v2-owned egress transport (the chokepoint). Adds:
- LITELLM_USE_V2_MCP_EGRESS flag + v2_egress_enabled().
- MCPEgressManager Protocol: the per-server egress operations the inbound handler invokes on the
manager (the swap surface). v1's MCPServerManager satisfies it today; MCPServerManagerV2 (later
steps) implements it via an UpstreamConnection. Registry/RBAC lookups are reused from v1 and are
not part of this contract; call_tool/dispatch is added when the v2 manager is assembled.
No behavior change; nothing is wired yet. Later steps add the UpstreamConnection, the
static-headers resolver, the per-user token bridges, then the flag-gated cutover that retires the
resolve_mcp_auth header graft.
Completes TokenExchangeConfig with the client authentication the exchange grant requires
(client_id/client_secret) plus scopes -- v1's exchange (and has_token_exchange_config) require
them, so the v2 config was incomplete. Then grafts the arm on the existing header seam, riding the
Subject edge (the inbound token is the Subject's inbound_token):
- request_token(endpoint, data, clock) is factored out of the client_credentials fetcher (both
hit a token endpoint and parse access_token/expires_in); the fetcher now calls it.
- HttpxTokenExchanger does the RFC 8693 grant via request_token: grant_type=token-exchange,
subject_token, subject_token_type, client_id/client_secret, audience=resource (RFC 8707), scope.
The inbound token is sent only to the exchange endpoint, never upstream.
- _to_server_spec maps has_token_exchange_config servers to TokenExchangeConfig (endpoint falling
back to token_url, resource = audience or url), placed before client_credentials to match v1's
cascade. _provider injects HttpxTokenExchanger; the per-user exchanged-token cache stays the
in-memory TokenStore (best-effort, re-exchangeable).
Intended divergence from v1: the exchange always binds to resource (audience or upstream URL,
RFC 8707); v1 sent audience only when explicitly set. Parity for servers with an explicit
audience; a new binding for those without (the resource-binding design).
Tests cover the exchanger grant shape + error mapping and the server->config mapping. 97 tests
pass; types.py adds no new errors, port bodies/bridge typecheck clean. Live e2e is deferred: it
needs an IdP exchange endpoint and an inbound subject_token (JWT), which the config-only local
harness does not set up.
BYOK is the api_key mode with the key seeded per-user; it grafts on the same header seam as the
shared key, riding the Subject edge. The v2 _api_key arm already resolves the Byok key_source via
the CredentialStore port, so this is graft wiring only:
- V1ByokCredentialStore (v2_port_bodies) bridges the CredentialStore port to v1's existing read
(db.get_user_credential -> find_unique + credential_b64 decrypt), keyed by
(subject_id == user_id, server_id). Missing row -> Ok(None) (the arm returns 401); DB outage ->
upstream_unavailable. An empty subject_id short-circuits to Ok(None) so an identity-less caller
never shares a credential slot (fail closed). The reader is injected so it stays unit-testable
without a DB.
- _to_server_spec maps an is_byok api_key server to ApiKeyConfig(key_source=Byok); the arm pulls
the per-user value from the store. _provider injects V1ByokCredentialStore.
Tests cover the store body (present / missing / empty-subject-skips-store / DB-error) and the
is_byok mapping. 92 tests pass; bridge typechecks clean, no new errors in the port bodies.
Live e2e is deferred: the store reads LiteLLM_MCPUserCredentials, so it needs a DB-backed proxy
with a seeded per-user credential (the local config-only harness has no DB).
Adds _to_subject(user_api_key_auth, subject_token) -> Subject, the single isolated mapping from
v1's authenticated principal onto the v2 Subject (so it can later swap to auth_v2's Principal):
subject_id <- user_id, tenant_id <- org_id (falling back to team_id), inbound_token <-
subject_token; an unauthenticated caller yields empty ids. resolve_mcp_auth and
resolve_v2_auth_value now accept user_api_key_auth and thread it (plus subject_token) through, and
_create_mcp_client passes its auth context in.
This is foundational for the per-user arms (BYOK api_key, token_exchange, authorization_code),
which must reject an empty subject_id rather than share one credential slot across callers. The
identity-free modes already grafted (none, api_key shared, client_credentials, aws_sigv4) ignore
the subject, so threading it is additive and backward-compatible; the params default to None.
Tests cover the mapping (org-over-team precedence, team fallback, missing user -> empty,
anonymous when no auth) and that threading identity doesn't change the grafted static modes. 87
tests pass; the bridge typechecks clean and no new errors land on the manager or token cache.
aws_sigv4 signs every request, so it cannot ride the header-extraction seam the other grafted
modes use; it grafts at MCPClient.aws_auth instead. _create_mcp_client now asks the bridge's
resolve_v2_aws_auth(server) for the signer when the flag is on and falls back to v1's
MCPSigV4Auth otherwise. The bridge wires HttpxSigV4Signer into the provider and maps a v1
aws_sigv4 server's aws_* fields onto AwsSigV4Config (assume-role / static keys / ambient),
deferring to v1 for the one shape v2 can't yet represent (assume-role with explicit base keys).
Tests cover the credential-source mapping, the signer signing a real request, the role+base-keys
defer-to-v1 case, the flag-off and non-aws defer paths. 83 tests pass; the bridge typechecks
clean and the manager imports the guarded hook.
HttpxSigV4Signer is the real SignerFactory body for the aws_sigv4 arm: it maps the typed
credential source (StaticKeys / AssumeRole / Ambient) onto v1's MCPSigV4Auth, so the signed
headers stay byte-identical to v1 by construction (the signer relocates into the v2 package when
v1 is retired). Credentials resolve eagerly off the event loop (asyncio.to_thread) so an
unassumable role or a missing ambient chain fails closed at build time rather than mid-request;
STS connection errors map to upstream_unavailable, everything else to misconfigured.
This is the seam-agnostic body only. Wiring it into the bridge and grafting it at
_create_mcp_client's aws_auth seam (aws_sigv4 signs per request, so it cannot ride the header
seam the other modes use) is a follow-up.
Tests cover signing a real request (AWS4-HMAC-SHA256 + X-Amz-Date), the session-token path
(X-Amz-Security-Token), region/service in the credential scope, and the error classification.
First real port body for the graft: HttpxClientCredentialsFetcher (v2_port_bodies.py), an
imperative-shell adapter that runs the RFC 6749 client_credentials grant via litellm's
configured httpx client, mirroring v1's grant for parity (client_id/client_secret/scope in the
body, parse access_token/expires_in, default TTL 3600; the arm's 60s _REFRESH_BUFFER handles
re-mint, matching v1's default buffer). The token response is validated with a pydantic model
rather than poking raw JSON. Error mapping: rejected grant (4xx) -> misconfigured (500, the
gateway's service-account config is wrong), endpoint down / 5xx / network -> upstream_unavailable
(503).
Wired into the bridge composition root (replaces the _Unwired fetcher; service-token store stays
in-memory, matching v1's per-worker caching) and grafted: _to_server_spec now maps v1 M2M
servers (oauth2 + oauth2_flow=client_credentials, i.e. has_client_credentials) to
ClientCredentialsConfig. The body lives on the v1/integration side so the v2 core keeps its
no-v1-imports invariant. The SDK's ClientCredentialsOAuthProvider is deferred to Phase 2 (it is
a connection-session-coupled httpx.Auth, attached when v2 owns the upstream MCP transport).
Tests: fetcher grant shape / token parsing / error mapping (rejected->500, 5xx->503,
network->503, missing access_token->500), the M2M adapter mapping, and an end-to-end graft test
(mocked token endpoint -> Bearer header). 74 tests pass; bridge typechecks clean.
The v2 resolver graft pulls in the expression library (via the outbound_credentials types). It
was declared in pyproject but uv.lock was never regenerated, so the runtime image (built with
uv sync --frozen) shipped without it; configuring any MCP server then crashed the proxy on
startup with ModuleNotFoundError: expression. Regenerate uv.lock to include expression, and
guard the bridge import in oauth2_token_cache so a missing v2 dependency degrades to v1 instead
of taking down the whole MCP feature.
Adds an info log on the success path of the v1->v2 graft so it's observable which servers the
v2 resolver handled and which header(s) it attached. Logs only header names, never values.
First strangler-fig graft of the v2 UpstreamCredentialProvider into the live v1 MCP request
path, off by default. When LITELLM_USE_V2_MCP_RESOLVER is set (the --use_v2_migration_resolver
CLI flag exports it), resolve_mcp_auth routes the none and api_key modes through the clean-room
v2 resolver and returns the resolved credential as a header dict, which MCPClient merges
verbatim; every other mode, a missing api_key, or any v2 error falls back to v1 unchanged. The
hook sits after the per-request override check so that precedence is preserved.
The bridge lives on the v1 side (v2 core keeps its no-v1-imports invariant). It adapts a v1
MCPServer to a v2 ServerSpec (none -> NoneConfig, api_key -> ApiKeyConfig on the X-API-Key
header), builds the provider with inert in-memory/unwired ports (none and api_key resolve from
config alone, so no real bodies are needed yet), runs resolve(), and extracts the produced
headers via httpx .raw to preserve casing (X-API-Key, not httpx's lowercased x-api-key).
Parity is asserted as the integration method: the v2-grafted upstream headers are byte-identical
to v1's for none and api_key, verified through the real MCPClient._get_auth_headers; flag off,
non-grafted modes, and tokenless api_key all defer to v1. v1 auth-priority tests still pass.
7 new tests; bridge typechecks clean and gates green.
The scheme enum assumed every static credential rides on the Authorization header with a prefix,
which is true for bearer/basic/token/raw but wrong for api_key: v1 writes that one to the
X-API-Key header (client.py:443-444), a different header entirely. Parity-testing the graft
surfaced it. API-key headers are not standardized (OpenAPI models the header name as
configurable; real upstreams vary - Atlassian uses Authorization: Bearer, MS Logic Apps and
Spring AI use X-API-Key, Azure APIM uses Ocp-Apim-Subscription-Key), so model the placement as
data: header_name (default Authorization) + value_prefix (default Bearer), like OpenAPI's apiKey
scheme. This expresses every v1 auth_type and any custom upstream header, with no per-scheme
enum and no leak.
Replaces ApiKeyConfig.scheme/header_for with header_name/value_prefix/header(); the arm builds
StaticHeaderAuth(value, header_name=name) via a small helper. The capability is not yet exposed
to admins (that is the deferred server-configuration phase); the v1->v2 adapter will fill the
pair from v1's auth_type. Test now covers bearer/basic/token/raw on Authorization plus X-API-Key
and a custom header. 56 tests, gates green.
Signs each outbound request with AWS SigV4 from the gateway's own AWS identity (never the
caller's), via a new SignerFactory port (botocore body deferred, faked in tests). The arm is
trivial - it delegates the whole thing to the factory - because all three credential sources
just build a signer differently.
Replaces AwsSigV4Config's bag of optionals with a discriminated credential sub-union
StaticKeys | AssumeRole | Ambient (defaulting to the ambient chain), so illegal combos are
unrepresentable, mirroring api_key's key_source. The factory owns the source match and error
mapping: creds unresolvable -> misconfigured (500), STS unreachable -> upstream_unavailable
(503); there is no user dimension, so no 401. The botocore body (3-way match, STS assume,
ambient resolution, temp-cred refresh, real signing) is deferred behind the port.
With this arm the resolver is complete: none, passthrough, api_key, authorization_code,
client_credentials, token_exchange, aws_sigv4. The _todo stub helper and the now-unused
AuthSpecKind import are removed; the not_implemented CredError variant stays as a valid part of
the error vocabulary.
Tests: the signer is returned and applied, config-error->500, STS-down->503, never-reads-inbound
(structural - build() takes only config), credentials default to ambient, the source
discriminator selects the right variant, illegal static_keys rejected, and the static secret is
masked. 55 tests, gates green.
Swaps the caller's live inbound token for a token bound to server.resource at the IdP's
exchange endpoint, via a new TokenExchanger port (hand-rolled RFC 8693 body deferred, faked in
tests). The inbound token is sent ONLY to the exchanger, never to the upstream - the upstream
gets the exchanged token; that is the core invariant distinguishing it from passthrough, and is
locked by a test. No inbound token -> unauthorized. Reuses the per-user TokenStore + Clock +
StoredToken; 'refresh' means re-exchange (no refresh token).
Folds in the scoping decisions: drop TokenExchangeConfig.audience (use server.resource for the
RFC 8707 indicator) and make token_exchange_endpoint optional (discovered via RFC 8414), so an
empty token_exchange config is valid. Error mapping via the exchanger: subject_token invalid ->
401 (user re-auths), config -> 500, endpoint down -> 503. Cache is best-effort (read failure
re-exchanges, write failure ignored), deliberately differing from authorization_code on the
same TokenStore since the exchanged token is re-exchangeable.
Tests: the swap invariant (inbound->exchanger, exchanged->upstream, bound to resource),
no-inbound->401, cached-fresh (no exchange), re-mint near expiry, subject-invalid->401,
config->500, down->503, cache-degrades, and discovery-default config. 48 tests, gates green.
The four fields (client_id, client_secret, authorization_url, token_url) were required, which
only models the legacy manual-registration case. Real MCP OAuth (Slack, Notion, Atlassian)
discovers endpoints (RFC 9728 -> RFC 8414) and registers the client via DCR (RFC 7591), so an
admin provides none of them - the discovered endpoints and DCR-registered client are obtained
at runtime and persisted by the AS surface, and resolve() reads only the per-user token from
the TokenStore. Make the four optional manual overrides; scopes is the one genuine config
field. The rejects-missing-required test moves to client_credentials (whose creds are
provisioned, not DCR'd), and a new test pins that an empty authorization_code config is valid.
Mints/serves one shared service-account token per server via the client_credentials grant.
Keyed by (server_id, resource) with no subject (ServiceTokenStore, a new port + in-memory
body), so every user shares the identity; the grant itself is delegated to a ClientCredentials
Fetcher port (SDK-backed body, faked in tests). The arm never reads the caller bearer (closes
the v1 M2M auth-bypass by construction), maps a rejected grant to misconfigured (500, the
operator's secret is wrong, no user to re-auth) and an unreachable endpoint to
upstream_unavailable (503).
The cache is treated as a pure optimization, not a source of truth: a read failure degrades to
a fresh mint and a write failure is best-effort (return the valid token, skip caching), because
an M2M token is always re-mintable from the secret - so a cache outage never fails a request
while the token endpoint is up. This is the explicit design choice starred on the Detailed
Phase 1 per-mode page, and it deliberately differs from authorization_code's propagate-on-write.
Tests cover cached-fresh (no fetch), mint+cache on miss, re-mint near expiry, rejected->500,
endpoint-down->503, never-reads-inbound, shared-across-subjects, and the read-degrade /
write-best-effort cache paths. 39 tests, gates green.
CredentialStore.get, TokenStore.get, and TokenStore.put returned bare values (T | None /
None), so a store/DB outage could only surface as a raised exception across the seam, and a
read could not distinguish 'not found' from 'backend down'. Change them to return
Result[..., CredError]: Ok(None) is a genuine miss (-> 401/412 to start the dance), while
Error(upstream_unavailable) is a store/DB outage (-> 503), per the plan's fail-closed-loud
invariant. The resolver's api_key and authorization_code arms unwrap with isinstance and
propagate the store error; a put failure on the refresh path also surfaces 503. Adds tests
for the per-user and authorization_code store-error paths (31 tests, gates green).
Stub arms returned of_misconfigured, which is documented as an operator config error (500) -
so an operator configuring a not-yet-built mode would get a misleading 'misconfigured' 500.
Add a dedicated not_implemented variant (-> 501 in http_status) and switch _todo to it; the
exhaustiveness gate forced the new arm in summary and http_status. Tests assert the stub arms
now signal not_implemented and that it maps to 501.
Per-user 3LO: resolve() reads the stored upstream token for (tenant, subject, server,
resource) from an injected async TokenStore, returns it as a Bearer if valid, refreshes it
proactively within a 60s window via an injected TokenRefresher (the real body is SDK-backed;
a fake is used in tests), and fails closed with unauthorized when there is no token or it is
expired without a refresh token. Refresh failures surface the refresher's own CredError
(unauthorized when the grant is rejected, upstream_unavailable when the endpoint is
unreachable). The OAuth dance that populates the store is the separate AS surface (S7); this
arm only reads and refreshes, and never replays the inbound caller bearer.
Design A (explicit lookup + proactive refresh returning a Bearer snapshot, refresher backed
by the SDK) over returning the SDK OAuthClientProvider directly: testable, explicit lifecycle,
clean fail-closed Result, with reactive-401 left to the transport. Adds StoredToken/TokenKey +
TokenStore (InMemoryTokenStore), Clock (SystemClock), and the TokenRefresher port; the provider
now takes these by DI. Access and refresh tokens are SecretStr. 29 tests, gates green.
The real CredentialStore body reads Prisma/Redis on LiteLLM's async stack, and the OAuth-flow
and SigV4 arms will do async I/O (token endpoints, RFC 8693, STS). A synchronous resolver would
block the event loop or force a breaking signature change once a caller exists. Define it async
now, before any runtime caller: CredentialStore.get, resolve(), and every arm are async, and the
tests await resolve() (asyncio_mode is auto). await sequences resolution before the upstream
call and the Result type still forces handling the missing case, so async introduces no
credential-less-call race.
Credential fields were plain str, so they would render verbatim in repr(), model_dump(), and
any structured log that serialises the config. Wrap them in pydantic SecretStr so they show as
'**********' everywhere while the resolver unwraps with get_secret_value(): client_secret on
the authorization_code and client_credentials configs, secret_access_key and session_token on
aws_sigv4, the shared api_key value, and the inbound passthrough token on Subject. The api_key
and passthrough arms unwrap at the point of use; a regression test asserts the value never
appears in model_dump_json.
The per-user api_key source was one PerUserKey variant that returned a single CredError on a
missing credential, collapsing v1's two distinct behaviours. Split it into Byok and
PerUserEnvVar (the ApiKeySource union is now SharedKey | PerUserEnvVar | Byok). Both still pull
the subject's value from the injected CredentialStore; they differ only when it is absent:
BYOK returns unauthorized (401 + WWW-Authenticate, the user must provide it) while the env-var
case returns the new precondition_required (412, a setup precondition). The edge mapping in
http_status renders those as 401 vs 412, and a comment marks that SharedKey is read straight
from ServerSpec.config rather than the per-user store.
Adds tests for both missing paths and the distinct status mapping.
Avoids the outbound_credentials/upstream_credentials.py redundancy; the file holds the one
resolve() entry point, so resolver.py names it for what it is. Renames the mapped test to
test_resolver.py to match, and updates the test import and the types.py docstring pointer.
Pure rename; UpstreamCredentialProvider keeps its name.
The subdomain resolves every egress credential mode (api_key, none, aws_sigv4, passthrough),
not just OAuth grants, so outbound_credentials names it accurately. Pure rename: relative
imports are unaffected; the spike import, the absolute test imports, and the path mentions in
the module/README docstrings are updated. The RFC 8693 token-type URN is left untouched.
Builds the api_key resolve() arm fully, including the injected credential pull. ApiKeyConfig
now carries a key_source discriminated union (SharedKey | PerUserKey): a shared key lives in
config, while a per-user / BYOK key is seeded per-subject and not static. header_for(value)
applies the v1 scheme prefix (Bearer/ApiKey/Basic/token/raw) to whichever value the arm
resolves.
UpstreamCredentialProvider gains a constructor that injects a CredentialStore (the new port
in credential_store.py, with an InMemoryCredentialStore body for tests and local wiring; the
durable DB-backed body lands later behind the same port). The api_key arm matches on
key_source: shared builds the header from config; per_user pulls the subject's secret via
self._credential_store.get(CredentialKey(tenant, subject, server)) and fails closed
(unauthorized) when absent. The pull lives inside resolve(); only the storage mechanics are
injected.
Tests cover the shared path across every scheme, the per-user hit, the fail-closed miss, and
per-subject isolation (another subject never receives u1's key), all constructed clean-room
with an in-memory store.
Moves the clean-room gateway package out of litellm/proxy/_experimental/mcp_server/v2
into its final home litellm/proxy/gateway/mcp (the folder graduation the migration plan
had slated for Phase 2, pulled forward so all further work lands at the final path with
no later import churn). Sitting outside _experimental/mcp_server also sharpens the
clean-room boundary: the gateway no longer nests under the v1 MCP code it must not import.
Pure relocation, no behavior change: git-tracked renames preserve history; relative imports
are unaffected; the test's absolute imports, the README's basedpyright command, and the
pyrightconfig venvPath depth are updated for the new location. Tests move to
tests/mcp_tests/gateway. black, ruff, basedpyright strict, and the 16 resolver tests all
pass at the new path.
Phase 1, step 1. ServerSpec now carries one per-mode `config` (the AuthConfig discriminated
union of seven frozen models, one per AuthSpecKind) and derives auth_spec_kind from it, so
the mode is a single source of truth and an illegal combination (e.g. an aws_sigv4 server
holding OAuth fields, or a mode missing required fields) is rejected at construction rather
than at call time.
resolve() moves to upstream_credentials.py and dispatches on the config variant via a
wildcard-free class-pattern match with an assert_never tail; basedpyright gates this shape
exactly like the enum (removing an arm fails reportMatchNotExhaustive + assert_never). Each
arm receives its own fully-typed config, so there are no None-checks.
The self-contained arms are real: none (NoOpAuth), api_key (StaticHeaderAuth carrying the
v1 scheme prefix), and passthrough (forwards the inbound token, fails closed when absent).
The OAuth-flow and aws_sigv4 arms are typed fail-closed stubs awaiting their collaborators
(token store, OAuth providers, RFC 8693 exchanger, SigV4 signer), which are the next step.
Tests construct Subject/ServerSpec directly with zero v1 fixtures: union rejection, the
three real arms across every api_key scheme, the no-inbound-token-replay invariant, and
fail-closed stubs.
The web edit that put authorization_code back on one line landed at 90 chars, which
fails the repo's black --check (CI lint-black). Trim the trailing comment so the line
stays single (matching its sibling enum members) and fits 88, rather than letting black
re-split it into the three-line form. Also drops the stale "sixth mode" wording in the
module docstring now that AuthSpecKind has seven members.
resolve() is the one function the whole seam exists for, yet it ended its match on
auth_spec_kind without the assert_never tail that both sibling matches (the spike's
label_enum and CredError.summary) carry and that the module README documents as
load-bearing. The compile-time gate (reportMatchNotExhaustive) already held, so this
is defence-in-depth: without the tail a bypassed gate (a stray case _ left in, which
suppresses the exhaustiveness check) lets the match fall through and return None,
silently violating the declared Result[httpx.Auth, CredError] contract. The tail turns
that into a loud failure and brings the seam in line with the documented pattern.
The first cut declared only the five OAuth-grant-shaped modes, which silently dropped
two live v1 auth_types: none (no upstream credential, the default for public upstreams)
and aws_sigv4 (per-request SigV4 signing for Bedrock AgentCore-style upstreams). Neither
had a home, so a server configured with them would have fallen through the resolver.
Adds both as AuthSpecKind members with their resolve() arms (Phase 0 stubs), and records
that the static-header family v1 splits across bearer_token/api_key/basic/token/authorization
collapses into the single api_key mode with the scheme carried as a parameter rather than
its own mode. The match-exhaustiveness gate forced the spike's label_enum to grow the two
arms too, which is the gate doing its job.
Pulls forward the minimum from S0 and S8 needed for the clean-room v2 mini-chassis,
the typed OAuth credential seam that Phase 1 gives a body. Nothing is wired into v1
yet; this phase has no runtime callers, so it is kept separate from the Phase 1
OAuth/credential build that follows.
Adds expression to the proxy extra and basedpyright to the dev group so the FP spine
is a declared dependency rather than relying on an ambient install. Ships the scoped
types.py slice: the CredError tagged union, frozen Subject/ServerSpec, the boundary
parser, and the resolve((subject, server)) -> Result[httpx.Auth, CredError] signature
dispatched exhaustively on the declared auth_spec_kind with stub arms (no credential
logic). Vendors Ok|Error as Result so wrong-side access is a type error.
Includes the match-exhaustiveness spike that de-risks the whole approach: it proves
Expression's @tagged_union plus match satisfies basedpyright strict
reportMatchNotExhaustive, and README.md records the verdict plus the remove-an-arm
reproduction showing the gate bites
completion_cost read service_tier straight from the request optional_params
and called service_tier.lower() on it, so a non-string value (dict/int/list,
reachable via allowed_openai_params/drop_params) raised AttributeError.
_response_cost_calculator swallowed that and returned response_cost=None, so
the request's cost was silently lost.
The isinstance guard alone is not enough: a surviving dict would crash again
downstream in _get_service_tier_cost_key, which also calls .lower(). A
request-level service_tier is only meaningful for pricing when it is a concrete
billable tier string, so coerce any non-string value to None and defer to the
tier the provider reports on the response usage, the same way "auto" already
does.
Adds a regression test driving a dict service_tier through completion_cost; it
raises AttributeError before the fix and prices at the served tier after.
test_basic_vertex_ai_pass_through_with_spendlog failed intermittently on
litellm_internal_staging (pipelines 82155, 82196, 82209, 82230) with "Spend
should be greater than before after 120s". Spend logging is async and batched,
so the pass-through call's cost sometimes had not landed within the 120s poll
window; one run ended on spend_after 0.0 because the final /global/spend/logs
read returned nothing and "or 0.0" recorded that as zero spend.
Widen the poll window to 240s and skip a transient empty read instead of
treating it as 0.0, so a momentary endpoint hiccup on the last poll no longer
fails an otherwise-billed call. The spend_after > spend_before assertion is
unchanged, so a genuinely unbilled call still fails the test
* test(proxy): poll for image-gen spend instead of a fixed 5s sleep
test_key_info_spend_values_image_generation failed once on litellm_internal_staging
(pipeline 82282) with "spend did not increase on an identical repeat image call"
(assert 0.24966 > 0.24966). The test made the second image call, slept 5s, then
read the key's spend once. Response caching is commented out in
proxy_server_config.yaml and no sibling test enables it, so the likely cause is
async/batched spend logging not having flushed the repeat call's cost within 5s,
which the build_and_test job aggravates by running every tests/test_*.py against
one shared proxy under pytest -n 4.
Poll the key's spend for up to 60s and break as soon as it grows. This removes
the timing flake while preserving the canary: if the repeat were genuinely
unbilled (for example the proxy response cache being on), spend never grows, the
poll times out, and the assertion still fails.
* test(pass_through): raise ruby assistants client request_timeout to 600s
The streaming assistants example in openai_assistants_passthrough_spec.rb hit
Net::ReadTimeout on litellm_internal_staging (pipeline 82280), failing at roughly
125s which is ruby-openai's default request_timeout of 120s. An assistants run
with the code_interpreter tool can occasionally take longer than that to stream
its first content back through the pass-through.
Raise the client's request_timeout to 600s, matching the 600s timeout the Python
pass-through e2e tests already use, so a slow-but-healthy streaming run no longer
trips the default read timeout.
* fix(health): correct bedrock embedding health checks
Health checks for Bedrock embedding deployments failed in two ways. A
deployment configured without an explicit model_info.mode was probed as
chat, so max_tokens was injected and Bedrock embeddings rejected it with
400 "extraneous key [max_tokens]". Separately, stripping the bedrock/
routing prefix dropped the provider, so a cross-region inference-profile
id like us.cohere.embed-v4:0 failed downstream with "LLM Provider NOT
provided".
Resolve the deployment mode from the model cost map (which understands
the bedrock/ and us./eu./apac. prefixes) before deciding whether to
inject max_tokens, and pin custom_llm_provider to bedrock when stripping
the prefix so the bare model id still resolves. ahealth_check now accepts
any string mode so the resolved embedding mode routes the probe to the
embedding handler.
* fix(health): preserve explicit custom_llm_provider on bedrock probe
The bedrock prefix-strip pinned custom_llm_provider to bedrock
unconditionally, so a deployment that set custom_llm_provider:
bedrock_converse had it overwritten at health-check time and the probe
hit the Invoke endpoint instead of Converse, a different request format
that can report a spurious failure. Only fill in bedrock when the
deployment left the provider blank, which still resolves bare
cross-region ids like us.cohere.embed-v4:0 while leaving an explicit
provider untouched.
* test(health): assert resolved mode reaches the ahealth_check probe
The existing tests check _resolve_health_check_mode and the params builder
in isolation, but nothing verified that _run_model_health_check actually
threads the resolved mode into litellm.ahealth_check. Without that, a
refactor that probed with model_info.get("mode") again would reintroduce
the chat fallback for embedding deployments while every test stayed green.
This drives _run_model_health_check with a bedrock embedding deployment and
asserts the probe is called with mode=embedding and the embedding params.
* fix(health): resolve probe mode once for reasoning_effort and audio_speech
The reasoning_effort and audio_speech branches read model_info.mode
directly, so an embedding deployment declared without an explicit mode (the
case this PR targets) was still treated as chat-like: a configured
health_check_reasoning_effort got injected into the embedding probe, which
embeddings reject as an unknown field, and an auto-detected audio_speech
deployment never had its voice set. Resolve the effective mode once from the
cost map and reuse it for the max_tokens, reasoning_effort, and audio_speech
decisions so they all agree with the mode threaded into ahealth_check.
Setting --max_requests_before_restart alone recycles every worker at almost the
same time once they have served a similar number of requests, which under
sustained load can drop a whole pod's capacity at once roughly every 7-10 days.
This exposes a jitter knob that adds a random amount in [0, jitter] to the
restart threshold per worker so restarts are staggered. It maps to uvicorn's
limit_max_requests_jitter and gunicorn's max_requests_jitter. uvicorn only
gained limit_max_requests_jitter in 0.41.0 while litellm still allows
uvicorn>=0.33.0, so the uvicorn path feature-detects the parameter via the
Config signature and warns instead of crashing on older versions. The flag has
no effect without --max_requests_before_restart, so the kwarg is not forwarded
in that case and a warning is printed on both the uvicorn and gunicorn paths.
Resolves LIT-3774
* fix(proxy): resolve list files credentials from team BYOK deployments
GET /v1/files without target_model_names now prefers the team's own
deployment (model_info.team_id) over shared global provider keys, so JWT
team auth lists files against the correct upstream account.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): scope list files credential lookup to team allowlist
Remove the unrestricted deployment scan that could leak global provider
keys to teams without access, normalize all-proxy-models to the team-scoped
model list, and fix TID251 violations by using dict instead of Dict/Any.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The OpenAI moderation guardrail (and the ai-platform-moderation guardrail
built on it) stamped the whole moderation model response into the guardrail
trace as guardrail_response. That blob carries the full category_scores map
plus categories and category_applied_input_types, which on OTEL backends that
index span attributes (for example ELK, which caps indexed attribute values at
1024 chars) overflows the limit and gets truncated, so the violated categories
cannot be reliably searched.
Extract the flagged category names from the moderation response and pass them
through tracing_detail to add_standard_logging_guardrail_information_to_request_data,
mirroring the Bedrock hook. Both the legacy and v2 OTEL integrations already
read violation_categories off the standard logging guardrail information and
emit it as a short, queryable guardrail_violation_categories attribute, so
dashboards can group and filter by violation category without parsing the large
guardrail_response blob.
Resolves LIT-3801
* ci: drop redundant mypy type-check gate, standardize on basedpyright
Type checking ran both mypy (via the pydantic.mypy plugin) and basedpyright.
pydantic v2 emits dataclass_transform, so basedpyright understands models
natively with no plugin, and its gated rules already cover what the mypy pass
caught (no-untyped-def, no-any-return, valid-type, import-not-found all map to
basedpyright equivalents). Running both meant two checkers, two budgets, and a
plugin only mypy could load.
This removes the mypy type-check gate: the lint-mypy/lint-mypy-budget-update
Makefile targets, the CI MyPy step, mypy-code-budget.json, the budget-ratchet
entry, and the vestigial [tool.mypy] pydantic plugin block (the gating pass used
litellm/mypy.ini, which never loaded the plugin). type_check_gate.py is
specialized to basedpyright since the mypy parsing path is now unused.
mypy stays a dev dependency because the Any-discipline gate
(scripts/check_any_discipline.py) imports it as a library to detect Any-typed
values; it is no longer run as a type checker.
* ci: remove the Any-discipline gate, rely on basedpyright's reportAny
The Any-discipline gate (scripts/check_any_discipline.py) was the last consumer
of mypy: it imported mypy as a library to detect values whose inferred type
contains Any, gated per-file against any-discipline-budget.json. basedpyright
already reports the same class of finding through reportAny/reportExplicitAny,
which are gated tree-wide in basedpyright-code-budget.json, so the separate gate
(and the mypy dependency behind it) is redundant.
Removes the gate end to end: check_any_discipline.py and its test, the
any-discipline CI job, the lint-any/lint-any-budget-update Makefile targets,
any-discipline-budget.json, litellm/mypy.ini, the .mypy_cache_any references,
and mypy from the dev dependencies. budget_ratchet_check.py drops the
any-discipline entry and the now-unused zero-floor mechanism (rewritten as a
comprehension). check_type_discipline.py drops the any-ok suppression token,
since # any-ok suppressed only the deleted gate; the 134 now-orphaned
# any-ok comments across 14 files are stripped (they never affected
basedpyright, which uses # pyright: ignore).
uv.lock is intentionally left untouched: uv still considers it consistent with
the mypy-removed pyproject (uv lock --check and uv sync --frozen both pass), and
a relock bumps 30+ unrelated packages because of the moving exclude-newer window.
A future intentional relock will prune the now-unreferenced mypy entry.
* build: relock to drop mypy from uv.lock
CI's uv 0.10.9 honors the repo's exclude-newer window and correctly flags the
lockfile as out of sync once mypy leaves pyproject; my earlier local uv 0.8.17
could not parse exclude-newer and silently passed --check. Relocking with the
pinned CI version removes only mypy and its transitive librt, with no other
version changes.
