2767 commits

Author	SHA1	Message	Date
ryan-crabbe-berri	ac56320f26	fix(agents): show an agent's attached virtual key in the UI (#29619) ... * fix(agents): show an agent's attached virtual key in the UI The A2A agent detail view never surfaced which virtual key was attached to an agent, so after assigning a key during agent creation there was no way to see it again. Surface the attached key(s) in the agent detail view, derived from the key table's agent_id foreign key the same way spend is already joined into the agent response. Backend adds an agent_id filter to /key/list (mirrors team_id) and enriches GET /v1/agents and GET /v1/agents/{id} with a non-secret key summary (alias, masked key_name, hashed token id). The frontend renders a Virtual Keys section in the agent detail view that lists the agent's keys and links through to the key detail, and the list view drops its fetch-500-keys-and- filter-client-side workaround in favor of the enriched response. The orphaned AgentCard and AgentCardGrid components, left behind when the agent list switched from a card grid to a table, are removed * fix(agents): redact attached virtual keys for non-admins _attach_keys_to_agents joins keys onto the agent response by agent_id with no caller scoping, but _redact_sensitive_agent_fields never cleared the new keys field. A non-admin able to view an agent therefore received the alias, masked name, and hashed token of every key attached to it, including keys owned by other users or teams; the old client-side path used the scoped key list, so this was a visibility regression. Clear keys in the redaction path so only admins see attached-key metadata. Adds an endpoint-level regression test asserting keys is populated for admins and null for non-admins, and a list-view test covering the Active vs Needs Setup badge that lost coverage when the agent card tests were removed. * fix(agents): satisfy strict lint and resync key/list types - use builtin list/dict generics in the new agent key helpers to stay under the UP006 strict-rule ceiling - swap @tremor/react for antd Typography in agent_virtual_keys (tremor is being phased out; the new component was the only unsuppressed import) - regenerate schema.d.ts so the /key/list agent_id query param is typed * style(agents): prettier-format key hook test and agent_info	2026-06-27 16:44:25 -07:00
tin-berri	5963b9320f	feat(mcp): cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] (#31493) ... * feat(mcp): encrypt+serialize codec for caching OAuth tokens in Redis (step 1b §1.5) The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss. * feat(mcp): DualCache-backed token cache backend (step 1b §1.5) The cross-replica TokenCacheBackend implementation that plugs into the foundation's CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected; a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss. * feat(mcp): Redis SET NX PX refresh coordinator (step 1b §1.5) The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis. * feat(mcp): Redis SET NX PX distributed lock (step 1b §1.5) The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh), release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired / not-held so a cache blip causes an extra refresh, never a crash on the resolve path. * feat(mcp): wire the cross-replica cache + coordinator into the per-user store (step 1b §1.5) Upgrade the composition root to use the DualCache-backed cache and SET NX PX refresh coordinator when Redis is wired, falling back to the foundation's in-process defaults on a single replica. Layers the cross-replica path on top of the single-replica dispatch store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): refresh on lock-backend error instead of serving a stale token The cross-replica refresh coordinator elected refreshers with a boolean acquire: a Redis transport error was caught and returned as False, which is indistinguishable from "another worker holds the lock". On a total Redis outage every worker therefore took the wait-then-reread branch and served the still-expired token upstream (the upstream then 401s), even though the lock and coordinator docstrings claimed a Redis blip "degrades to an extra refresh". Make acquire tristate (LockAcquisition: ACQUIRED / HELD / ERROR) so the coordinator can tell a busy holder from a dead backend, and refresh anyway on ERROR. This single-flight lock is a load optimization, not a correctness mutex, so failing open is correct: it degrades a lock-backend outage to the no-coordinator behavior (an extra refresh), never a stale bearer. Add a regression test asserting an acquire error refreshes rather than re-reading the expired token, and update the docstrings to match. * style(mcp): wrap redis lock signatures at line-length 88 for CI ruff format * fix(mcp): a refresh loser surfaces None, not a stale token, when the winner failed The cross-replica coordinator's losers re-read the token the winner persisted. If the winner's refresh failed, the store still holds the expired token, so the loser re-read it and RefreshingTokenStore handed that expired bearer to the caller (the upstream then 401s) instead of the re-auth challenge the winner returned via None. Make the loser's re-read expiry-aware, mirroring refresh_latest_token: a re-read that is still expired surfaces None so the arm challenges. This only affects the loser path; the winner's freshly refreshed token is returned directly by the coordinator and is unaffected. * fix(mcp): log per-user token decrypt failures at debug, matching v1 When a cached blob cannot be decrypted (e.g. after a salt or master-key rotation) the codec logged a full traceback at error level, since decrypt_value_helper defaults to exception_type=error. v1's MCPPerUserTokenCache passed exception_type=debug on the same path. The blob is ciphertext so this is log noise only, but matching v1 avoids error-level traceback spam on stale entries after a key rotation * fix(mcp): namespace the refresh lock key and fence its release with a token The Redis lock wrote its key through the raw client from init_async_client(), bypassing RedisCache's namespace, so two deployments sharing one Redis collided on mcp:refresh_lock:<user>:<server> for any overlapping (user, server) and a colliding deployment skipped the refresh and challenged its own users. The lock now runs every key through an injected namespace_key wired to RedisCache.check_and_fix_namespace, matching the namespace its token cache already uses release() also deleted the key unconditionally, so a holder whose lock PX-expired and was re-acquired by another worker could delete the new holder's lock and let a third worker run a duplicate refresh, recreating the rotating refresh_token race. acquire now writes a unique per-acquisition token generated by the coordinator and release deletes only when the key still holds that token, via a compare-and-delete Lua script Adds regression tests: release with a stale token is a no-op while the owner's release deletes; keys are namespaced before reaching Redis; the coordinator acquires and releases with the same token * fix(mcp): fail open when the per-user token cache delete errors DualCache swallows get/set errors internally but not delete, and the Redis layer underneath re-raises through its circuit breaker. So a Redis outage on the delete() path escaped CachedOAuthTokenStore.fetch()'s unauthorized branch (which deletes before returning None) and invalidate(), turning a cache blip into a 500 instead of the v1-style fallback. Catch in the backend so delete degrades to the TTL-bounded stale entry like get/set already do. * style(mcp): reformat outbound-credentials files to line-length 120 The merge from staging brought in ruff's line-length 120, but these two PR-authored files were still wrapped at the old width, so the diff-scoped ruff format --check in CI flagged them. Pure reformatting; no behavior change. * fix: harden mcp oauth redis refresh coordination * fix(mcp): make the per-user token cache backend airtight on boundary failures get/set now degrade a cache or codec failure to the safe value (miss / no-op) in the backend itself rather than relying on DualCache and decrypt_value_helper happening to swallow internally, matching delete() and v1's MCPPerUserTokenCache. This upholds the layer's boundary-failure-is-a-miss contract regardless of the injected collaborators, so a Redis outage or an undecryptable entry reads as a cache miss that re-reads the DB instead of a 500. Adds contract tests for the cache raising on get/set/delete and the codec raising on encode. * test(mcp): pin per-user cache get() to a miss when decrypt raises Greptile's out-of-diff repro had the decrypt reject a blob with ValueError (bad ciphertext after key rotation); cover that exact raise path, not just the decrypt-returns-None case, so get() is regression-locked to read it as a miss. * refactor(mcp): use frozen dataclasses for the trivial DI constructors Replace the hand-written self._<arg> = arg constructors on OAuthTokenCacheCodec, RedisRefreshCoordinator, RedisDistributedLock, and DualCacheTokenCacheBackend with frozen slotted dataclasses, matching the rest of this layer. Fields take the former parameter names so the constructor API (and the tests' keyword args) are unchanged; KW_ONLY preserves the keyword-only collaborators. * fix: serialize lazy per-user oauth store rebuild * fix(mcp): stop losers challenging mid-refresh by decoupling wait from lease TTL wait_timeout_seconds defaulted to the same 10s as lock_ttl_seconds, but the holder renews its lease while a slow token endpoint runs, so a loser waiting past 10s bailed and re-read the still-expired DB token, challenging the user even though a valid refresh was in flight. Bound the holder's renewal with a refresh budget so its lock-hold is finite, and set the loser's wait to outlast that budget (refresh_budget_seconds + one lease tail) so a loser only re-reads once the holder has finished or its bounded lease has lapsed, never mid-refresh. * fix: allow concurrent lazy OAuth fetches without Redis --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-27 16:27:34 -07:00
Yassin Kortam	b2e708d5ae	feat(prometheus): add per-team litellm_team_members_metric gauge (#31506) ... Emit litellm_team_members_metric on every team member add and delete, labelled by team and team_alias and set to the team's authoritative member count. Because it is set from the current membership rather than incremented or decremented, it tracks the count up and down, never goes negative, and self-corrects on the next change after a proxy restart. Bulk member add is covered for free since it delegates to team_member_add, and the helper no-ops when the Prometheus callback is not registered. Resolves LIT-3082	2026-06-27 12:19:50 -07:00
Yassin Kortam	1883f975e2	fix(proxy/auth): honor user_api_key_cache_ttl for management-object cache writes (#31504) ... general_settings.user_api_key_cache_ttl was ignored for every management-object write into user_api_key_cache. The configured value is propagated to the cache's default_in_memory_ttl at startup, but DualCache only applies that default when no explicit ttl kwarg is passed, and every management-object writer passed ttl=DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL (60s), which always won. So keys, teams, users, budgets, object permissions, vector stores, JWT user syncs and MCP caches all expired after 60s regardless of the setting. Adds get_management_object_ttl(cache) in user_api_key_cache.py, which returns the configured default_in_memory_ttl and falls back to the 60s constant only when no default is set, and routes every management-object writer through it. The helper takes a DualCache so it works at the many call sites that are typed UserApiKeyCache but exercised with a bare DualCache. Also covers the spend-update writeback in update_cache (async_set_cache_pipeline), which hardcoded ttl=60 on the same key/user/team objects and reset an active key's cache entry back to 60s on every priced request, so the configured TTL was never observed for keys receiving traffic. Resolves LIT-3338	2026-06-27 12:19:28 -07:00
Yassin Kortam	63490655ad	fix(pass_through): log pre-call guardrail blocks at WARNING, not ERROR with a traceback (#31500) ... A pre-call guardrail block on a pass-through endpoint (e.g. OpenAI moderation flagging disallowed content) was logged at ERROR level with a full stack trace, even though the guardrail is working as designed and the client correctly receives the 4xx. The generic except in pass_through_request logged every exception via verbose_proxy_logger.exception(), so an intentional block produced scary traceback noise for operators tailing logs. Branch on the existing CustomGuardrail._is_guardrail_intervention classifier (the same predicate pipeline_executor already uses) so guardrail interventions log once at WARNING without a traceback while genuine failures keep their ERROR and traceback. This covers every guardrail that signals a block through the shared typed exceptions or an HTTPException 400, not just OpenAI moderation, and leaves the client-facing response unchanged. Resolves LIT-3538	2026-06-27 12:18:40 -07:00
Yassin Kortam	c33a7f8757	fix(proxy): cancel upstream LLM stream when client disconnects during time-to-first-token (#31499) ... create_response buffers the first streamed chunk (to detect error-only streams) before handing the StreamingResponse to Starlette. Starlette only starts listening for client disconnects once it is serving that response, so a disconnect during a long time-to-first-token left the upstream LLM call running until the request timeout. This races the first-chunk fetch against an http.disconnect monitor; on disconnect it cancels the fetch, which propagates into async_streaming_data_generator's cleanup (records the 499 and closes the upstream stream), and returns a 499. Resolves LIT-3568	2026-06-27 12:07:23 -07:00
Yassin Kortam	437acc9b09	perf(proxy): bound event-loop blocking from oversized requests (#31497) ... Skip token counting in Router._pre_call_checks when no deployment in the group declares max_input_tokens, and skip the full-body surrogate-repair regex in _read_request_body above a configurable size, raising the existing 400 immediately. Resolves LIT-3541	2026-06-27 12:06:48 -07:00
Yassin Kortam	80d3b69d9c	fix(pass-through): remove stale routes by key so the registry stops growing every reload (#31314) ... The 30s add_deployment_job re-runs initialize_pass_through_endpoints, which re-registers every config/DB pass-through endpoint. Endpoints without a persisted id get a fresh uuid each cycle, so their route key ("{id}:{type}:{path}:{methods}") changes every reload. The stale-route cleanup called remove_endpoint_routes(route_key), but that helper matches entries by endpoint_id, so it never matched a route key and never deleted anything. The registry grew by one entry per route per reload, turning the per-cycle cleanup and the per-request is_registered_pass_through_route scan into a CPU sink that eventually pins a core and slows every endpoint. Pop the stale key from the registry directly in O(1). openai_routes is left alone: its append is path-deduped and the path is still owned by the live endpoint re-registered under a new id in the same cycle. Resolves PERF-13	2026-06-27 10:34:21 +03:00
tin-berri	4b398ef6d4	feat(mcp): migrate authorization_code MCP to the v2 resolver (single-replica) [1/2] (#31473) ... * feat(mcp): implement the authorization_code resolver arm Resolve a user's authorization_code token through the injected OAuthTokenStore: present -> Authorization: Bearer <access_token>; absent -> the RFC 9728 WWW-Authenticate OAuth challenge; store unavailable -> the same challenge (not a 500), since a transient outage is not a definite absence. UpstreamCredentialProvider gains the oauth_token_store collaborator (fail-closed null default); per-subject isolation comes from keying the fetch on subject_id. Not live until to_server_spec maps authorization_code and a v1-backed token source is wired (next steps). * feat(mcp): v1-backed OAuth token source for authorization_code V1PerUserTokenStore reads the user's stored access token through v1's mcp_per_user_token_cache (Redis-backed, encrypted) and wraps it in an OAuthToken. v1 holds only the access token (its cache TTL is the lifetime), so no expires_at/refresh_token yet; the v2 cache holds it for its default TTL and the OAuth challenge drives re-auth once v1's cache drops it. Additive: nothing wires it yet, so no behavior change. Step 1b swaps it for a v2-native token store behind the OAuthTokenStore seam. * style(mcp): modern type annotations in the authorization_code arm and source * refactor(mcp): share v1's OAuth egress core; make V1PerUserTokenStore refresh-capable Extract v1's per-user OAuth egress (Redis cache, else DB read with the refresh_token grant, then re-cache) from _get_user_oauth_extra_headers_from_db into resolve_user_oauth_access_token in db.py; the v1 header builder is now a thin wrapper over it and its callers are unchanged. V1PerUserTokenStore (the v2 OAuthTokenStore adapter) resolves through that same core via an injected server lookup, so the authorization_code arm injects exactly the token v1 would, with the same silent refresh, rather than a Redis-only read that can never refresh. One resolution implementation, two thin adapters (header dict and OAuthToken). Behavior-preserving: the existing v1 egress tests pass unchanged, and the arm is not wired into the live path yet (that lands with to_server_spec + the manager). * feat(mcp): route oauth2 per-user (authorization_code) servers through the v2 resolver to_server_spec maps an oauth2 server to AuthorizationCodeConfig when it relies on per-user tokens (needs_user_oauth_token and not delegate_auth_to_upstream); client_credentials (M2M), delegated upstream OAuth, token exchange, and SigV4 still defer to v1. The manager injects V1PerUserTokenStore (resolving through v1's shared egress core) into the credential provider. The v2 path is live but still defers to a token v1 places in extra_headers; the cutover that makes v1 step aside lands next, alongside the unified challenge. * feat(mcp): per-server fail-closed OAuth challenge at the v2 egress When an authorization_code server has no usable per-user token, the arm returns a semantic unauthorized and the graft builds the 401 where the full MCPServer is in hand: a relative, per-server RFC 9728 resource_metadata pointer (/.well-known/oauth-protected-resource/mcp/{name}) that names the server's own authorization server, instead of the resolver's earlier root pointer which resolved to the gateway's generic PRM. Relative, so it is correct behind a reverse proxy without request context. The listing-phase 401 still emits the RFC 8414 authorization_uri form; both now target the same server, so the remaining difference is cosmetic and unifies in a later PR. * feat(mcp): cut the call_tool egress over to v2 for authorization_code servers _resolve_oauth2_headers_for_tool_call steps aside (builds no header) when to_server_spec maps the server, so the v2 resolver drives the token-present case instead of being shadowed by a token v1 places in extra_headers. Non-migrated oauth2 (delegate, client_credentials) and BYOK still build their header on v1. With this, v2 owns the authorization_code egress end to end: inject the refreshed per-user token when present, raise the per-server fail-closed 401 when absent. * feat(mcp): cut the tools/list connection over to v2 for authorization_code servers The listing connection's per-user OAuth header is no longer built by v1 for migrated servers; the v2 resolver drives it at connect time, ending the double-resolution where v1 built the token into extra_headers and the v2 graft then deferred to it. Safe because the preemptive 401 (in the streamable-http and SSE handlers) already challenges a missing token before the listing connection runs, so the connection is only reached with a token present. Non-migrated oauth2 (delegate) and the rest still build their header on v1. With this, resolve_credentials' result is honored on every authorization_code upstream path: tool calls and listing. * feat(mcp): route the preemptive 401 existence check through the v2 resolver The discovery-phase 401 no longer calls v1's _get_user_oauth_extra_headers_from_db to decide whether a migrated server has a token; it asks the v2 resolver via a new has_user_oauth_token manager method (to_server_spec + to_subject + resolve_credentials, Ok means a token exists). With this, every authorization_code resolution runs through the v2 resolver: the call_tool egress, the listing connection, and the discovery challenge. Delegate servers short-circuit before the check (the client completes PKCE with the upstream). The challenge itself still emits the RFC 8414 authorization_uri form; the format unification stays a follow-up. * refactor(mcp): extract the authorization_code arm into a helper Mirror the api_key arm's structure: the inline AuthorizationCodeConfig body moves into _authorization_code(subject, server), keeping resolve_credentials a flat one-line-per-arm dispatch. The helper is annotated with the concrete StaticHeaderAuth it returns rather than the abstract httpx.Auth (which api_key uses) because a new method carrying the unresolved httpx.Auth return would add reportUnknownMemberType; the concrete type is both precise and budget-neutral. * fix(mcp): emit the canonical WWW-Authenticate header name in the OAuth challenge raise_user_oauth_challenge emitted the header lowercase while the sibling raise_public and every resource_metadata (RFC 9728) emitter use the canonical WWW-Authenticate; align it. HTTP header names are case-insensitive on the wire so this is cosmetic for compliant clients, but it keeps the challenge builders consistent and matches RFC 6750. * feat(mcp): v2-native per-user token read store (step 1b inner store) Reads the user's persisted authorization_code credential and returns a typed OAuthToken (access token, epoch expiry, refresh token), validating the decoded blob at this boundary so no Any leaks past it. The raw inner store that RefreshingTokenStore/CachedOAuthTokenStore wrap; the DB read + decode collaborator is injected so it stays testable. Not yet wired - V1PerUserTokenStore is still the composition-root store until the refresher and cross-worker cache land. * feat(mcp): v2-native authorization_code token refresher (step 1b) The refresh_token grant for the authorization_code mode: POSTs the RFC 6749 refresh_token grant to the server's token endpoint, persists the rotated triple, and returns the new typed OAuthToken for RefreshingTokenStore to cache. HTTP post and persist are injected so the grant + response parsing are testable without a live IdP/DB. Also extends the TokenRefresher seam with (user_id, server_id), which the foundation's refresh(token) lacked but the grant (server config) and persist (key) need. * feat(mcp): wire the v2-native per-user OAuth store into the resolver (step 1b piece 4) Assemble Cached(Refreshing(V2PerUserTokenStore)) at the composition root and replace V1PerUserTokenStore in mcp_server_manager. The chain is built lazily on first fetch (its cache/DB/ Redis collaborators are LiteLLM globals not ready at import); when Redis is wired it uses the cross-replica path (DualCache cache + SET NX PX coordinator), else the in-process defaults. The DB read, refresh-grant POST, and persist acquire their globals per call like v1. authorization_code resolution now reads/refreshes through the v2-native lifecycle, not v1's core. * refactor(mcp): delete the unwired V1PerUserTokenStore adapter (step 1b piece 5) Piece 4 replaced V1PerUserTokenStore with the v2-native chain at the composition root, leaving the adapter with no callers, so remove it and its test. The shared v1 read/refresh core (resolve_user_oauth_access_token and friends) stays - delegate's egress in server.py still uses it - and comes out with the delegate migration. * fix(mcp): green CI for authz_code dispatch (format + UTC expiry + v2-seam tests) - ruff format per_user_oauth_store.py (clears the lint check) - v2_token_store._iso_to_epoch: anchor a tz-naive expiry to UTC before .timestamp(), matching v1's db.py _remaining_token_seconds (Greptile P1) so a non-UTC host doesn't read the expiry as local time and skew refresh timing - test_mcp_stale_session: repoint the 3 discovery tests off the removed v1 _get_user_oauth_extra_headers_from_db onto the v2 has_user_oauth_token seam; the delegate test now asserts the existence check is never consulted (delegate short-circuits to the resource_metadata 401 before any token lookup) - test_mcp_server_manager: repoint test_deferred_mode_uses_v1_auth_value at M2M (oauth2 client_credentials), which is still a deferred mode, since per-user oauth2 (authorization_code) now routes to the v2 resolver Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): caller Authorization must not override the stored per-user OAuth token A caller with a valid x-litellm-api-key could include their own "Authorization: Bearer <chosen>" header and have the proxy execute tools against that bearer instead of the user's stored OAuth credential. For a v2-migrated authorization_code server the caller's Authorization was seeded into extra_headers, and the graft's apply-if-absent then dropped the resolved per-user token in its favor. v1 prevented this by overwriting a stale client Authorization with the stored token; this restores that precedence on both egress paths (connect + call_tool). - _should_strip_caller_authorization: also strip for migrated per-user OAuth (authorization_code) servers - the v2 resolver injects the stored token, so a caller-forwarded Authorization must not be forwarded upstream. Delegate / pass-through (to_server_spec is None) keep forwarding the caller's bearer. - both seed sites (_prepare_mcp_server_headers, _call_regular_mcp_tool) drop only the Authorization from the caller's oauth2_headers (via _without_authorization), keeping any other forwarded header and any hook/static Authorization (which still wins, as in v1). - regression test for the call_tool path; updated the two tests that asserted the old (vulnerable) forwarding to assert the secure behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): preserve recorded OAuth scopes across authorization_code refresh When a refresh response omits `scope` (RFC 6749 §5.1, where omission means unchanged), the v2 refresher persisted scopes=None and overwrote the user's recorded grant. v1 carried the prior scopes forward via `or cred.get("scopes")`; the v2 path lost that because OAuthToken did not model scopes OAuthToken now carries scopes, V2PerUserTokenStore populates them on read, and AuthorizationCodeRefresher carries them forward for both the persisted write and the returned/cached token, so repeated refreshes do not erode them. A present `scope` in the response still replaces the prior grant Adds regression tests: a refresh omitting `scope` preserves the prior scopes, and a present `scope` overrides them * fix(mcp): keep user token in authorization_code tools preview After to_server_spec maps oauth2 onto the v2 resolver, the interactive tools preview for an unsaved authorization_code server read the per-user token store, found nothing, and fail-closed with a 401, so the create/test tab could no longer list tools The preview now routes the just-authorized token (forwarded in oauth2_headers) through mcp_auth_header, so _create_mcp_client takes the per-request-override v1 path and uses it directly, matching v1's preview. Gated to the v2-mapped oauth2 case; M2M, delegate/passthrough, and token-exchange keep their existing preview path Adds tests: interactive oauth routes the forwarded token to mcp_auth_header, M2M and token-exchange do not * fix(mcp): stop caller-supplied auth from overriding stored authorization_code tokens A caller-supplied per-request override (mcp_auth_header / x-mcp-auth / x-mcp-<alias>-authorization) disabled the v2 resolver in _create_mcp_client for any spec, so an authenticated user with a stored authorization_code token could force an arbitrary upstream bearer and bypass the stored credential and its save-time validation. _create_mcp_client now keeps the v2 spec for authorization_code and ignores the override; other modes keep the client-side-credentials override The create/test tools preview no longer relies on that override path. It resolves the just-authorized, not-yet-persisted token through the v2 resolver via a one-shot PresentedOAuthTokenStore passed as cred_provider - the same path runtime uses for the stored token - so preview and runtime resolve identically. This replaces the mcp_auth_header routing added earlier Adds tests: a caller override cannot bypass the v2 resolver for authorization_code; the interactive preview resolves via the presented store rather than a caller header; M2M and token-exchange build no presented provider * feat(mcp): cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] (#31474) * feat(mcp): encrypt+serialize codec for caching OAuth tokens in Redis (step 1b §1.5) The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss. * feat(mcp): DualCache-backed token cache backend (step 1b §1.5) The cross-replica TokenCacheBackend implementation that plugs into the foundation's CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected; a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss. * feat(mcp): Redis SET NX PX refresh coordinator (step 1b §1.5) The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis. * feat(mcp): Redis SET NX PX distributed lock (step 1b §1.5) The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh), release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired / not-held so a cache blip causes an extra refresh, never a crash on the resolve path. * feat(mcp): wire the cross-replica cache + coordinator into the per-user store (step 1b §1.5) Upgrade the composition root to use the DualCache-backed cache and SET NX PX refresh coordinator when Redis is wired, falling back to the foundation's in-process defaults on a single replica. Layers the cross-replica path on top of the single-replica dispatch store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): refresh on lock-backend error instead of serving a stale token The cross-replica refresh coordinator elected refreshers with a boolean acquire: a Redis transport error was caught and returned as False, which is indistinguishable from "another worker holds the lock". On a total Redis outage every worker therefore took the wait-then-reread branch and served the still-expired token upstream (the upstream then 401s), even though the lock and coordinator docstrings claimed a Redis blip "degrades to an extra refresh". Make acquire tristate (LockAcquisition: ACQUIRED / HELD / ERROR) so the coordinator can tell a busy holder from a dead backend, and refresh anyway on ERROR. This single-flight lock is a load optimization, not a correctness mutex, so failing open is correct: it degrades a lock-backend outage to the no-coordinator behavior (an extra refresh), never a stale bearer. Add a regression test asserting an acquire error refreshes rather than re-reading the expired token, and update the docstrings to match. * style(mcp): wrap redis lock signatures at line-length 88 for CI ruff format * fix(mcp): a refresh loser surfaces None, not a stale token, when the winner failed The cross-replica coordinator's losers re-read the token the winner persisted. If the winner's refresh failed, the store still holds the expired token, so the loser re-read it and RefreshingTokenStore handed that expired bearer to the caller (the upstream then 401s) instead of the re-auth challenge the winner returned via None. Make the loser's re-read expiry-aware, mirroring refresh_latest_token: a re-read that is still expired surfaces None so the arm challenges. This only affects the loser path; the winner's freshly refreshed token is returned directly by the coordinator and is unaffected. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "feat(mcp): cross-replica single-flight refresh for the v2 per-user OA…" (#31492) This reverts commit cd2fb6b0f2. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 21:19:57 -07:00
yuneng-jiang	e4aedb0342	Merge pull request #31391 from BerriAI/litellm_multipart_file_upload ... fix(passthrough): forward all multipart files with repeated field names	2026-06-26 19:54:44 -07:00
Krrish Dholakia	99b1a323c1	feat(guardrails): add headroom guardrail for message compression (#31407) ... * feat(guardrails): add headroom guardrail for message compression Adds a headroom guardrail that compresses request messages via POST /v1/compress before they reach the LLM. The guardrail implements apply_guardrail so it runs on the unified guardrail path; it receives pre-built structured_messages (OpenAI format) from the translation layer, calls the headroom compression service, and returns the compressed messages as structured_messages. Set x-headroom-bypass: true on the request to skip compression. Also adds structured_messages write-back support to the OpenAI and Anthropic translation handlers: when apply_guardrail returns structured_messages, those are written to data["messages"] directly (OpenAI) or reverse-translated via anthropic_messages_pt (Anthropic) instead of falling through to the existing text-patch path. This is a prerequisite for any guardrail that needs to replace the full message list rather than patch individual text spans. * fix(guardrails/headroom): add @log_guardrail_information to populate guardrail_information in spend logs * style: fix ruff format violations * fix(lint): replace deprecated typing aliases with builtin generics (UP006/UP037) * fix(guardrails): only write back structured_messages when guardrail actually changed them * fix(guardrails/headroom): raise 502 when compression returns empty message list * fix(guardrails/headroom): catch transport errors and fix stale debug log * fix(guardrails/anthropic): strip system messages before anthropic_messages_pt reverse-translation * fix(guardrails/anthropic): strip cache_control from thinking blocks after write-back * debug(headroom): add INFO logging to trace guardrail execution * debug(headroom): use print() for immediate visibility * debug(headroom): print request_data keys to diagnose metadata dict mismatch * fix(guardrails/anthropic): propagate guardrail info to logging_obj.metadata for spend log * fix: use model_call_details litellm_params metadata on Logging object * fix(guardrails/anthropic): write guardrail info to litellm_params attr not model_call_details copy * fix: read slg_info from litellm_metadata when metadata key absent * fix: write slg_info to both litellm_params attr and model_call_details copy * chore: remove debug prints; fix now verified end-to-end * refactor(guardrails): move spend-log sync to shared helper in custom_guardrail.py - Add _sync_guardrail_info_to_logging_obj in custom_guardrail.py; call it from both async and sync wrappers in @log_guardrail_information, fixing guardrail_information=null in spend logs for all passthrough routes (/v1/messages, /v1/responses, etc.) in one place - Remove the 35-line inline sync block from the anthropic translation handler - Wrap response.json() in try/except in headroom.py to 502 on HTML/truncated responses - Drop redundant headers.get(BYPASS_HEADER.lower()) — header key already lowercase - Add regression tests for _sync_guardrail_info_to_logging_obj * fix(lint): reduce _sync_guardrail_info_to_logging_obj complexity below C901 threshold * fix(lint): simplify _sync_guardrail_info_to_logging_obj to reduce McCabe complexity * fix(lint): extract _append_slg_to_litellm_params to reduce McCabe complexity * fix(lint): extract _write_back_structured_messages to reduce process_input_messages complexity	2026-06-26 19:36:44 -07:00
Mateo Wang	4157f3b580	fix(passthrough): schedule spend logging via durable logging worker (#31485) ... Pass-through success logging was scheduled with a bare asyncio.create_task whose return value was discarded, for non-streaming HTTP, streaming, and the vertex live websocket paths. The event loop keeps only a weak reference to such tasks, so under GC or load the task can be collected before it finishes writing the SpendLogs row; a request then returns 2xx to the caller yet never produces a costed spend log. This is the most likely cause of the flaky vertex passthrough e2e test and a rare real source of unbilled pass-through spend. Route these coroutines through GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue instead, matching how the SDK completion path already enqueues async logging. The worker holds a strong reference in its _running_tasks set and drains on shutdown via flush/stop/clear_queue and the atexit handler, so the write can no longer be dropped mid-flight.	2026-06-26 18:45:10 -07:00
tin-berri	2e69708ef8	feat(mcp): shared OAuth token foundation - challenge, store seam, expiry-aware cache, single-flight refresh (#31275) ... * feat(mcp): let CredError.of_unauthorized carry a 401 challenge The unauthorized case becomes a structured Unauthorized (detail + optional WWW-Authenticate header + optional structured body) instead of a bare string, and raise_public emits the header and body when present. This lets a mode reproduce a rich 401 challenge (e.g. BYOK's provisioning prompt) through the generic resolver edge. of_unauthorized's new params are keyword-only and default to None, so existing callers and the summary string are unchanged. * fix(mcp): make Unauthorized a frozen dataclass to keep the type budget flat CredError's unauthorized payload was a pydantic BaseModel, whose base resolves as unknown in this repo's basedpyright (every model in the file trips reportUntypedBaseClass plus an unknown model_config), so the tagged-union case read as unknown and the public edge's challenge access added reportUnknownMemberType errors over the per-rule ceiling. A frozen dataclass is fully typed here, so error.unauthorized resolves directly with no cast or accessor and the per-rule basedpyright counts match base. * feat(mcp): OAuth token store seam + expiry-aware cache for authorization_code Lay the foundation for the authorization_code resolver arm: OAuthToken (access_token, expires_at, refresh_token), the OAuthTokenStore Protocol seam, TokenStoreUnavailable for outages, and CachedOAuthTokenStore, an expiry-aware cache that serves a token only while unexpired, caches the "not authorized" None for a default TTL, and propagates a store outage without caching it. Mirrors the BYOK store/cache pattern, adapted for tokens. Refresh and distributed single-flight are deferred to the hardening step. * feat(mcp): proactive token refresh with self-cleaning single-flight Add TokenRefresher (a mode-supplied seam: mint a fresh token from an expired one and persist it) and RefreshingTokenStore: when the stored token is near expiry, the first caller refreshes while concurrent callers await the same in-flight task and share its result, so the IdP is not stampeded. The task self-cleans (a done-callback drops its entry), so the map is bounded by in-flight refreshes rather than by distinct users/servers, and is detached from the caller so a cancelled caller does not abort the refresh. An expired token the refresher cannot renew surfaces as None so the arm challenges, never a stale bearer; it composes under CachedOAuthTokenStore. OAuthToken's repr masks the access/refresh tokens so a stray log cannot leak them. Cross-replica single-flight (Redis) and reactive-401 refresh are the later distributed hardening. * style(mcp): modern type annotations (dict/tuple/X | None) + sorted imports in the token modules * refactor(mcp): FIFO cache eviction, fix stale single-flight comment + refresh_token docstring * refactor(mcp): cache positive tokens only, matching v1 (no negative caching) CachedOAuthTokenStore no longer caches the "not authorized" None result; every miss re-reads the inner store. v1's per-user token cache never caches misses, so a token written by the OAuth flow is visible on the next request without an invalidation hook, and uniformly across replicas since the in-process cache holds no stale None to clear. invalidate() now only covers rotation or revocation of a cached token. Negative caching (with distributed invalidation) can return later if a slow DB-backed v2-native source makes per-miss reads expensive. * fix(mcp): default OAuth expiry skew to 60s, the industry standard The proactive token-refresh / cache-expiry buffer defaulted to 30s, which is an outlier among OAuth clients. Spring Security uses 60s as both its JWT clock-skew tolerance and its refresh buffer, and 60s sits inside RFC 7519's "a few minutes" leeway while preserving nearly all of a typical token's life; 30s was untested, so pin the default with two boundary-probe regression tests. * refactor(mcp): thread user_id/server_id through the TokenRefresher seam The refresh seam took only the OAuthToken, but a refresher needs the server's config (token endpoint, client credentials, scopes) to run the grant and the (user_id, server_id) key to persist the minted token, neither of which is derivable from the token. Widen TokenRefresher.refresh to (user_id, server_id, token) and pass them through from RefreshingTokenStore so each stacked mode PR plugs into the final seam rather than forcing a later signature change across the stack. * feat(mcp): inject cache-backend and refresh-coordinator seams (cross-replica token caching) Make CachedOAuthTokenStore's storage and RefreshingTokenStore's single-flight injectable so a cross-replica deployment can back them with Redis without touching the resolver. The defaults preserve today's behavior exactly: InMemoryTokenCacheBackend (the bounded per-process dict) and InProcessRefreshCoordinator (the asyncio single-flight). A distributed deployment injects a shared DualCache-backed backend and a SET NX PX coordinator. invalidate() is now async (the backend may be). The cache stores via the backend with a TTL derived from the token's expiry; the coordinator threads a reread callback for the cross-replica case (losers re-read the persisted token) that the in-process default ignores. * fix: reread oauth token before refresh --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-26 18:27:18 -07:00
Yassin Kortam	c14329128b	fix(guardrails): match policy-pipeline block response to direct guardrail attachment (#31421) ... When a guardrail blocked a request through a flow-builder policy pipeline, the proxy discarded the guardrail's own exception and synthesized a generic guardrail_pipeline_error response, so the same guardrail produced a different HTTP response and trace span depending on whether it was attached directly or via a policy. The pipeline now carries the guardrail's original exception and re-raises it verbatim on block, enriching it with the blocking guardrail's name and mode exactly as the direct path does, so the two attachment methods are indistinguishable to clients and tracing. The generic pipeline error remains only as a fallback for blocks with no underlying exception (e.g. a guardrail that could not be found). Resolves LIT-4041	2026-06-26 14:25:10 -07:00
Yassin Kortam	ce658367a4	fix(auth): cache auth-path team object under canonical team_id key (#31418) ... The auth builder cached the team object under the raw `valid_token.team_id`, while `get_team_object`, `_cache_team_object`, and `_update_team_cache` all read and write under `team_id:{id}`. The raw-key write was therefore never served back, and on a non-team (personal) key, whose team_id is None, the original unguarded version passed a None key straight to the cache layer; the in-memory cache tolerates None keys but Redis rejects them with a NoneType key error, so with `enable_redis_auth_cache: true` the team object never reached the L2 cache and every request fell back to Postgres. Write under the canonical `team_id:{id}` key, keeping the existing guard that skips the write when team_id is None. Add a regression test that drives the real auth builder for a team-scoped key against an in-memory cache and asserts the team object is served back under `team_id:{id}` and never under the raw team_id or a None key. Resolves LIT-4000	2026-06-26 23:36:50 +03:00
Yassin Kortam	f2fa23b0ec	fix(guardrails): instrument during-call and post-call guardrail latency (#31414) ... litellm_guardrail_latency_seconds was only emitted for pre-call guardrails. during_call_hook and post_call_success_hook ran guardrails without recording any latency, so during-call and post-call guardrail time was invisible in the metric and leaked into litellm_overhead_latency_metric, making the documented "subtract guardrail latency from overhead" workaround under-report total guardrail time. Extract the find-the-PrometheusLogger-and-record step into _emit_guardrail_metrics and add _run_guardrail_with_metrics, a single wrapper that times a guardrail coroutine, classifies its outcome (success / intervened / error), enriches any raised HTTPException, and records the latency under the given hook_type. Route the pre-call emit, during_call_hook, and post_call_success_hook through it so every guardrail phase contributes to the metric the same way. Resolves LIT-3999	2026-06-26 13:07:50 -07:00
Yassin Kortam	7209e139d6	fix(spend): fold logs-tab total into the page query to avoid a separate COUNT(*) (#31423) ... The spend-logs UI list endpoint (/spend/logs/ui, /spend/logs/v2) ran a standalone SELECT COUNT(*) before the page query to compute total_pages. On sharded engines like YugabyteDB a COUNT(*) is a distributed RPC that contacts every tablet leader and aggregates partial results regardless of row count, so it hits the distributed RPC timeout and the logs tab 500s even on a one-minute window with a couple of rows. The startTime range cannot prune tablets because rows hash to tablets on request_id, not startTime. Fold the count into the same scan as the page data with COUNT(*) OVER () and read total off the returned rows, dropping the helper column before serialisation. One distributed scan per page load instead of two; the response shape is unchanged. An empty page carries no count row, in which case the total is zero. Resolves LIT-4027	2026-06-26 13:01:05 -07:00
Yassin Kortam	f55d13ebba	fix(team): persist budget_duration on /team/member_add member budgets (#31443) ... /team/member_add could not set budget_duration on an individual member budget. add_new_member created the budget row with only max_budget and allowed_models, and TeamMemberAddRequest had no budget_duration field, so a member added with an explicit per-member budget while the team ran a recurring member budget got a lifetime cap instead of a recurring allowance. Thread budget_duration from TeamMemberAddRequest through _process_team_members into add_new_member, and pull the member-budget resolution into a helper that writes budget_duration plus a computed budget_reset_at. When only a budget_duration is supplied and the team has a default member budget, the default is cloned and its reset window overridden so the member keeps the default's max_budget rather than becoming uncapped; a duration with no team default creates a window-only budget. Invalid durations are rejected with a 400 before any DB write, symmetric with /team/member_update. The available-team self-join bypass only grants the ability to join, so reject per-member budget and model controls (max_budget_in_team, budget_duration, allowed_models) for non-admin self-join callers in _validate_team_member_add_permissions, before any DB write. Otherwise a self-joining non-admin could set their own cap, reset window, or model scope past the team default; admins, team admins, and org admins are unaffected and a clean self-join still inherits the team default budget. Resolves LIT-4052	2026-06-26 12:59:30 -07:00
yucheng-berri	e99151bb95	feat(guardrails): make the Generic Guardrail resilient to built-in tools and errors (adopted from #31286) (#31461) ... * fix(guardrails): stop Generic Guardrail API 500 on built-in tools Requests carrying built-in tools (code_interpreter, file_search, ...) crashed the Generic Guardrail with a 500. GenericGuardrailAPIRequest.tools validated each tool against ChatCompletionToolParam, whose base TypedDict requires a function block, so a tool like {"type": "code_interpreter"} raised a Pydantic ValidationError before the request was ever sent. Type the field with a permissive GuardrailToolParam model (type required, extra=allow) so built-in tools validate and their config is forwarded to the guardrail intact instead of being stripped. * feat(guardrails): add complete fail-open (fail_on_error) to Generic Guardrail The Generic Guardrail already honored unreachable_fallback, which fails open only on network-unreachable errors. This wires up the existing generic fail_on_error config (so far implemented only by Model Armor) so that fail_on_error=false degrades any guardrail error to a critical-log warning and lets the request proceed as if the guardrail were absent. Only a valid guardrail response can act: a parsed BLOCKED decision still raises, while endpoint errors, malformed responses, and internal serialization or validation errors all fall through when fail_on_error=false. To cover that last class, the request construction now runs inside the protected block, so the kind of validation error that previously surfaced as a 500 is caught here too. Defaults to true (fail closed), matching today's behavior; turning it off is an explicit availability-over-security choice and is logged at critical level on every bypass. * test(guardrails): cover fail_on_error on the response path The existing fail_on_error tests all drive the request path. Add response-path (input_type=response) coverage: an endpoint error proceeds unchanged under fail_on_error=false, and a valid BLOCKED decision still raises. Guards against a future regression that special-cases input_type in the error handling. * style(guardrails): black-format the fail-open guard expression CI runs black (line-length 88) over litellm/; the unreachable_fail_open assignment exceeded it. Wrap it to satisfy the formatter. * fix(guardrails): validate tools into GuardrailToolParam at the call site Changing the request field to List[GuardrailToolParam] left the construction passing List[ChatCompletionToolParam] (list is invariant), which tripped the basedpyright reportArgumentType budget gate. Validate each tool explicitly, which is what Pydantic did implicitly, so the types line up with no Any or suppression and the serialized payload is unchanged. * fix(guardrails): make fail-open log message accurate for non-network errors The fail-open path is now shared by fail_on_error, so it fires for any guardrail error, not just unreachability. The log said 'unreachable' even for an HTTP 400 or a malformed response; reword to 'error' (the status code and exception are already logged). Addresses the Greptile review's only finding. * fix(guardrails): align GenericGuardrailAPIResponse.tools with GuardrailToolParam Greptile flagged that the request side moved to GuardrailToolParam but the response side still annotated tools as List[ChatCompletionToolParam], which mandates a function block and contradicts the new built-in-tools support. Update the response annotation (and the now-unused import) so the two sides agree. Runtime is unchanged; from_dict stores the raw dicts and the only consumer assigns through to GenericGuardrailAPIInputs without inspecting the elements. --------- Co-authored-by: Itay Ovadia <itay@sun.security>	2026-06-26 11:25:56 -07:00
Sameer Kankute	133da06aa3	chore: litellm oss staging (#31185) ... * fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped The Total Tokens Over Time and Total Requests Over Time AreaCharts on the Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow once totals pass the hundred-million mark — leading digits of labels like "100.00M" / "4500.00M" got clipped against the chart edge. The requests chart was worse: it formatted with toLocaleString(), so billion-scale request counts produced "1,000,000,000" (13 chars) and overflowed immediately. Fix in two places so neither alone has to carry the whole margin: - activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and switch the requests chart to the shared valueFormatter so it uses the same compact k/M/B suffixes as the tokens chart. - value_formatters.tsx: add a >= 1e9 branch to valueFormatter / valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping every formatted label at most 7 chars. Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com> * Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * docs(readme): add Deploy on AWS/GCP with Terraform section Adds a quickstart for the two published Terraform modules on the public registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste main.tf for each cloud, the one-time GCP Artifact Registry remote-repo command, and pointers to the registry pages for the full input surface. Sits inside the Get Started section, between the gateway/SDK table and Run in Developer Mode -- where someone scanning the README for "how do I deploy this" will land. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): add 1-click deploy buttons for AWS + GCP GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo and walks through `terraform apply` via the existing DeployStack tutorial (already shipped at terraform/litellm/gcp/examples/default/ TUTORIAL.md). User just picks a project. AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an in-browser, already-authenticated shell. User runs four commands (clone + cd + cp tfvars + terraform apply) once inside. There's no native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN "Launch Stack" + CodeBuild would be needed for that, and that's a separate piece of work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): move AWS + GCP deploy buttons next to Render button * docs(readme): unify deploy button sizes and badge styles * docs(readme): bump deploy button height to 48 to match Render/Railway * docs(readme): bump AWS/GCP badge height to compensate for SVG padding * docs(readme): bump AWS/GCP badge height to 72 * docs(readme): bump AWS/GCP badge height to 84 * fix(readme): make deploy buttons same height (48px) https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc * docs(readme): flag GCP project ID substitution in image_registry * docs(readme): equalize deploy button heights and fix Cloud Shell button font GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px. Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface. * docs(readme): collapse Railway deploy anchor to a single line The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline. * Add Claude Fable 5 cost map entries as a data-only hotfix Backports only the model map changes from #30064 so deployments on released litellm versions pick up Fable 5 pricing, context window, and the adaptive thinking flag through the hosted cost map fetch without upgrading. Includes the supports_sampling_params flag on the 28 Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read by the gating that ships with the next release) and the matching one-line schema declaration so the map validation test passes. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano Three bugs in model_prices_and_context_window.json: 1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K max output, but the values were set as max_input=128000, max_tokens=272000. This caused token limit errors when sending prompts over 128K tokens to GPT-5 Pro. 2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was 272000, but GPT-5.4 Mini shares the same 1,050,000 token context window as GPT-5.4. This was inconsistent with the azure/ variants which already correctly had 1,050,000. 3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini, max_input_tokens was 272000 instead of 1,050,000. Source: OpenAI model documentation and contextwindows.dev which aggregates official context window sizes. Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini should be 400K; their 272K values are correct per OpenAI docs) * fix: also correct max_output_tokens for gpt-5-pro (272000→128000) Per reviewer feedback, max_output_tokens was left at 272000 while max_tokens was corrected to 128000, causing an internal inconsistency. Both should be 128000 per OpenAI docs. * fix(cost): price gpt-image generated output tokens as image tokens (#31147) The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return usage with no output token breakdown — litellm's `ImageUsage` has no `output_tokens_details` field — so generated-image OUTPUT tokens were priced at the text rate (`output_cost_per_token`) instead of the image rate (`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x undercount on the dominant cost component (image output is ~74% of spend). This also affects azure gpt-image, which shares this calculator. The OpenAI gpt-image cost calculator re-implemented usage handling instead of reusing `calculate_image_response_cost_from_usage`, the shared helper that azure_ai/gemini/vertex_ai already use. That helper classifies generated output tokens as image tokens when the provider does not itemize output, and splits text/image when it does. Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage` (pre-transformed chat Usage objects are still costed directly). Adds a regression test for the no-breakdown ImageUsage case (gpt-image-2). * fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098) A bare application-inference-profile ARN passed as bedrock/arn:... fell through to the invoke route, which cannot derive a provider from the opaque profile id and raised 'Unknown provider=None'. The converse route needs no provider, so detect these ARNs in get_bedrock_route and route them to converse, matching the behavior of the already-documented bedrock/converse/arn:... workaround. Explicit invoke/ prefixes still win, and they remain a dead end for these ARNs by design (no provider derivable). System-defined inference-profile ARNs that embed a known model, and other opaque ARN types (provisioned-model, imported-model, custom-model-deployment) that are frequently invoke-only, are deliberately left on their current routes; tests guard both boundaries. * fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060) _add_tool_choice_required_message appended the "select a tool" prompt to the caller's messages list in place, so transform_request corrupted the caller's conversation history and appended a duplicate prompt on every retry. Build and return a new list instead so the call stays idempotent. Adds a regression test asserting the input messages list is unchanged across repeated transform_request calls. Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com> * fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996) gpt-4o-transcribe and compatible ASR backends return a diarized_json response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8. TranscriptionUsageDurationObject typed seconds as int, so parsing the response raised a pydantic ValidationError (int_from_float). That error surfaces as an APIConnectionError which the router treats as retryable, so it keeps re-calling the upstream (200 every time) until the upstream rate-limits and returns 429 to the caller. OpenAI specs this field as a float (see openai SDK UsageDuration.seconds), so widen seconds to float. With the parse succeeding there is no exception left to retry, which removes the loop. Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> * fix(deepseek): drop non-function tools before chat completions call (#30910) * fix(deepseek): drop non-function tools before chat completions call DeepSeek's /chat/completions only accepts tools of type "function". Requests bridged from /v1/responses can carry responses-API-native tool types, for example a Codex CLI tool typed "namespace", which DeepSeek rejects with "unknown variant 'namespace', expected 'function'" so the whole request fails (issue #30722). Filter unsupported tool types in the DeepSeek request transform so the function tools still go through; when nothing callable remains, also drop the now-dangling tool_choice and parallel_tool_calls Fixes #30722 * test(deepseek): cover async tool filtering and document tool_choice assumption Add an async_transform_request regression test so the sync and async tool filtering paths cannot silently diverge, and document in _drop_unsupported_tools that only non-function tools are dropped, so a function-named tool_choice always references a surviving tool * feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840) * feat(ui): surface team budget on key overview when key has no own budget (#30801) * feat(ui): surface team budget on key overview when key has no own budget * fix(ui): replace IIFE with derived variable and use find() for team budget display * fix(anthropic): emit replayable streaming thinking blocks (#31022) * feat(proxy): read cold-storage prompts back in the logs detail view (#30364) * feat(proxy): read cold-storage prompts back in the logs detail view When a deployment offloads prompts and responses to cold storage instead of Postgres, the spend-log row holds only "{}" placeholders plus a metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed nothing. The detail endpoint only read the placeholder columns and never fetched the object back. Resolve the payload per row based on actual content, not a config flag: if Postgres has content, return it; otherwise read the exact stored object key and fetch from the configured cold storage backend through ColdStorageHandler. Reading the persisted key is a single GET. The key embeds a microsecond timestamp that cannot be reconstructed from the millisecond-precision startTime column, and listing the day's prefix to match on request_id would be too expensive for this per-open path. Also teach the detail drawer's pretty-view parser to accept a bare messages array. The cold storage payload carries the prompt as a top-level messages list with no proxy_server_request, so without this the output rendered while the input stayed blank. ColdStorageHandler gains an optional injected logger so the resolver can be unit tested without monkeypatching. Postgres-stored prompts are unaffected: the fast path returns the existing columns and the request-body object still renders the same way. * Update litellm/proxy/spend_tracking/spend_management_endpoints.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure Add unit tests for ColdStorageHandler (injected logger, graceful None when no logger is configured, and resolution of a configured logger from the callback registry) and a regression test asserting a cold storage backend exception degrades to the Postgres values instead of surfacing a 500. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068) * fix(mavvrik): advance metricsMarker after upload + fix scheduler startup Two bugs fixed: 1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a successful GCS upload, so metricsMarker stayed at 0 and every daily run re-exported the same dates in an infinite catch-up loop. Fix: add _update_metrics_marker(date_epoch) called at the end of deliver() after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS file is already committed). A 410 raises consistent with the rest of the destination. 2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call has triggered lazy instantiation of MavvrikFocusLogger, so it found no logger instance and silently skipped registering the daily export job. Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call _init_custom_logger_compatible_class to force instantiation before the APScheduler job is registered. * fix(mavvrik): catch up from earliest window when metricsMarker=0 When the connector is freshly registered, metricsMarker=0 parses to None. The catch-up block was guarded by `if last_ingested and ...` which skipped it entirely for None, so only yesterday was exported instead of the full _MAX_CATCHUP_DAYS window. Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup). The existing > 7 day warning only fires for non-None markers that are old. * fix(mavvrik): use now as end_time for yesterday's export window LiteLLM_DailyUserSpend rows for a given date get their updated_at bumped by the spend flush job throughout the next morning. The core database query filters on updated_at, so capping end_time at midnight (yesterday + 1 day) missed any spend rows flushed after midnight. Fix: pass now (cron fire time) as end_time for the daily "yesterday" window so all fully-settled rows are captured regardless of when the flush job ran. Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per row in the exported FOCUS CSV. * fix(mavvrik): also use now as end_time for catch-up windows * fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class Calling it with only logging_integration raised TypeError at proxy startup because internal_usage_cache and llm_router have no defaults. Also fix test name to reflect the actual status code (5xx not 4xx) used in the mock. * ci: retrigger CI run * feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757) * Add optional `instruction` passthrough to the rerank API vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction` field (folded into the model's chat_template_kwargs and consumed by the chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently dropped it: RerankRequest / OptionalRerankParams had no such field, so the outgoing body was rebuilt without it. Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(), get_optional_rerank_params, and the hosted_vllm transformation into the request body, only when non-None. When callers omit it, model_dump(exclude_none) drops the field and the outgoing request is byte-for-byte unchanged — fully backward-compatible. (DeepInfra already forwards `instruction` via non_default_params; this formalizes the field in the shared types.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: thread `instruction` as a typed param + cover rerank_utils Per PR review (greptile P2 + codecov): - Make `instruction` a typed, named argument on the rerank provider interface instead of recovering it from the opaque `non_default_params` blob. Adds `instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params` and every provider override, and forwards it explicitly from `get_optional_rerank_params`. hosted_vllm now reads the named param directly. It is still also surfaced in `non_default_params` so providers that read it there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction` as a named param rather than leaving it in **kwargs. - Add get_optional_rerank_params unit tests (present + absent) to cover the previously-uncovered threading line flagged by codecov. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scan rerank `instruction` through request guardrails The rerank guardrail translation (CohereRerankHandler.process_input_messages) only scanned `query`, so the newly added `instruction` field reached the backend model unscanned. Since instruction-aware rerankers (hosted vLLM / Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller could place content there to bypass configured rerank request guardrails. Generalize the handler to scan every user-controlled text field (`query` and `instruction`) in one apply_guardrail call and write each sanitized value back by index. Query-only requests are unchanged (single-element list at index 0); non-string fields are left untouched. Adds tests covering instruction scanning, PII masking write-back, and the non-string case. Addresses the Veria AI security review on PR #30757. * test: narrow Optional results before len() to satisfy basedpyright budget The lint gate (basedpyright delta-vs-base budget) flagged one new reportArgumentType: len(result.results) where results is List[RerankResponseResult] | None. Assert results is not None first to narrow the type before len()/indexing. * fix: read rerank `instruction` from kwargs to satisfy basedpyright budget The basedpyright delta-vs-base gate flagged one new reportArgumentType: the Router forwards rerank calls via an untyped `**kwargs` unpack (`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a typed named param on the public `rerank`/`arerank` entrypoints made pyright check that key against `str | None`, adding an error at router.py with no real safety gain. Read `instruction` from kwargs in `rerank` instead. It remains fully typed where it matters - threaded as a typed argument through `get_optional_rerank_params` and each provider's `map_cohere_rerank_params` (the original Greptile P2 ask). Whole-repo reportArgumentType is back to the base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(github_copilot): synthesize empty choices at the provider seam (#30929) Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with choices=[], either carrying Anthropic-native content blocks or, for the max_tokens=1 probe Claude Code sends, no content at all. github_copilot is dispatched through the OpenAI SDK handler, which calls convert_to_model_response_object directly and never invokes GithubCopilotConfig.transform_response, so the empty-choices guard there surfaced as a 500 Instead of synthesizing choices inside the shared convert_to_model_response_object (which would silently turn empty choices into a fabricated success for every provider), add a no-op transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig overrides it to synthesize choices from Anthropic-native content, reusing its existing parsing, and the OpenAI SDK handler routes its parsed response through the hook before generic conversion. The core utility keeps treating empty choices as an error for all other providers Fixes: https://github.com/BerriAI/litellm/issues/30927 Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com> * fix(router): stop fallback lookups from mutating the router fallbacks config (#30624) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens * test: scope local cost map env var with monkeypatch to avoid test pollution * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764) * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold _mask_value did partial reveal by showing the first visible_prefix and last visible_suffix characters, but for a value whose length was at or below visible_prefix + visible_suffix (8 by default) it returned the value verbatim. A value of exactly 8 chars fell through the length guard and computed masked_length == 0, reconstructing the original string with no mask characters; anything shorter hit the early return. Either way short credentials were emitted in plaintext. mask_dict routes real secrets through this path, so an 8-char-or-shorter redis password, api key, or token could be written to logs and the UI unmasked. The sibling helper mask_sensitive_keys already guards this case; _mask_value now does the same by fully masking any value at or below the threshold. * fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers Fully masking short values is the right default for secret masking, but CooldownCache reuses the masker purely to truncate exception messages to the first 50 characters, and it relies on short messages being returned readable. Masking those blanked out short exception text and broke its tests. Add a mask_short_values flag (default True, secure) and have CooldownCache pass False so it keeps the truncation behavior, while every secret-masking caller still gets short values fully masked. * fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview MCPDebug uses the masker to preview auth tokens in debug headers and documents that values of 10 chars or fewer are shown unchanged so token types stay distinguishable. Pass mask_short_values=False so that diagnostic behavior is preserved while secret maskers keep masking short values. * fix(mcp_debug): mask short auth values in debug headers instead of echoing them Earlier this masker opted out of short-value masking to keep a token preview, but that echoes short authorization and token values verbatim in debug response headers, which is the same leak this change is meant to close. Auth material should never be emitted in full, so mask short values here too; the first/last character preview still applies to longer tokens. Only CooldownCache keeps the opt-out, since it truncates exception text rather than masking secrets. * test(mcp_debug): assert masked short value preserves length * refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917) Fireworks AI deprecated audio inference on 2026-06-10 (https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation). Live API testing confirms the endpoint is already non-functional: a valid Fireworks API key receives HTTP 401 "Unauthorized" from api.fireworks.ai/inference/v1/audio/transcriptions for every request, regardless of payload. The audio-prod.api.fireworks.ai host referenced in the test suite returns 401 for every path; the entire host is decommissioned. Remove the dead FireworksAIAudioTranscriptionConfig class and every reference to it across the codebase: - Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line config class that inherited from OpenAIWhisperAudioTranscriptionConfig) - Remove the Fireworks branch from ProviderConfigManager.get_provider_audio_transcription_config() in litellm/utils.py; update the stale comment in get_optional_params_transcription that referenced fireworks ai - Remove the FireworksAIAudioTranscriptionConfig entries from LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in litellm/_lazy_imports_registry.py - Remove the TYPE_CHECKING re-export in litellm/__init__.py - Remove the transcription branch in the fireworks_ai case of get_supported_openai_params() in litellm/litellm_core_utils/get_supported_openai_params.py - Remove the whisper-v3 and whisper-v3-turbo entries from model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json (both had mode: audio_transcription and zero-cost pricing) - Remove the TestFireworksAIAudioTranscription test class and its imports from tests/llm_translation/test_fireworks_ai_translation.py No other provider is affected. The openai_compatible_providers list, FireworksAIMixin, and the OpenAI Whisper transcription handler all stay because they are shared with other Fireworks endpoints and other providers. The provider_endpoints_support.json registry already had audio_transcriptions set to false for fireworks_ai. * feat: add darkbloom provider (#30876) * feat: add darkbloom provider * fix: document darkbloom provider endpoints * fix: address darkbloom review feedback * fix: update darkbloom tool metadata * fix: fail fast for non-Postgres database URLs (#30883) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging * Validate DIRECT_URL alongside DATABASE_URL startup guards * fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946) * fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608) * test(bedrock): mid-stream server errors trigger streaming fallback (#24608) * style(bedrock): black-format stream-error helper (#24608) * fix(mcp): re-land native tool preservation with typed annotations (#30645) * fix(mcp): preserve native tools in semantic filter hook with typed annotations * fix(mcp): tighten _is_mcp_tool Chat Completions shape check * fix(sambanova): return embeddings supported params instead of dropping them (#30937) * fix(router): send fallback metadata when streaming (#30914) When a streaming request triggers a fallback, there was previously no way to know it happened. This commit addresses this in a few ways: 1. The response now correctly populates the fallback headers (`x-litellm-attempted-fallbacks`) so callers know a fallback happened. 2. The correct model ID is passed in the streaming chunks. 3. A streaming chunk with the fallback error can be optionally sent back to the client (opt-in) by passing `include_fallback_errors: true` in the request. The format of the fallback errors while streaming is intentionally OpenAI compatible to not break existing libraries that parse these events. It was tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not delieved unexpectedly to callers by default. * fix(mistral): drop output-only reasoning fields from input messages (#30884) LiteLLM attaches reasoning_content and thinking_blocks to assistant responses. Replaying those assistant turns verbatim forwarded the fields back to Mistral, whose input schema forbids unknown keys, so the whole request failed with a 422 extra_forbidden and reasoning models became unusable across multiple turns. Strip both fields from assistant messages before the request is built, in a spot that runs ahead of the image/file branch so it applies on every path. Fixes #30835 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652) * fix(perplexity): bill search queries at the per-request price, not 1/1000 The fallback cost calculator divided search_context_cost_per_query by 1000, but that field stores the per-request price in USD: sonar is {low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published $5/$8/$12 per 1,000 requests expressed per request. The gemini cost calculator reads the same field per request with no division (its docstring calls it "the per-request cost"). The division understated search cost by 1000x on every Perplexity call that falls back to manual calculation (i.e. when the API does not return a pre-computed usage.cost). Use the value directly. Update the tests that had encoded the /1000 factor in their expectations, and drop an unused import flagged by ruff in the touched test file. * test(perplexity): update integration test search-cost expectations to per-request The integration tests still encoded the old /1000 search-cost factor, so they failed once the fallback calculator was corrected to bill search_context_cost_per_query per request. Update the four expected-cost computations (and the high-volume dollar-value comments) to match. * test(perplexity): drop unused mock imports flagged by ruff * fix: include model_access_groups when expanding all-team-models in get_team_models (#30622) * fix(fireworks_ai): return None for transcription in get_supported_openai_params Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is decommissioned. Without an explicit transcription branch, requests with request_type='transcription' fell through to the else and returned FireworksAIConfig chat-completion params. Return None instead to signal the provider does not support transcription. * fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting Without an operator gate, any authenticated caller could set include_fallback_errors=True, trigger a fallback, and read raw upstream exception messages from the x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event. Strip include_fallback_errors from request data in common_processing_pre_call_logic when expose_fallback_errors_to_caller is not set, so the router never builds the error list. Also gate _should_include_fallback_errors on the same setting as a secondary check for the streaming SSE injection path. * test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test The operator gate added in e7ff3e1 means include_fallback_errors is only honoured when general_settings.expose_fallback_errors_to_caller is True. Set that flag via monkeypatch in the test that exercises the emit path. * test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos test_convert_url called convert_url_to_base64 against a live picsum.photos URL and asserted nothing, so it added no real signal and broke CI whenever the host was unreachable (it was returning 522 and blocking this branch). Replace the live call with a mocked HTTP client and assert the produced base64 data URL, so the conversion path is exercised deterministically with no network dependency. This suite runs under VCR, which is why a transport level mock (respx) does not reliably intercept; mocking the client object itself is robust regardless. * fix(interactions): drop role from Interaction response to match Google spec Google removed the output-only role field from the Interaction schema (it now lives only on Turn), so the live OpenAPI compliance canary started failing with 'role' not in spec. Reconcile our generated types by removing role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse, stop stamping role=model in the responses-to-interactions transformation, and update the compliance and integration tests accordingly. Turn.role is kept since the spec still defines it. * fix: align all-team-models sentinel access * fix(router): forward include_fallback_errors through multi-hop fallbacks run_async_fallback received include_fallback_errors as an explicit named parameter, so it was bound out of **kwargs and never reached the nested async_function_with_fallbacks call. Multi-hop fallback chains (a fallback group that itself fails over) therefore stopped collecting fallback errors beyond the first hop when a caller opted in. Re-inject the flag into kwargs before the nested call so inner hops keep accumulating errors, which add_fallback_headers_to_response already merges across levels. * fix(router): stop fallback lookups from mutating the router fallbacks config get_fallback_model_group resolved a bare-string fallback by popping it out of the fallbacks list it was handed. That list is frequently the live router.fallbacks config, so a single lookup permanently removed the entry and the configured fallback stopped applying to later requests until restart. The pop also ran inside enumerate(), shifting indices and skipping an adjacent string fallback. Read the item instead of popping it, and add a regression test that fails on the old mutating behavior --------- Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> * fix(sambanova): update pricing, deprecate retired models, and add missing models (#30016) * feat(bedrock): add amazon.titan-embed-g1-text-02 embedding model support - Add model to provider routing allowlist in embedding.py - Add request transformation using AmazonTitanG1Config - Add response transformation using AmazonTitanG1Config - Add pricing metadata to model_prices_and_context_window.json - Add unit tests for embedding and model info Fixes missing cost tracking reported in #29786 Related to VANDRANKI/litellm PR #29790 * style: fix syntax error, trailing whitespace and missing newline * style: apply black formatting to embedding.py * style: apply black formatting to test_bedrock_embedding.py * fix(sambanova): update pricing, fix context windows, add deprecation dates, and add missing models * fix(sambanova): sync model_prices_and_context_window_backup.json with primary * fix(sambanova): fix indentation on Meta-Llama-3.2-1B-Instruct deprecation_date * fix(bedrock): add amazon.titan-embed-g1-text-02 to unmapped model error message * style: apply black formatting to embedding.py * fix(sambanova): correct indentation on DeepSeek-V3.2 entry * fix(sambanova): replace gemma-3-12b-it with gemma-4-31B-it (verified pricing) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info (#30880) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info get_model_info rebuilt ModelInfo by copying a fixed allow-list of input/output_cost_per_token_above_<N>_tokens keys (128k/200k/272k/512k), so any other threshold a user registered was dropped before reaching _get_token_base_cost, which already reads an arbitrary threshold out of the key name. Custom tiers such as above_500k_tokens were silently ignored and billing fell back to the base per-token rate. Carry over any _above_<N>_tokens cost key present on the source cost-map entry that the fixed fields miss Fixes #30344 * test(cost): keep suite hermetic by popping the temp tiered-pricing model Wrap the regression body in try/finally so litellm.model_cost no longer leaks the litellm-test-non-standard-tier entry into later tests that iterate or reset the global cost map. Addresses Greptile review thread. * fix: resolve UP045 lint violations (Optional[X] -> X | None) Convert Optional[X] type annotations to X | None syntax across rerank transformations, spend tracking, and other modules to satisfy ruff strict gate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: run black formatting on UP045-fixed files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused Optional imports after UP045 migration Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: black format cold_storage_handler.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): correct OSS staging branch name in guard-main-branch errors Co-authored-by: Cursor <cursoragent@cursor.com> * fix: strip trailing zeros from M/B spend formatter * fix: address focus and streaming edge cases * feat: add LAR-1 semantic routing strategy Optional router strategy that picks a deployment tier from request_kwargs.metadata.lar1 (confidence, evidence, time). Deployments are tagged with model_info.type (cloud-smart, cloud-fast, local, deep). Thresholds are configurable via routing_strategy_args. Includes 30 unit tests and an Ollama example config. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mavvrik): advance metricsMarker on empty-content deliver When deliver() receives empty content (no spend data for a date), it now registers with Mavvrik and PATCHes the metricsMarker before returning instead of short-circuiting. Dates with zero spend no longer stall marker advancement, preventing unnecessary catch-up API calls on subsequent runs. * style: black format mavvrik_destination * fix: handle empty mavvrik exports and lar1 reset * test: add regression test for _reset_custom_routing_strategy * fix(test): mock async destination.deliver in mavvrik export window test * style: ruff format spend_management_endpoints after merge * fix(router): apply LAR-1 strategy atomically so invalid thresholds don't leave partial state apply_lar1_routing_strategy set router.routing_strategy to "lar1" before constructing LAR1RoutingStrategy, whose __init__ validates thresholds via _normalize_thresholds and raises on a misconfigured (out-of-order or out-of-range) set. On a live update_settings call with bad thresholds the router was left advertising routing_strategy="lar1" with no custom selector bound, while the previous strategy's selectors stayed registered. Build (and validate) the strategy before mutating any router state, so a threshold error leaves the router exactly as it was. Add a regression test that asserts a failed switch keeps the prior strategy intact. --------- Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com> Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com> Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: xbrxr03 <abrarhabib03@gmail.com> Co-authored-by: hayden <sktpghks138@gmail.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Wassim Badraoui <98709649+Wassbdr@users.noreply.github.com> Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com> Co-authored-by: Neimar Avila <neimar.avila@gmail.com> Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> Co-authored-by: Jerry-Scintilla <jerrycaocao@126.com> Co-authored-by: AlexBGoode <me.at.forum@gmail.com> Co-authored-by: Carsten Boloz <cdboloz1@gmail.com> Co-authored-by: jesco <team@srswti.com> Co-authored-by: Praveen Ghuge <pghuge@digitalex.io> Co-authored-by: Jim Smith <j.h.smith@ieee.org> Co-authored-by: David J. M. Karlsen <david@davidkarlsen.com> Co-authored-by: Vedant Agarwal <43557509+Vedant-Agarwal@users.noreply.github.com> Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: bhumikadangayach <139267865+bhumikadangayach@users.noreply.github.com> Co-authored-by: Ewertonslv <ewertoncom297@gmail.com> Co-authored-by: carlsonchik <carlsonchik@users.noreply.github.com>	2026-06-26 09:17:44 -07:00
Sameer Kankute	687a62e561	fix(cli): mint per-session agent credential on lite login (#31072) ... * fix(cli): mint per-session agent credential on lite login The `lite login` command was producing a shared UI session token that broke agent use in three ways: a $0.25 budget cap (from max_ui_session_budget) that killed agent sessions in minutes, a fixed identity "cli-jwt-token" shared across every user preventing per-session spend attribution, and auth gated behind EXPERIMENTAL_UI_LOGIN so the token was rejected on default deployments. This fixes all three. Each login now generates a unique cli-session-{uuid} token with no per-key budget cap (enforced via shared team/user counters instead), and the decrypt path activates for any non-sk- token without requiring EXPERIMENTAL_UI_LOGIN. * fix(cli): address review feedback on EXPERIMENTAL_UI_LOGIN gate and e2e test Restore EXPERIMENTAL_UI_LOGIN=false as an explicit opt-out: operators who set it to false keep the old boundary; unset (new default) and true both attempt NaCl decryption, which fails closed for non-blob tokens. In the e2e test: replace the silent Redis fallback with pytest.skip so a missing Redis instance is explicit rather than silently degrading to a directly-minted token. Write the seeded flow back as JSON (proxy reads it via json.loads on cache fetch) instead of Python repr, and build the updated flow immutably. * fix(key-management): cap CLI session token delegation budget to team ceiling A CLI session token intentionally carries max_budget=None to avoid a per-session LLM spend cap. The key-generation delegation check (GHSA-q775-qw9r-2r4g) previously skipped non-admin callers with max_budget=None, treating them as having unlimited delegation authority. This allowed any internal user with a lite login session to mint virtual keys with arbitrary budgets. Adds is_session_token=True to UserAPIKeyAuth for CLI session tokens and uses the caller's team budget as the delegation ceiling in that case, so the effective limit is min(requested_budget, team.max_budget) rather than unbounded. * chore: regenerate dashboard OpenAPI types The is_session_token field added to UserAPIKeyAuth cascades to the dashboard schema. Regenerate types from the updated OpenAPI spec. * fix(key-management): block personal key budget delegation from CLI session tokens When team_table is None (personal key, no team_id in request), the personal key has no team-budget enforcement at request time. A session token therefore cannot delegate any explicit max_budget for a personal key -- that would open a budget bypass path. Block the request with a clear 400 directing the caller to use a team_id instead. * test(auth): add unit coverage for non-admin CLI session token production path * fix(type-check): use model_validate in _return_user_api_key_auth_obj to fix reportArgumentType gate UserAPIKeyAuth(**user_api_key_kwargs) spread triggers a basedpyright reportArgumentType error for each named field in UserAPIKeyAuth because the dict's inferred value type (str | Span | LitellmUserRoles | Unknown) is not assignable to each field's specific type. Adding is_session_token: bool introduced +2 more such errors, breaching the gate cap. model_validate accepts an untyped dict without per-field argument checking, which eliminates the +2 new errors and also ratchets down the pre-existing 333 errors at those call sites. basedpyright-code-budget.json is updated to reflect the new lower baseline (1814, down from 1934). * fix(type-check): ratchet down reportArgumentType baseline only The previous lint-budget-update captured all baselines from the local environment, raising many ceilings vs the merge-base and failing the non-gating budget_ratchet_check. Restore staging's values for every rule and only lower reportArgumentType (1934 -> 1814) to reflect the reduction from switching to model_validate in _return_user_api_key_auth_obj. * fix(auth): set max_budget on CLI session token to enforce max_ui_session_budget CLI session tokens were missing max_budget, so _virtual_key_max_budget_check had no per-session ceiling to enforce. Operators relying on max_ui_session_budget could be bypassed for the full token lifetime. Mirrors the existing UI token path. * revert(auth): remove max_ui_session_budget from CLI session token max_ui_session_budget defaults to $0.25 and is sized for the UI chat pane (10-min sessions). CLI sessions are 24-hour tokens for real work; capping them at that ceiling would throttle users under their actual user/team budget. Budget enforcement for CLI sessions is via the shared user and team counters as originally intended. * fix(auth): cap CLI session at max_ui_session_budget only when user and team have no budget When neither the user nor their team has a budget configured, CLI sessions were fully uncapped. The poll endpoint now looks up the real user and team objects from DB; if both have no max_budget, it passes litellm.max_ui_session_budget as the token's per-key ceiling. Users or teams that already have a budget configured are unaffected and continue to rely on the shared counters. * fix(auth): fix black formatting and update test mock for cli_poll_key budget lookup The get_user_object and get_team_object async calls in cli_poll_key were not mocked in the existing test, causing MagicMock await errors. Patch both functions at the auth_checks module level. Also apply black formatting to ui_sso.py which CI rejected. * fix(auth): skip fallback budget cap when team lookup fails for cli session token * test(auth): pin cli session budget cap to user/team budget presence The session_max_budget fallback in cli_poll_key only applied max_ui_session_budget when neither the user nor the resolved team had a budget. The existing coverage exercised only the team-lookup-failure branch. Add two regression tests: a user with a configured budget must not receive the fallback cap, and a session with no user and no team budget must fall back to max_ui_session_budget. Mutating either guard out of the branch now fails these tests. * fix: remove CLI poll session budget cap * revert(auth): restore CLI session fallback budget cap Bugbot autofix (60b81fb8) removed the user/team budget lookup in cli_poll_key and stopped passing max_budget to the session token, making CLI sessions fully uncapped whenever neither the user nor the team has an explicit budget. That reintroduces the unbounded-spend bypass veria flagged as High ("CLI session budget bypass"): on deployments that rely on max_ui_session_budget rather than per-user/team budgets, a completed lite login could run LLM calls with no ceiling for the whole token lifetime. The fallback only applies when no other budget bounds the session, so users and teams with a configured budget are unaffected and keep relying on their shared counters. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-26 09:05:15 -07:00
Sameer Kankute	2b496bc7f7	fix(proxy): restore wildcard expansion in /v1/model/info (#31444)	2026-06-26 08:50:58 -07:00
Sameer Kankute	e5da5a3b6d	fix(proxy): skip model override when response has no model field (#31183) ... * fix(proxy): skip OpenAI model override for search responses Search responses omit a model field by spec but still set model on the request for routing, which caused noisy errors and dict injection. * fix(proxy): drop redundant search-specific model override skip The silent return for responses without a model field already covers SearchResponse objects; remove the extra search type check. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): skip model override for dict responses without model key Dict-shaped responses (e.g. search) must not get a spurious model field injected when they never had one; only override when model is present. Co-authored-by: Cursor <cursoragent@cursor.com> * test(proxy): cover swallowed setattr failure in model override --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-25 21:02:52 -07:00
Sameer Kankute	7eacdd5258	chore: litellm oss staging 250626 (#31305) ... * fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes #30926) * style: format common_utils.py with black * fix(anthropic): extract api_base from litellm_params in batches/files validate_environment * fix(anthropic): scope Bearer key check to custom api_base endpoints * fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives The Anthropic streaming protocol emits `message_start.usage.output_tokens=1` as a placeholder cursor; the real cumulative output count only arrives in the final `message_delta` event. When a stream is cancelled before `message_delta` lands (common for thinking models on long-tail prompts), ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left completion_tokens stuck at 1. Because 1 is truthy, the `completion_tokens or token_counter(text=...)` fallback in calculate_usage() never fired, and requests were billed for 1 output token even when several thousand tokens of text had actually streamed. Fix: track whether any chunk's completion_tokens exceeded 1 (saw_non_cursor_completion). If the only update we saw was the cursor, reset completion_tokens to 0 so the text-based fallback estimates from the real completion content. Legitimate 1-token completions (model returns "Yes." etc.) are unaffected in practice — token_counter on a 1-token completion_output also yields ~1, so billing stays approximately correct. Tests: - TestAnthropicCursorBug (6 cases) — pins the post-fix behavior - TestNonAnthropicStreamingIntact (2 cases) — guards against regression on providers without the cursor pattern All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests still pass. * fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival Addresses both Greptile P2 threads on PR #30420: CLASS A — Anthropic-specific heuristic was applied globally ============================================================ The `completion_tokens == 1 and not saw_non_cursor_completion` reset lived in provider-neutral `streaming_chunk_builder_utils.py`. Any non-Anthropic provider that legitimately reports completion_tokens=1 in a single usage chunk (perfectly normal for short OpenAI / Bedrock / Vertex single-token replies with stream_options.include_usage=true) would have its value silently rewritten to 0 and re-billed via token_counter — producing a different number than what the provider actually charged. Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved from the first chunk's `_hidden_params` (the same field set by streaming_handler.py:722 on the live path). Unknown / missing provider is treated as non-Anthropic and skips the reset, so newer providers and custom plugins are also safe by default. CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies ============================================================ Previous condition was `usage_chunk_dict["completion_tokens"] > 1`, which never fires for an Anthropic stream where the model legitimately emits exactly one output token (e.g., "Yes."). Anthropic still sends message_start (output_tokens=1, the cursor) AND message_delta (output_tokens=1, the real value) — same value, but two distinct usage events. The old check couldn't tell that apart from a cancelled stream where only message_start landed. Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion` when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR (2) we've seen >=2 completion-bearing usage events (positive evidence that message_delta arrived). Cancelled cursor-only streams still have exactly one event and still hit the reset; cache chunks with completion_tokens=0 don't count toward the threshold. Tests ============================================================ - _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default "anthropic") so the gate is exercised by every existing test — none of them needed assertion changes besides the legitimate-single- token case, which now expects exactly 1 (was a fuzzy 0..3 range). - New: test_anthropic_cache_only_chunks_after_message_start_still_resets - New: test_non_anthropic_provider_completion_tokens_one_not_reset - New: test_unknown_provider_completion_tokens_one_not_reset 11/11 tests pass. * chore: add Co-authored-by trailer for attribution Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> * fix(anthropic): preserve messages cache usage * style(anthropic): format messages cache usage helper * fix(anthropic): accept integral float cache token counts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(anthropic): accept integral float cache token counts * test(anthropic): cover cache usage edge cases * fix(gemini): preserve thoughtSignature for server-side tool responses When Gemini API returns toolCall and toolResponse parts, they might have different thoughtSignatures. Previously, LiteLLM merged them into a single dict, overwriting the response's thoughtSignature with the call's. This fix extracts them separately and re-injects them correctly. TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * fix(gemini): address PR comments on thoughtSignature handling - Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature - Add missing assertions in existing tests - Add new unit tests for orphan-response signature handling TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * feat(mcp): include server alias and server_id in mcp_info response - Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint - Update rest_endpoints.py to surface alias from server config - Add test coverage in test_mcp_server.py and test_rest_endpoints.py Fixes #31015 * fix(proxy): reject non-finite spend via validate_finite_spend A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a shared finite-value guard, defined above the litellm.proxy.* imports to avoid the module-level cyclic-import warning. * fix(proxy): require admin for any /key/update spend, reject non-finite Gate the admin check on the presence of `spend` (not a value diff): the DB spend lags the live cross-pod counter, so an "unchanged" spend on the non-admin path let a key owner / team member overwrite the live counter below real usage. Also reject NaN/+-inf spend before the DB write. * fix(proxy): invalidate spend counter on /user/update spend change A direct spend change on /user/update wrote the DB row but left the warm cross-pod counter at the stale value, so enforcement kept reading the old spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB), and reject non-finite spend before the write. * fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244) The semantic cache's embedding model is a proxy Router alias whose AWS credentials (aws_role_name, aws_session_name) live only in the Router deployment's litellm_params. The sync embedding paths called litellm.embedding() directly, bypassing the Router, so they could neither resolve the alias nor assume the configured role; cross-account Bedrock semantic caching failed with "bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding during cache construction, while llm_router is still None. Fix A: make the sync paths mirror the already-correct async paths. A shared, dependency-injected helper (litellm/caching/_embedding_router.py) decides whether to route through llm_router.embedding(...) when the model is a Router deployment, else fall back to direct litellm.embedding(...). Redis and qdrant sync set_cache/get_cache now precompute the embedding and pass vector= to the backend, exactly as the async astore/acheck already do. Both async _get_async_embedding methods are unified onto the same helper and now forward the caller's full metadata instead of a hand-picked subset. Fix B (Redis only): defer redisvl index construction from __init__ into a lazy, memoized llmcache property, so the dimension-probe embedding fires on first cache use, after llm_router is wired. A failed build is not memoized, so a transient outage recovers on the next request. Known limitation: resolve_embedding_router gates on an exact model-name match (same as the shipped async path); wildcard/alias/team-public routes still fall back to direct embedding. Tracked as a follow-up. * fix(cache): harden embedding-router and shrink Any surface (review) Address review feedback on the semantic-cache aws-role fix (#28244): - resolve_embedding_router now skips deployment entries missing model_name instead of raising KeyError on a malformed model_list (Greptile P2); add a regression test that fails on the old direct-key access. - Replace the `**kwargs: Any` passthrough on the four cache _get_embedding / _get_async_embedding helpers with an explicit, typed `metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only ever consumed kwargs["metadata"], so this is behavior-preserving, makes the forwarded field obvious at the call site, and removes three bare-Any annotations (keeps the strict-rule ANN401 budget within ceiling). - Note in _build_llmcache that redisvl's dimension-probe embedding adds one extra billable embedding on the first cache request (Greptile P2). * fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist" Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash * test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression Adds the regression coverage codecov flagged on the two responses to completion bridge guard lines and the bedrock route-prefix helper. The handler tests drive both the sync and async fallback paths with litellm.completion and litellm.acompletion mocked, and assert the forwarded kwargs carry _skip_responses_api_bridge=True, so dropping either flag line fails the suite. The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids still do, exercising both branches of _model_has_route_prefix. Also aligns update_messages_with_model_file_ids model_id to Optional[str], matching its Responses API sibling, so the defensive model_info fallback no longer introduces a new reportArgumentType in completion(); the file-id lookup narrows model_id before the dict get * chore(ui): sync generated OpenAPI types for optional test_connection mode The test_model_connection mode body param default changed from chat to None so the mode is auto-detected from model capabilities, which makes the field optional in the proxy OpenAPI spec. Regenerate the committed schema so the dashboard types match: mode becomes optional and the description and default JSDoc follow the spec, keeping the Check UI API Types Sync gate green * refactor(bedrock): match all explicit route prefixes at path-segment boundary Migrates the remaining substring route checks to the existing _model_has_route_prefix helper so every explicit route token matches only as a leading path segment, consistent with get_bedrock_route and the mantle route. Covers _explicit_converse_route, _explicit_claude_platform_route, _explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route, _explicit_converse_like_route, _explicit_async_invoke_route and _explicit_openai_route. This also stops invoke/ from substring-matching async_invoke/. Route precedence and order are unchanged, and a note on the segment invariant is added to the helper docstring * test(bedrock): cover explicit route prefix segment matching Exercises all eight migrated _explicit_*_route helpers (converse, converse_like, invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each matches its token as a leading path segment and rejects the token glued to a preceding segment, so reverting any method to the old substring check fails the suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete improvement of the segment-boundary migration * test(proxy): assert negative spend is allowed (one-time grant use-case) Negative spend is intentionally permitted so admins can grant extra allowance for the current budget period only, without raising the recurring budget ceiling. Cover it explicitly in validate_finite_spend and via the /user/update invalidation test. * fix(google_genai): forward native generateContent top-level fields Google's native generateContent REST body carries safetySettings, toolConfig, cachedContent and labels at the top level as siblings of generationConfig. The proxy's :generateContent endpoint spread them into agenerate_content as loose kwargs and then dropped them, so callers had to wrap them in extra_body for them to take effect; safetySettings, for instance, was silently ignored The provider config now exposes the native top-level field names and setup_generate_content_call collects whichever are present, merging them into the outgoing request body through the existing extra_body merge so they reach Google verbatim. An explicit extra_body still wins on conflict. The sync generate_content_stream path now also forwards systemInstruction, matching the other three entry points Fixes #12671 Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK * fix(proxy): resolve env refs for DB-stored models * fix(proxy): restrict DB env ref resolution * fix(proxy): block team DB env ref resolution * fix(lint): resolve ANN401/UP045/C901 strict-gate violations - Replace Optional[X] with X | None (UP045) in 8 files - Replace Any return/param types with concrete types or object (ANN401) - Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix Users who pass a key already prefixed with "Bearer " get Authorization: Bearer. All other keys continue to use x-api-key, preserving backward compatibility with custom api_base endpoints that expect x-api-key rather than Authorization. Also consolidates get_auth_header to reuse _make_api_key_auth_header helper, eliminating the duplicated custom-endpoint routing logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base The backwards-compat change broke existing tests that verify the intentional Bearer-for-custom-base behavior (Fixes #30926). Restore original logic while keeping the _make_api_key_auth_header helper for code deduplication. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag Previously the auth-header switch from x-api-key to Authorization: Bearer applied unconditionally for non-sk-ant- keys on a custom api_base, silently breaking existing deployments that proxied to gateways expecting x-api-key. Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header, get_anthropic_headers, and get_auth_header. validate_environment reads it from litellm_params so callers can opt in per-model without any API surface change. Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981) DEL was the only Redis cache operation that skipped check_and_fix_namespace, so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM error on deployments with an ACL restricting DEL to the litellm:* pattern, and a silent no-op on all other deployments since the un-prefixed key was never stored. * style(anthropic): reformat common_utils.py with Black (--target-version py312) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve cache metadata and spend counters * style: apply ruff format to streaming_iterator.py * refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate Extract Anthropic message_start cursor reset into _reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter invalidation into _invalidate_user_spend_counter_if_changed, keeping both _calculate_usage_per_chunk and _update_single_user_helper under the max-complexity ceiling. Use builtin generics in the new signatures so no new UP006 violations are introduced. Behavior unchanged. --------- Co-authored-by: rupak-eng <rupakji99@gmail.com> Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com> Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com> Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com> Co-authored-by: Andrii Butko <booandrew23@gmail.com> Co-authored-by: Kent <kingdooo@gmail.com> Co-authored-by: kunal2002 <k.nayyar2002@gmail.com> Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com> Co-authored-by: jesco-absolut <team@srswti.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Matt Hill <mhill@dataminr.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-25 21:00:28 -07:00
Shivam Rawat	eb15fe667d	refactor(passthrough): address multipart review feedback ... Build form_data_dict in one pass with groupby instead of rescanning form_items per field name, and assert on the files list directly in the boundary regression test so repeated field names are not collapsed by dict(). Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-25 19:28:59 -07:00
Shivam Rawat	ef0785881a	fix(passthrough): forward all multipart files with repeated field names ... Passthrough multipart uploads used form.items() and a files dict, so only the last file under a repeated field name reached the upstream. Read multi_items() and send httpx a list of file tuples instead. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-25 19:14:13 -07:00
ryan-crabbe-berri	f16af8853b	feat(mcp): opt-in least-privilege default for team key MCP access (#31380) ... * feat(mcp): add require_key_mcp_access_defined to stop keys inheriting team MCP servers By default a virtual key that grants no MCP servers of its own inherits its team's full MCP server list. The new general_settings flag require_key_mcp_access_defined (default false) flips this so the team list acts purely as a ceiling: a key reaches only the servers it grants explicitly (or via an access group), and inherits none. This mirrors the existing require_end_user_mcp_access_defined setting. The default is unchanged, so existing deployments keep today's behavior until they opt in. The no-mcp-servers sentinel and key access-group grants are unaffected. * docs(mcp): note require_key_mcp_access_defined effect in resolver docstring	2026-06-25 18:49:15 -07:00
yucheng-berri	9203488578	feat(spend): store litellm_call_id on spend logs for DB-to-trace correlation (#31344) ... * feat(spend): store litellm_call_id on spend logs for DB-to-trace correlation Successful spend logs keyed request_id to the provider response id while tracing uses x-litellm-call-id, so a DB row could not be correlated with its trace; this only worked for failures, where request_id already fell back to the call id. Add a nullable litellm_call_id column to LiteLLM_SpendLogs, populate it in get_logging_payload, and surface it in the spend logs read endpoints so correlation works both directions for successful calls Fixes LIT-3868 * chore: sync schema.prisma copies from root * test(spend): cover cache-hit and missing-response-id paths for litellm_call_id Lock the intended behavior surfaced in review: on a cache hit request_id gets the uniqueness suffix while litellm_call_id stays the raw call id, and when the provider returns no id request_id falls back to the call id so both columns match. Both assertions fail when the populate line is reverted * test(spend): ignore litellm_call_id in spend logs payload comparisons get_logging_payload now always writes litellm_call_id, so the full-payload comparisons in test_spend_management_endpoints.py saw an unexpected key and failed. litellm_call_id is a per-request runtime uuid like request_id, which is already ignored, so add it to ignored_keys * test(logging): ignore litellm_call_id in gcs pubsub spend logs comparison The gcs pubsub spend logs payload comparison flags any key present in the actual payload but absent from the golden snapshot. get_logging_payload now always emits litellm_call_id, a per-request runtime uuid like request_id which is already ignored, so add it to ignored_keys * refactor(spend): store litellm_call_id in spend log metadata, drop column Switch DB-to-trace correlation off a dedicated column and onto the existing metadata JSON, avoiding a schema migration entirely. litellm_call_id is now written into spend log metadata (already selected and re-hydrated on the read paths) instead of a new LiteLLM_SpendLogs column, so the three schema.prisma copies and the migration are reverted and the read SELECTs go back to their original form. Correlation is queryable via metadata->>'litellm_call_id' Trade-off: an unindexed JSON lookup rather than an indexed column; acceptable for this use case and removes all migration risk * refactor(spend): thread litellm_call_id into _get_spend_logs_metadata Set litellm_call_id beside the other computed metadata values inside _get_spend_logs_metadata rather than mutating clean_metadata back in the caller, matching how applied_guardrails, cost_breakdown and the rest are threaded. No behavior change; the value still comes from kwargs with a litellm_params fallback --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-06-25 17:45:37 -07:00
yucheng-berri	71ee1a852a	fix(proxy/client): redact api key from key/info client error messages (#31342) ... * fix(proxy/client): redact api key from key/info client error messages The keys management client builds GET /key/info?key=<key> and lets the requests HTTPError propagate. str(HTTPError) renders the failing request URL verbatim ("... for url: .../key/info?key=sk-..."), so any caller that logs the exception leaks the full key; the 401 branch leaked the same way through UnauthorizedError(str(orig_exception)) Redact both branches with the existing redact_secrets helper so the secret-bearing query param is scrubbed to ?REDACTED while the status code, reason, and response object are preserved. Server-side responses already mask the key, so this closes the remaining client-side surface * fix: preserve key info unauthorized response --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-25 17:35:15 -07:00
Mateo Wang	6db55e0aa5	feat(mcp): add mcp_xff_num_trusted_hops to harden X-Forwarded-For client IP resolution (#31257) ... * feat(mcp): add mcp_xff_num_trusted_hops to harden XFF client IP resolution MCP per-server IP access control reads the client IP from X-Forwarded-For and trusts the leftmost entry. Behind an append-style proxy or load balancer (AWS ALB, nginx with $proxy_add_x_forwarded_for, HAProxy, Envoy, Cloudflare), a client can prepend an arbitrary value to the header, so the leftmost entry is attacker-controllable even when the direct peer is a trusted proxy. An attacker can therefore spoof an internal IP and reach servers marked available_on_public_internet=false. This adds an optional mcp_xff_num_trusted_hops general setting modelled on Envoy's xff_num_trusted_hops. When set to N, the client IP is read N entries from the right of the chain (where N is the number of trusted appending proxies in front of the gateway) instead of the leftmost value, so any entries a client prepends are ignored. It composes with mcp_trusted_proxy_ranges, which still validates the direct peer, and only takes effect once that check passes; without a validated direct peer the gateway keeps failing closed, so hop counting cannot be abused by a direct-to-pod attacker. The chain must contain at least N valid entries or resolution fails closed. Default is unset, preserving existing behaviour. * chore(ui): regenerate dashboard schema for mcp_xff_num_trusted_hops * fix(mcp): warn when mcp_xff_num_trusted_hops is below the minimum A 0 or negative value is silently treated as disabled, which could leave an operator believing they enabled append-style X-Forwarded-For hardening while client IP resolution stays on the spoofable leftmost value. Emit a warning, consistent with how the module already surfaces invalid CIDR config, so the misconfiguration is visible in logs. * fix(mcp): reject mcp_xff_num_trusted_hops < 1 at config-parse time Add a ge=1 bound to the ConfigGeneralSettings field so the update_config_general_settings path rejects 0 and negative values with a clear validation error instead of accepting them, and self-documents the valid range. The runtime warning stays as defense-in-depth for raw-dict config that bypasses model validation. * style(mcp): black-format ip_address_utils.py * fix(mcp): fail closed when mcp_xff_num_trusted_hops is set but invalid A present-but-invalid mcp_xff_num_trusted_hops (non-integer, or below 1) previously made _resolve_num_trusted_hops return None, which the caller treated identically to "unset" and silently fell back to the legacy leftmost X-Forwarded-For value. An operator who set the value to harden client IP resolution but typo'd it would get weaker security than before, with no fail-closed signal. Model the setting as a tagged union (_HopCountUnset, _HopCountInvalid, _HopCount) so the three states are distinct: unset keeps the legacy path, a valid count drives hop-counting, and an invalid value fails closed (returns "") instead of reverting to the spoofable leftmost address. The caller matches on the union exhaustively. Add a parametrized regression test asserting get_mcp_client_ip returns "" for 0, -1, "abc", and 1.5 even with a spoofed internal leftmost entry, and update the resolver unit tests for the new return type.	2026-06-25 07:31:29 -07:00
tin-berri	0f5603895c	fix(mcp): challenge delegate-auth OAuth servers with upstream resource_metadata (#31255) ... An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the user to sign in. On an unauthenticated initialize the gateway answered locally (200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop either connected empty or hit "OAuth probe timeout after 10000ms". #30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth servers, expecting the upstream to emit its own challenge. On initialize the gateway never probes upstream, so no challenge ever reached the client. Replace the `continue` with a preemptive 401 carrying the proxied resource_metadata (RFC 9728) challenge, the same form passthrough servers and MCPUpstreamAuthError already use. This keeps #29770 fixed (still no authorization_uri) while restoring the upstream PKCE sign-in prompt.	2026-06-24 20:50:21 -07:00
tin-berri	f426912ba1	fix(mcp): resolve toolset tools by the server's known prefix (#31254) ... * fix(mcp): resolve toolset tools by the server's known prefix Toolsets store {server_id, bare tool_name} and reconcile that against the live prefixed tool name at list time. The reconciliation chopped the live name at the first MCP_TOOL_PREFIX_SEPARATOR with no server context, so a server whose prefix contains the separator (a hyphenated alias, or the UUID server_id used as the prefix when a server has no alias) had its tools silently dropped from /toolset/<name>/mcp while listing fine everywhere else. Strip the exact known prefix for the tool's server_id instead of guessing the boundary, on both the resolve and filter sides Also render toolset tools as {server-prefix}-{tool} in the dashboard picker result and chips; this is display only, the persisted record stays {server_id, bare tool_name} Resolves LIT-3419 * test(mcp): add focused unit tests for strip_known_server_prefix Cover the LIT-3419 cases directly on the helper with real MCPServer objects: clean prefix round-trip, hyphenated alias, UUID server_id fallback, unprefixed passthrough, and the server=None legacy fallback	2026-06-24 20:50:16 -07:00
Mateo Wang	9c41077786	fix(mcp): warn loudly when X-Forwarded-For is present but use_x_forwarded_for is off (#31266) ... * fix(mcp): warn loudly when X-Forwarded-For is present but use_x_forwarded_for is off When a request carries an X-Forwarded-For header but use_x_forwarded_for is unset, get_mcp_client_ip silently falls back to the direct peer's IP (the load balancer / reverse proxy). That peer almost always sits inside mcp_internal_ip_ranges, so the 'Internal network only' (available_on_public_internet: false) restriction trusts every external caller as internal and effectively exposes those servers. Emit a one-shot loud error pointing the operator at use_x_forwarded_for instead of hard-failing: on a deployment with no load balancer, a crafted X-Forwarded-For header must not be able to take the service down, and a one-shot log keeps a flood of crafted headers from spamming the logs. * fix(mcp): re-arm XFF-disabled warning on config change and harden test assertion Address PR review: tie the one-shot warning flag to the observed use_x_forwarded_for value so it re-arms whenever the setting is seen enabled, restoring the diagnostic on a later rollback to disabled. Also assert against str(call_args) so the test survives a positional-to-keyword logger refactor.	2026-06-24 20:49:32 -07:00
Mateo Wang	257d67167f	fix(mcp): correct misleading no-trusted-proxy warning for XFF access control (#31264) ... * fix(mcp): correct misleading no-trusted-proxy warning for XFF access control * test(mcp): assert the no-trusted-ranges warning was logged instead of relying on StopIteration	2026-06-24 20:49:29 -07:00
mubashir1osmani	0e1d0f4742	fix(proxy): stop double-decrypting email/slack alerting env vars in get_config (#31117) ... * fix(proxy): stop double-decrypting email/slack alerting env vars in get_config proxy_config.get_config() already returns environment_variables decrypted (the DB overlay decrypts them in _update_config_fields, and YAML values are plaintext), so the /get/config/callbacks slack and email blocks were running decrypt_value_helper() a second time on plaintext. That second decrypt always failed and the helper swallowed the error and returned None, so every SMTP_* field came back blank when the Admin UI reloaded the email settings, and the proxy logged a misleading "Did your master_key/salt key change recently?" error even when nothing changed. Consume the already-decrypted values directly, matching process_callback's handling of the same dict for langfuse/datadog/etc. Sensitive-value masking is preserved. Fixes #19221 * fix(proxy): preserve a cleared slack webhook instead of falling back to OS env Use an explicit is-not-None guard rather than truthiness when deciding whether to fall back to os.getenv for SLACK_WEBHOOK_URL. With `or`, a webhook the admin cleared (stored as "") is falsy and would surface a stale SLACK_WEBHOOK_URL from the OS environment; only a truly absent key should trigger the OS lookup. No decryption is reintroduced.	2026-06-24 19:19:08 -07:00
ishaan-berri	4efce809d0	feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through callbacks (#31134) ... * feat(proxy): add logging_endpoints package init * feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through the success/failure callback fan-out * feat(proxy): register callback_logs_router * test(proxy): add logging_endpoints test package init * test(proxy): cover /v1/callbacks/logs replay, admin guard, and partial-failure handling * refactor(proxy): move callback-logs request/response models to litellm/types/proxy * refactor(proxy): wrap callback-logs replay in CallbackLogsReplayer class with payload logging * test(proxy): update callback-logs tests for class-based replayer and separated types * fix(proxy): cover /v1/callbacks/ in backend component allowlist The new /v1/callbacks/logs route was dropped by both component allowlists, failing test_gateway_plus_backend_covers_full_app. It's an admin-only spend-logging route, so it belongs on the backend (control plane) alongside the existing /callbacks family. * refactor(proxy): use builtin dict/list generics in callback-logs endpoint Switch Dict/List from typing to builtin dict/list to satisfy the ruff strict-rule budget (UP006). * refactor(proxy): use builtin dict/list generics in callback-logs types UP006: builtin generics over typing.Dict/List. * chore(ui): regenerate schema.d.ts for /v1/callbacks/logs Run npm run gen:api to add the CallbackLogRecord/CallbackLogsRequest/ CallbackLogsResponse types and the /v1/callbacks/logs path, keeping the dashboard types in sync with the proxy OpenAPI spec. * fix(proxy): force stream=False when replaying callback logs A replayed StandardLoggingPayload is a terminal, fully-aggregated event — the producer (e.g. the rust realtime gateway) already collected the whole session before POSTing. Marking the rebuilt Logging object as streaming made async_success_handler wait for a complete_streaming_response that never arrives, so the spend log was never written. Realtime sessions now land in LiteLLM_SpendLogs. * feat(litellm-rust): CustomLogger callback layer posting to /v1/callbacks/logs integrations/ mirrors litellm/integrations/: a sync, typed CustomLogger trait (base contract), a typed StandardLoggingPayload, and LiteLLMPythonProxyAPILogger — the first concrete logger, owning a bounded channel + background worker that batches and POSTs to the Python proxy's /v1/callbacks/logs. * feat(litellm-rust): RealTimeStreaming per-session log collector 1:1 with Python's RealTimeStreaming: observe() accumulates O(1) usage/model/id per event (never buffers frames); log_messages() builds one StandardLoggingPayload on session close and fans out to the CustomLogger callbacks. request_id == the OpenAI realtime session id (sess_…), with the gateway id as fallback. * feat(litellm-rust): wire realtime logging into the splice (lock-free observe) The collector is owned on the splice task and observed via a synchronous &mut callback threaded through providers::realtime::realtime() — no Arc/Mutex/atomic on the per-frame hot path. On session close the bridge flushes one payload. AppState carries the registered loggers; main spawns the proxy logger. * docs(litellm-rust): ai-gateway realtime logging architecture * docs(litellm-rust): document request-log egress to the LiteLLM control plane Add a 'Request logging' guide to the ai-gateway README: how to point the gateway at a LiteLLM proxy via LITELLM_PROXY_BASE_URL (+ LITELLM_MASTER_KEY for the admin-only /v1/callbacks/logs POST), and the non-blocking / one-payload-per-session behavior. * feat(litellm-rust): make log-egress tunables env-overridable Channel capacity, batch size, and flush interval now read from LITELLM_LOG_CHANNEL_CAPACITY / LITELLM_LOG_BATCH_SIZE / LITELLM_LOG_FLUSH_INTERVAL_MS, falling back to the DEFAULT_* consts on missing/invalid/non-positive values. Grouped behind an EgressTunables::from_env() read once at logger construction. * docs(litellm-rust): document log-egress tuning env vars * docs(litellm-rust): require constants in a crate-level constants.rs Mirror of Python's litellm/constants.py rule — magic numbers and fixed strings go in src/constants.rs, not inline in feature modules; env-overridable tunables keep their DEFAULT_* value there. * refactor(litellm-rust): move ai-gateway constants into constants.rs Per the new rule: the log-egress defaults (proxy base, ingest path, channel capacity, batch size, flush interval) and the realtime provider default move to crates/ai-gateway/src/constants.rs; modules import from it. * ci: run logging_endpoints tests in the proxy-infra coverage shard tests/test_litellm/proxy/logging_endpoints wasn't in any coverage-uploading job, so callback_logs_endpoints.py showed only import-level coverage (~35%) on codecov/patch despite being ~98% covered locally. Add it to proxy-infra's test-path so the test is exercised under --cov. * fix(litellm-rust): hash the master key before logging — never send the raw credential Greptile/Veria P1: user_api_key_hash was the plaintext LITELLM_MASTER_KEY, which fans out to spend logs and every callback (Langfuse/Datadog) and could be recovered from logs. SHA-256 it (auth::hash_token, matching the proxy's hash_token); the field is named *_hash and the proxy stores it verbatim when it isn't sk-prefixed, so the DB value is identical with zero plaintext exposure. * fix(litellm-rust): observe realtime logging on upstream events only Greptile P1: observe ran on the client->upstream arm too, so an authenticated client could send a fabricated response.done and inflate its own spend log. session.created/response.done are server->client events; observe the upstream arm only. * feat(proxy): bound callback-logs batch + return per-record failures Greptile P2: cap /v1/callbacks/logs at MAX_CALLBACK_LOG_RECORDS (default 1000, env-overridable) so one POST can't trigger an unbounded callback/DB fan-out; and return per-record {index, error} failures so a caller (the rust gateway) can distinguish a transient callback error from a structurally bad payload. * chore(ui): regenerate schema.d.ts for CallbackLogFailure / failures field * fix(constants): make MAX_CALLBACK_LOG_RECORDS a plain constant It doesn't need to be env-configurable (only the rust egress tunables are). As an os.getenv var it tripped tests/documentation_tests/test_env_keys.py, which requires every env key to be documented in the (separate-repo) config_settings.md. Plain constant → not scanned → code-quality + documentation checks pass. * docs(litellm-rust): trim ai-gateway ARCHITECTURE.md to one diagram + notes * docs(litellm-rust): tighten the README request-logging section * docs(litellm-rust): ARCHITECTURE.md is just the diagram (gateway = inference, spend = callback) * docs(litellm-rust): drop em-dashes from the request-logging section --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>	2026-06-24 15:25:10 -07:00
tin-berri	bbef1b84ab	feat(mcp): graft v2 resolver onto _create_mcp_client (none + api_key static family) (#31058) ... * feat(mcp): add v1 bridge + none/api_key resolver arms (unwired) PR4a of the MCP v2 outbound-credential migration, stacked on the resolver skeleton. Builds the bridge for the first live modes without wiring it onto the request path: - resolver.py: the none arm (NoOpAuth) and the api_key shared-key arm (StaticHeaderAuth from the config); the BYOK source and the other five arms stay not_implemented. - adapter.py: the v1 <-> v2 edge (to_subject, to_server_spec, raise_public, should_defer). to_server_spec maps only none + the static-header family and returns None to defer every other mode to v1. Imports v1, kept out of the package __init__ so the resolver core stays v1-free. - MCPClient gains an optional resolved_auth that feeds the factory's auth= slot, taking precedence over the SigV4 aws_auth; default None keeps current behavior. Nothing calls these from _create_mcp_client yet, so production behavior is unchanged; the graft lands in PR4b. Unit tests cover the two arms, the full mapping table, and the auth plumbing. * feat(mcp): graft v2 resolver onto _create_mcp_client for migrated modes Wire the none + api_key static-family resolver arms from PR4a onto v1's live request path. In _create_mcp_client's HTTP/SSE branch, to_server_spec decides per mode: a migrated mode resolves through the injected UpstreamCredentialProvider and feeds the resulting httpx.Auth into the new resolved_auth slot; every other mode returns None and falls through to the unchanged v1 construction. resolve_mcp_auth now runs only when the mode defers, so a migrated server skips the v1 token-exchange / M2M I/O. stdio is untouched: auth_type/auth_value never reach the upstream on the stdio path (_get_auth_headers is HTTP/SSE only), so there is nothing to graft there. No v1 code is deleted yet; resolve_mcp_auth's static return still backs stdio and the not-yet-migrated modes until later PRs retire it. * test(mcp): cover the v2-resolver graft in _create_mcp_client Regression tests for the PR4 graft. Migrated HTTP modes resolve through the provider into resolved_auth: none -> NoOpAuth, and the static api_key family emits the right header per scheme (X-API-Key, Bearer, token, raw authorization, base64 basic). Deferred modes (oauth2) and a missing static token fall back to v1's auth_value. A stdio server with a migrated auth_type still defers to v1, since httpx.Auth never reaches the subprocess. A resolver Error is mapped to the public HTTP contract (401) via an injected provider, exercising the DI seam. * fix(mcp): defer to v1 when an inbound credential would be overridden The graft attaches the resolved static credential as an httpx.Auth, whose auth flow writes its header after extra_headers. That silently overrode an inbound Authorization: a per-request mcp_auth_header override, or a header supplied via a guardrail hook / static_headers / forwarded caller header. v1 lets those win, so the graft had inverted the credential precedence for the migrated static modes. Mirror the v2 egress credential-isolation invariant: defer the request to v1 when mcp_auth_header is set, or when the header the resolved credential would write is already present in extra_headers. none writes no header, so it never defers. * test(mcp): cover the credential-isolation defer guard Regression tests for the precedence fix. A per-request mcp_auth_header override and an Authorization already present in extra_headers (guardrail hook like the JWT signer, static_headers, or a forwarded caller header) both defer a migrated static server to v1 so the inbound credential wins; none stays on v2 and does not clobber an inbound Authorization since NoOpAuth writes nothing. The deferred cases assert resolved_auth is None, which fails if the guard is removed. * refactor(mcp): resolve inbound-header conflict on v2 instead of deferring For an Authorization already supplied via extra_headers (a guardrail hook such as the JWT signer, static_headers, or a forwarded caller header), keep the request on the v2 path and skip resolved_auth rather than deferring to v1. The inbound header still wins since nothing overwrites it, but hooks no longer pin a v1 fallback, which is what lets resolve_mcp_auth be retired once the remaining modes migrate. The mcp_auth_header per-request override still defers to v1, since that value becomes the upstream credential rather than sitting in extra_headers; that defer falls away once the per-user modes stop writing mcp_auth_header. * fix(mcp): clear UP037 lint gate and fix allowed-servers test under the graft adapter.py uses `from __future__ import annotations`, so the quoted "UserAPIKeyAuth" / "MCPServer" annotations in to_subject/to_server_spec/_shared_key_spec were unnecessary and pushed UP037 over the strict-rule budget; drop the quotes. test_list_tools_only_returns_allowed_servers passed a MagicMock as user_api_key_auth. The graft now builds a Subject from the principal, and the MagicMock's non-string org_id/user_id fail Subject validation, so the listing came back empty. Use a real UserAPIKeyAuth instead (MagicMock for an injected dependency was the anti-pattern here). * test(mcp): assert config token via resolved_auth, not the headers dict test_mcp_server_config_auth_value_header_used inspected _get_auth_headers(), but the graft now carries the static credential on the client's httpx.Auth (resolved_auth) and writes the header at send time, so that dict is empty. Assert the header the StaticHeaderAuth emits onto the request instead. Both config keys (authentication_token, auth_value) stay covered. * chore(typecheck): set reportMatchNotExhaustive slack to 0 The previous slack of 3 put the ceiling at baseline + slack = 4, so a newly non-exhaustive match (for instance dropping an Error arm off a Result match) could land without tripping the gate. Setting slack to 0 pins the ceiling at the current baseline of 1, so any added non-exhaustive match now fails CI while the one pre-existing violation in router.py stays within budget	2026-06-24 14:53:33 -07:00
tin-berri	6003187165	fix(mcp): let proxy admins assign MCP servers to teamless keys (#31126) ... Creating or updating a key with a specific (non-allow_all_keys) MCP server or access group failed with a 403 when the key had no team: Key is not in a team. Only globally available (allow_all_keys) MCP servers can be assigned validate_key_mcp_servers_against_team computed the allowed set as team servers + allow_all_keys servers. For a teamless key the team set is empty, so the allowed set collapsed to just allow_all_keys servers and any explicitly-picked server or access group was rejected. This was asymmetric with runtime: get_allowed_mcp_servers honors a teamless key's own object_permission.mcp_servers verbatim, with no team gate and no allow_all_keys filter. So the create/update path refused to persist a grant the run path would have served. Thread is_proxy_admin into the validator from both call sites (/key/generate and /key/update). When a key has no team and the caller is a proxy admin, the requested servers and access groups are folded into the allowed set so the existing subset checks pass. A proxy admin can already reach every MCP server, so there is nothing to escalate. Non-admins and every team-scoped key are unchanged. Resolves LIT-3815	2026-06-24 13:20:11 -07:00
mubashir1osmani	56825926af	fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads (#31036) ... * fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker because the request body was buffered and multiplied 2-3x in size. The create-file path is now streaming end-to-end: transform_create_file_request returns a ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks (Content-Range, 308 between chunks) so the transformed payload is never held in full. The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of reading the whole body, and batch rate limiting counts tokens and models in a single streaming pass. Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is intentionally not read. Also removes the unreachable VertexAIFilesHandler create path and everything only it kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced. * fix(batches): return original JSONL on unparseable row to avoid silent batch truncation The streaming rewrite of replace_model_in_jsonl accumulated physical lines and skipped a row on JSONDecodeError to support multi-line objects, but a genuinely malformed or truncated row never completes: it poisons the buffer, swallows every following row, and the function still returned the partial rewrite (the rows before the bad one, already model-rewritten) as if the batch were complete. That turned the pre-rewrite behavior of returning the original file unchanged (so the provider rejects the bad batch loudly) into a silent partial submission. Restore the original-content fallback: when an unparseable remainder is left after the loop, return the original file_content (rewinding a consumed seekable source) instead of the truncated output. The multi-line happy path is unchanged. * test(batches): mock resumable GCS upload in vertex batch prediction test The vertex batch file-create path now streams to a GCS resumable session via _aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the existing test's post mock no longer intercepted the upload and a real request hit GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the resumable protocol itself is covered in test_vertex_ai_files_streaming.py. * fix(batches): resilient per-row token accounting; no hard-block on count failure The batch input-file pass iterated a generator whose json.loads raised on a malformed line; the outer except caught it and stopped the loop, so any body.model on rows after a bad line was never collected and the model allowlist check ran against a partial set. It also hard-blocked the batch with a 400 whenever token counting raised, a backwards-incompatible change from the prior swallow-and-proceed behavior that breaks legitimate rows the token counter cannot measure (e.g. some multimodal content). Iterate the JSONL line-by-line and account each row independently. A malformed line is skipped (its request cannot run upstream anyway) and a row the counter cannot measure falls back to a conservative size-based estimate. The loop never aborts, so the allowlist check always sees every parseable model, and the token total is never zeroed, so a crafted uncountable row still cannot evade the TPM limit, without hard-rejecting a legitimate batch. * perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types Three review follow-ups on the resumable batch upload: - _aresumable_chunked_upload pulled chunks from a synchronous generator that runs the per-row transform inline on the event loop thread, blocking other requests between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread. - _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly chunk-aligned upload finalizes on its last data chunk instead of an extra zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request. - valid_content_type now accepts the MIME types clients label .jsonl batch uploads with (text/plain, application/json, ndjson, ...), so such a batch file no longer silently bypasses the streaming path into the buffered media upload. * fix(vertex/files): keep legacy bucket_name as GCS bucket fallback The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present * style: sort imports in llm_http_handler to satisfy I001 budget --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-06-24 13:19:57 -07:00
mubashir1osmani	e0c8a6b483	fix(proxy): expand all-proxy-models sentinel in direct access lookup (#31153) ... A user provisioned with "All Proxy Models" stores the literal "all-proxy-models" sentinel in user.models. get_direct_access_models looked that string up as a real model_name via get_model_list, which matched no deployment, so /v2/model/info marked every model direct_access=false and the Models + Endpoints page rendered empty for such users when they have no teams. The model dropdown / Playground worked because get_key_models already expands the sentinel to the full proxy model list, hence the inconsistency in the report. Expand the sentinel to all non-team deployment ids via get_model_ids(exclude_team_models=True), the same call the PROXY_ADMIN branch in the caller already uses. This fixes both /v1/model/info and /v2/model/info since they share _populate_team_access_on_models. Empty user.models stays "no direct access" to match get_key_models semantics. Fixes #22791	2026-06-23 22:23:14 -07:00
tin-berri	360adbe765	fix(mcp): resolve config-defined servers in per-user credential and env-var endpoints (#31171) ... The per-user BYOK, OAuth (OBO), and env-var management endpoints resolved the target MCP server through a DB-only lookup (get_mcp_server / get_all_mcp_servers_for_user). A server defined in config.yaml lives only in the in-memory registry and never gets a row in LiteLLM_MCPServerTable, so those endpoints raised 404 "MCP Server <id> not found" (or 403 for non-admins) before any credential could be stored, leaving config-server users unable to connect and forced to re-authorize forever. Route all three through a single registry-aware resolver: DB first, then the in-memory registry (built into LiteLLM_MCPServerTable via _build_mcp_server_table, the same fallback fetch_mcp_server already uses), then the canonical get_allowed_mcp_servers authorization the MCP gateway enforces on tool calls. Admins get a 404 for an unknown id; non-admins get 403 for a missing-or-forbidden server so server ids stay non-enumerable. This also closes a gap where the two store endpoints performed no per-server authorization at all.	2026-06-23 21:38:10 -07:00
ishaan-berri	1d5ab42e14	feat: add minimal rust router + axum ai-gateway calling router.realtime (2/2) (#31135) ... * add CoreError::Routing variant for deployment selection failures * add minimal Rust Router (simple-shuffle) mirroring router.py spec * add litellm-router crate manifest * add ai-gateway POST /v1/realtime handler calling router.realtime * add ai-gateway health routes * wire ai-gateway routes into the axum app * add ai-gateway AppState holding the shared router * add ai-gateway axum server entrypoint * add litellm-ai-gateway binary crate manifest * docs: add ai-gateway folder-architecture AGENTS.md * register router + ai-gateway crates and axum/rand deps in workspace * update Cargo.lock for router + ai-gateway crates * split router: extract model_list types into deployment module * split router: extract routing policy into strategy module * split router: move Router orchestration into router module * router lib: wire submodules and re-export public API * add read_model_list helper reusing ProxyConfig env/secret resolution * add GIL-activity tracker (records acquisitions, 30s window) * add GET /health/gil endpoint for polling GIL activity * add pyo3 load_router_from_config bridge (feature-gated, load-time only) * register /health/gil route in ai-gateway * wire build_router: load from python config when feature enabled * add optional pyo3 dep + python-config feature to ai-gateway * update Cargo.lock for optional pyo3 dependency * fix: satisfy strict ruff budget (FA100) in read_model_list * test: cover read_model_list env resolution + empty config * ai-gateway: bind localhost by default, warn on bad PORT/missing keys, wire gateway key * ai-gateway: add gateway_key to AppState for realtime auth * ai-gateway: require bearer auth + map unknown model to 404 on /v1/realtime * ai-gateway: move python interop into python/ with load-time-only AGENTS.md * ai-gateway: document auth, gil, and python folder in AGENTS.md * core: add router module (model_list types + simple-shuffle selection) * ai-gateway: dispatch realtime via core router + providers (drop router crate dep) * update Cargo.lock: fold router into core * workspace: drop crates/router member and litellm-router dep * read_model_list: reuse ProxyConfig.get_config (includes + os.environ + DB) instead of thin yaml read * ai-gateway: constant-time bearer compare + 500 (not 503) for unconfigured key * ai-gateway: trim stored gateway key to match trimmed bearer token * ai-gateway: add subtle dep for constant-time comparison * workspace: add subtle dependency * update Cargo.lock for subtle * core router: make strategy a folder (one module per strategy, simple_shuffle) * providers: make realtime() a streaming splice (client stream <-> OpenAI) instead of collect * providers: add futures-channel dev-dep for the streaming live test * ai-gateway: make /v1/realtime a WebSocket (auth before upgrade, splice typed events) * ai-gateway: dispatch realtime as a stream splice * ai-gateway: route /v1/realtime via GET (WebSocket), drop POST * ai-gateway: enable axum ws feature + futures-util * update Cargo.lock for ws feature + futures-channel * core router: add has_deployment() for pre-flight model checks * ai-gateway: extract auth into auth/ module (single master key, LITELLM_MASTER_KEY) * ai-gateway routes: adopt router()-per-module template + merge in app() * ai-gateway: document auth/ + routes template in AGENTS.md * ai-gateway: realtime route as thin handler + service + transport * ai-gateway: auth as a RequireMasterKey extractor (idiomatic axum FromRequestParts) * ai-gateway: docs for auth extractor + simplified route template * ai-gateway: collapse realtime route to mod.rs + service.rs; docs for extractor/template * providers realtime: enforce idle timeout around the splice (reap stalled sessions) * ai-gateway: rename realtime service timeout param to idle_timeout --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>	2026-06-23 19:16:34 -07:00
yucheng-berri	c2e06890ad	fix: tighten role-based visibility of config and MCP fields (#30587) ... * fix: redact config and MCP secrets in read-only admin views GET /config/field/info and the MCP server list/detail endpoints returned secret-bearing fields to any caller with an admin view, including read-only admins. They now return those fields in full only to a full PROXY_ADMIN; every other caller gets the reduced, non-admin view, while non-sensitive fields remain readable. Regression tests cover the role-based visibility on both endpoints, including that a full admin still sees everything needed to populate the edit form. * fix: redact nested secrets in config field info for non-admins /config/field/info returned structured general_settings fields verbatim to any admin-view caller, so a view-only admin reading database_args received the nested aws_web_identity_token (a DynamoDB role-assumption credential) in plaintext. Recurse into dict/list field values and redact secret leaves for non-PROXY_ADMIN callers, leaving non-secret siblings and full-admin reads unchanged * fix: redact secret config values in /config/list for non-admins /config/list shared the same _user_has_admin_view gate as /config/field/info but returned each field value unredacted, so a view-only admin reading the list received pass_through_endpoints upstream Authorization headers verbatim. Route every general_settings value through a shared role-aware redactor (extracted from /config/field/info) covering the top-level and nested field paths, so non-PROXY_ADMIN callers get secret-bearing fields redacted while full-admin reads stay unchanged * chore(ci): allowlist _redact_secret_values_in_obj in recursive_detector The config secret redactor recurses over JsonValue, which is acyclic, and its depth is bounded by the operator-authored general_settings schema. Add it to the recursive_detector ignore list alongside the other bounded nested-redaction helpers (mask_dict, _redact_sensitive_litellm_params) * proxy: cap recursive secret redaction depth at 10 Match the cap on _redact_sensitive_litellm_params (the closest analog in the proxy, also recursive, key-name driven, returns a sentinel). The previous justification — bounded by operator-authored schema depth, JsonValue acyclic — is true today but is a property of the threat model, not an enforced invariant of the function. If a code path is ever added that pipes external input into general_settings (config import, migration tooling, JWT-driven settings, …) the assumption silently breaks. A local cap makes the invariant local. The cap branch fails closed: at _REDACT_SECRET_MAX_DEPTH the whole subtree is replaced with 'REDACTED' rather than returned verbatim. A future refactor that flips this to fail-open would let a deeply nested credential leak; the new regression test test_redact_secret_values_in_obj_fails_closed_at_max_depth guards against that. Updates the recursive_detector ignore-list rationale to point at the numeric cap rather than the structural argument. * test: actually exercise the depth cap in fails-closed test The previous fixture stored the leaf under the secret-named key 'aws_web_identity_token', which the recursor's key-name short-circuit redacts regardless of the cap — so the test passed both with and without the cap in place. Empirically confirmed: under an uncapped mutant the old fixture still hides the secret (key-name catches it), the new fixture leaks it (only the cap can stop it). Swap the leaf key to a non-secret name so the cap is the only redaction path exercised, making the test fail on mutation as advertised.	2026-06-23 15:25:30 -07:00
tin-berri	7020e1e5f7	feat(mcp): add resolve_credentials dispatch skeleton (#31056) ... PR3 of the MCP v2 outbound-credential migration, stacked on the typed vocabulary. Adds resolver.py: UpstreamCredentialProvider.resolve_credentials dispatches on the declared AuthConfig variant with one arm per mode, a wildcard-free match plus an assert_never tail so a missing arm fails basedpyright's exhaustiveness gate. Every arm is a not_implemented stub returning a typed CredError; each mode's real body and seam land in follow-up PRs. Pure v2, no v1 imports, nothing wired onto a request path.	2026-06-23 09:11:49 -07:00
Sameer Kankute	80c5a84871	chore: litellm oss staging (#30968) ... * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens * test: scope local cost map env var with monkeypatch to avoid test pollution * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764) * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold _mask_value did partial reveal by showing the first visible_prefix and last visible_suffix characters, but for a value whose length was at or below visible_prefix + visible_suffix (8 by default) it returned the value verbatim. A value of exactly 8 chars fell through the length guard and computed masked_length == 0, reconstructing the original string with no mask characters; anything shorter hit the early return. Either way short credentials were emitted in plaintext. mask_dict routes real secrets through this path, so an 8-char-or-shorter redis password, api key, or token could be written to logs and the UI unmasked. The sibling helper mask_sensitive_keys already guards this case; _mask_value now does the same by fully masking any value at or below the threshold. * fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers Fully masking short values is the right default for secret masking, but CooldownCache reuses the masker purely to truncate exception messages to the first 50 characters, and it relies on short messages being returned readable. Masking those blanked out short exception text and broke its tests. Add a mask_short_values flag (default True, secure) and have CooldownCache pass False so it keeps the truncation behavior, while every secret-masking caller still gets short values fully masked. * fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview MCPDebug uses the masker to preview auth tokens in debug headers and documents that values of 10 chars or fewer are shown unchanged so token types stay distinguishable. Pass mask_short_values=False so that diagnostic behavior is preserved while secret maskers keep masking short values. * fix(mcp_debug): mask short auth values in debug headers instead of echoing them Earlier this masker opted out of short-value masking to keep a token preview, but that echoes short authorization and token values verbatim in debug response headers, which is the same leak this change is meant to close. Auth material should never be emitted in full, so mask short values here too; the first/last character preview still applies to longer tokens. Only CooldownCache keeps the opt-out, since it truncates exception text rather than masking secrets. * test(mcp_debug): assert masked short value preserves length * refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917) Fireworks AI deprecated audio inference on 2026-06-10 (https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation). Live API testing confirms the endpoint is already non-functional: a valid Fireworks API key receives HTTP 401 "Unauthorized" from api.fireworks.ai/inference/v1/audio/transcriptions for every request, regardless of payload. The audio-prod.api.fireworks.ai host referenced in the test suite returns 401 for every path; the entire host is decommissioned. Remove the dead FireworksAIAudioTranscriptionConfig class and every reference to it across the codebase: - Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line config class that inherited from OpenAIWhisperAudioTranscriptionConfig) - Remove the Fireworks branch from ProviderConfigManager.get_provider_audio_transcription_config() in litellm/utils.py; update the stale comment in get_optional_params_transcription that referenced fireworks ai - Remove the FireworksAIAudioTranscriptionConfig entries from LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in litellm/_lazy_imports_registry.py - Remove the TYPE_CHECKING re-export in litellm/__init__.py - Remove the transcription branch in the fireworks_ai case of get_supported_openai_params() in litellm/litellm_core_utils/get_supported_openai_params.py - Remove the whisper-v3 and whisper-v3-turbo entries from model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json (both had mode: audio_transcription and zero-cost pricing) - Remove the TestFireworksAIAudioTranscription test class and its imports from tests/llm_translation/test_fireworks_ai_translation.py No other provider is affected. The openai_compatible_providers list, FireworksAIMixin, and the OpenAI Whisper transcription handler all stay because they are shared with other Fireworks endpoints and other providers. The provider_endpoints_support.json registry already had audio_transcriptions set to false for fireworks_ai. * feat: add darkbloom provider (#30876) * feat: add darkbloom provider * fix: document darkbloom provider endpoints * fix: address darkbloom review feedback * fix: update darkbloom tool metadata * fix: fail fast for non-Postgres database URLs (#30883) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging * Validate DIRECT_URL alongside DATABASE_URL startup guards * fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946) * fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608) * test(bedrock): mid-stream server errors trigger streaming fallback (#24608) * style(bedrock): black-format stream-error helper (#24608) * fix(mcp): re-land native tool preservation with typed annotations (#30645) * fix(mcp): preserve native tools in semantic filter hook with typed annotations * fix(mcp): tighten _is_mcp_tool Chat Completions shape check * fix(sambanova): return embeddings supported params instead of dropping them (#30937) * fix(router): send fallback metadata when streaming (#30914) When a streaming request triggers a fallback, there was previously no way to know it happened. This commit addresses this in a few ways: 1. The response now correctly populates the fallback headers (`x-litellm-attempted-fallbacks`) so callers know a fallback happened. 2. The correct model ID is passed in the streaming chunks. 3. A streaming chunk with the fallback error can be optionally sent back to the client (opt-in) by passing `include_fallback_errors: true` in the request. The format of the fallback errors while streaming is intentionally OpenAI compatible to not break existing libraries that parse these events. It was tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not delieved unexpectedly to callers by default. * fix(mistral): drop output-only reasoning fields from input messages (#30884) LiteLLM attaches reasoning_content and thinking_blocks to assistant responses. Replaying those assistant turns verbatim forwarded the fields back to Mistral, whose input schema forbids unknown keys, so the whole request failed with a 422 extra_forbidden and reasoning models became unusable across multiple turns. Strip both fields from assistant messages before the request is built, in a spot that runs ahead of the image/file branch so it applies on every path. Fixes #30835 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652) * fix(perplexity): bill search queries at the per-request price, not 1/1000 The fallback cost calculator divided search_context_cost_per_query by 1000, but that field stores the per-request price in USD: sonar is {low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published $5/$8/$12 per 1,000 requests expressed per request. The gemini cost calculator reads the same field per request with no division (its docstring calls it "the per-request cost"). The division understated search cost by 1000x on every Perplexity call that falls back to manual calculation (i.e. when the API does not return a pre-computed usage.cost). Use the value directly. Update the tests that had encoded the /1000 factor in their expectations, and drop an unused import flagged by ruff in the touched test file. * test(perplexity): update integration test search-cost expectations to per-request The integration tests still encoded the old /1000 search-cost factor, so they failed once the fallback calculator was corrected to bill search_context_cost_per_query per request. Update the four expected-cost computations (and the high-volume dollar-value comments) to match. * test(perplexity): drop unused mock imports flagged by ruff * fix: include model_access_groups when expanding all-team-models in get_team_models (#30622) * fix(fireworks_ai): return None for transcription in get_supported_openai_params Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is decommissioned. Without an explicit transcription branch, requests with request_type='transcription' fell through to the else and returned FireworksAIConfig chat-completion params. Return None instead to signal the provider does not support transcription. * fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting Without an operator gate, any authenticated caller could set include_fallback_errors=True, trigger a fallback, and read raw upstream exception messages from the x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event. Strip include_fallback_errors from request data in common_processing_pre_call_logic when expose_fallback_errors_to_caller is not set, so the router never builds the error list. Also gate _should_include_fallback_errors on the same setting as a secondary check for the streaming SSE injection path. * test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test The operator gate added in e7ff3e1 means include_fallback_errors is only honoured when general_settings.expose_fallback_errors_to_caller is True. Set that flag via monkeypatch in the test that exercises the emit path. * test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos test_convert_url called convert_url_to_base64 against a live picsum.photos URL and asserted nothing, so it added no real signal and broke CI whenever the host was unreachable (it was returning 522 and blocking this branch). Replace the live call with a mocked HTTP client and assert the produced base64 data URL, so the conversion path is exercised deterministically with no network dependency. This suite runs under VCR, which is why a transport level mock (respx) does not reliably intercept; mocking the client object itself is robust regardless. * fix(interactions): drop role from Interaction response to match Google spec Google removed the output-only role field from the Interaction schema (it now lives only on Turn), so the live OpenAPI compliance canary started failing with 'role' not in spec. Reconcile our generated types by removing role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse, stop stamping role=model in the responses-to-interactions transformation, and update the compliance and integration tests accordingly. Turn.role is kept since the spec still defines it. * fix: align all-team-models sentinel access * fix(router): forward include_fallback_errors through multi-hop fallbacks run_async_fallback received include_fallback_errors as an explicit named parameter, so it was bound out of **kwargs and never reached the nested async_function_with_fallbacks call. Multi-hop fallback chains (a fallback group that itself fails over) therefore stopped collecting fallback errors beyond the first hop when a caller opted in. Re-inject the flag into kwargs before the nested call so inner hops keep accumulating errors, which add_fallback_headers_to_response already merges across levels. --------- Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-23 07:31:44 -07:00
tin-berri	69b0dd2da0	fix(mcp): stop auth failures on the /mcp path surfacing as cancelled tool calls (#31011) ... user_api_key_auth raises ProxyException, not HTTPException, on an auth failure. The streamable-HTTP and SSE MCP handlers only re-raised HTTPException to preserve status and headers, so a ProxyException fell through to the catch-all and was flattened to a generic 500, dropping the real status (for example 401) and any WWW-Authenticate challenge. MCP clients render a 500 on the JSON-RPC POST as a cancelled or terminated session, and an OAuth client never receives the 401 it needs to re-authenticate. Because auth runs before server routing, one rejected credential fails every targeted server at once. Map ProxyException back to its real status and headers in both handlers (handle_streamable_http_mcp, handle_sse_mcp) via a small _proxy_exception_to_http_exception helper inserted before the generic except Exception. A genuine auth failure now returns its real status; a key sent without the documented Bearer prefix gets a clear 401 telling the caller to fix the header rather than a cancelled session. Regression tests assert that a ProxyException(401) raised during auth propagates as a 401 with WWW-Authenticate from both the streamable-HTTP and SSE handlers, and unit-test the converter for the 401/403/non-numeric-code cases.	2026-06-23 00:14:21 -07:00
tin-berri	c18a870746	feat(mcp): scaffold outbound_credentials package with typed Result (#31047) ... * feat(mcp): scaffold outbound_credentials package with typed Result PR1 of the MCP v2 outbound-credential migration. Adds the litellm/proxy/_experimental/mcp_server/outbound_credentials/ subpackage with a hand-rolled Ok | Error Result union (pure stdlib + typing_extensions, no new dependency) and its package surface. Nothing imports this on a live request path yet, so production behavior is unchanged; later PRs add the typed config vocabulary, the resolve_credentials dispatch, and the v1 graft. * feat(mcp): add outbound_credentials typed vocabulary (#31049) PR2 of the MCP v2 outbound-credential migration, stacked on the result.py scaffolding. Adds types.py (the AuthConfig discriminated union over seven frozen per-mode configs, CredError as an expression @tagged_union, Subject, ServerSpec, and the parse_auth_spec_kind boundary parser) and httpx_auth.py (NoOpAuth, StaticHeaderAuth). Pulls in expression>=5.6.0,<6.0 on the proxy extra for the tagged union. Construction-time tests prove illegal mode/field combinations are rejected. Nothing is wired onto a request path yet; the resolver lands next.	2026-06-23 00:10:19 -07:00
yuneng-jiang	6f6aec2930	fix(proxy): serialize team budget_limits to JSON in jsonify_team_object (#31045) ... POST /team/new with any budget_limits returned 500 because jsonify_team_object serialized members_with_roles but left budget_limits as a raw Python list, which Prisma's Json column rejects. /team/update and /key/generate worked only because each json.dumps the windows itself. Serialize budget_limits in the shared helper, guarded by isinstance(list) so the pre-serialized /team/update path is unaffected.	2026-06-22 19:10:34 -07:00
Yassin Kortam	b24b964e04	fix(passthrough,streaming): recover cost on interrupted and agentic Anthropic streams (#31035) ... Streaming and pass-through requests could be logged with $0 cost or dropped from SpendLogs entirely while the upstream provider still billed every token. This closes the leak paths not already covered by #30160, #30787 and #30788. - Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and async). Large agentic tool-use / thinking streams can make assembly re-raise as APIError from inside the except-StopIteration handler, where the sibling except does not catch it, so it escaped __next__/__anext__ and dropped the request; recover best-effort usage from the raw chunks instead - Add a usage-only fallback for Anthropic streaming pass-through: when stream_chunk_builder returns None or raises, rebuild usage from the message_start / message_delta SSE events via AnthropicConfig.calculate_usage so cache, web-search and geo tokens are priced instead of left at $0 - Decode buffered pass-through bytes with errors="replace" so a stream cut mid-multibyte-sequence still logs the usage events already received - Record response_cost into model_call_details on the pass-through success path (it is read from there, not from kwargs), matching the gemini/cohere/openai handlers - Name the key (alias + masked key) in the virtual-key BudgetExceededError so operators don't have to reverse-map spend back to a key	2026-06-22 18:51:13 -07:00
ryan-crabbe-berri	3615049071	feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools (#31031) ... * feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools GET /v1/mcp/tools returns the MCP tools available to the calling key, the same data already exposed through /mcp/tools/list and /mcp-rest/tools/list, both of which are in llm_api_routes. The /v1/mcp/tools path was in no route group, so virtual keys created from the UI (which default to allowed_routes=["llm_api_routes"]) got a 403 listing tools one way but not the other. Add it to mcp_inference_routes. Unlike /v1/mcp/server, this path has no management write counterpart, so it does not need the method-aware carve-out used for server discovery. * test(proxy): parametrize MCP inference route check over the full endpoint set	2026-06-22 17:36:26 -07:00