mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-11 03:38:38 +00:00
657 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yassin Kortam
|
3ea7f98725
|
feat(proxy): configure the coordination redis independently of the response cache (#32661)
* fix(proxy): build redis usage cache from REDIS_* env when cache backend is not Redis Selecting a semantic (or any non-Redis-KV) response cache left redis_usage_cache unset, silently downgrading cross-pod rate limits, parallel-request limits, spend coordination, and the pod lock manager to per-pod in-memory state. Fall back to a standalone RedisCache built from REDIS_* environment variables, mirroring the existing use_redis_transaction_buffer escape hatch, which now shares the same helper. Resolves LIT-3861 * feat(proxy): configure the coordination redis independently of the response cache Adds general_settings.coordination_redis, an explicit block for the Redis the proxy uses for cross-pod rate limits, parallel-request limits, spend tracking, the pod lock manager, and shared health checks. Resolution order is the explicit block, then a plain-Redis response-cache backend, then the REDIS_* environment. Cluster and sentinel targets are supported, and a cluster target now builds a RedisClusterCache so cluster-aware consumers take the cluster path. Admins can configure it from the Caching page of the dashboard via /coordination_redis/settings, which reports which source is in effect, redacts credentials on read, and offers a connection test. Settings saved there are read back at startup so they take effect on restart. Also fixes redis client construction so an explicitly configured host outranks REDIS_URL in the environment. Previously the url branch stripped the caller's host and port, so an explicit block, or a connection test typed into the dashboard, silently targeted whatever REDIS_URL named * fix(ui): move coordination_redis_settings into renamed _components directory --------- Co-authored-by: Yucheng Zhu <yucheng@berri.ai> |
||
|
Tin Chi Lo
|
5ec4c162ea | fix(mcp): let a missing server 404 on a dcr_bridge enablement instead of a misleading 400 | ||
|
Tin Chi Lo
|
41a43d5283 | feat(mcp): add dcr_bridge column and plumbing for client-forwarded auth modes | ||
|
Tin
|
aa351311c0 |
fix(mcp): spare BYOK rows when purging stale OAuth tokens and invalidate caches on server delete
LiteLLM_MCPUserCredentials stores BYOK API keys in the same column as per-user OAuth tokens, so the purge on a mint-relevant config change now deletes only rows whose payload decodes as an OAuth2 credential, each by its (user_id, server_id) pair, instead of every row for the server. An api_key server whose url changes purges nothing. delete_mcp_server now also invalidates each enumerated user's cached token so a re-created server reusing the id cannot serve tokens minted for the deleted one, and both cache drops are best-effort |
||
|
Tin
|
c75184bec9 |
fix(mcp): make the pre-update identity snapshot advisory so a read failure cannot fail the edit
The snapshot read only feeds the stale-token purge decision; leaving it unguarded meant a failed read would 500 an edit whose update would have succeeded, and it broke test_edit_mcp_server_redacts_credentials, whose mocked prisma is not awaitable on the un-patched get_mcp_server path. A failure now logs and skips the purge, consistent with the purge half already being best-effort. Adds the first endpoint-level coverage of the edit purge wiring: purge on a mint-relevant change, no purge when the identity is unchanged, and edit success with purge skipped when the snapshot read raises |
||
|
Yassin Kortam
|
cda99a08c8
|
fix(proxy): surface OAuth error params in SSO callback (#32433)
When an IdP denies SSO access it redirects back to /sso/callback with error and error_description query params and no code param. The callback previously fell through to the provider token exchange, which failed with a generic "'code' parameter was not found in callback request" 400 that hides the real denial reason. Raise a 401 that surfaces the IdP's error and description instead. Ported from #26640 with conflicts resolved against current staging |
||
|
devin-ai-integration[bot]
|
93c047d52e
|
feat(proxy): make Microsoft Graph endpoint configurable for GCC High (LIT-4282) (#32517)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
f982b67d78
|
fix(proxy): harden secret name validation for external secret manager integrations (LIT-4201) (#32092)
key_alias can become the secret name used by external secret manager integrations (HashiCorp Vault, CyberArk Conjur) when store_virtual_keys is enabled. Add raise_if_unsafe_secret_name, a shared validation check applied unconditionally before a secret name reaches either integration or the /key/generate, /key/update, and /key/regenerate API boundary, independent of the existing enable_key_alias_format_validation opt-in flag. Also hardens the Vault URL builder to percent-encode reserved characters in secret_name (preserving "/" and "@"), and switches the Conjur policy body to a real YAML serializer instead of raw string interpolation. |
||
|
Yassin Kortam
|
6f6bd45681
|
perf(auth): negative-cache missing user/key lookups on the request hot path (#32368) | ||
|
tin-berri
|
1fb2b4aef4
|
fix(mcp): drop the cached per-user OAuth token when the credential row changes (#32302)
* fix(mcp): drop the cached per-user OAuth token when the credential row changes The v2 authorization_code chain Cached(Refreshing(V2PerUserTokenStore)) caches a positive token until its expires_at (or 300s without one), and CachedOAuthTokenStore.invalidate had no callers, so a re-authorization or revocation wrote the DB while egress kept serving the replaced token from the in-process cache until its TTL. LazyPerUserOAuthTokenStore now exposes invalidate, MCPServerManager threads it to the write side, and the three credential write sites (the OAuth callback, the Tools-tab persist endpoint, and the revoke endpoint) drop the cache entry after the row changes. The v2 refresher's own persist stays untouched; RefreshingTokenStore already feeds the rotated token back into the cache in the same fetch * test(mcp): pin cache invalidation on the revoke already-gone branch Greptile's review flagged that only the happy-path delete asserted the invalidate; a refactor moving the call inside the try block would silently skip the cache drop when the row was already deleted by a concurrent request while the cache still held the revoked token. The new test fails on exactly that mutation * test(mcp): cover invalidate on the redis-backed lazy store path Codecov flagged the redis fast path of LazyPerUserOAuthTokenStore.invalidate as unexercised; the existing invalidate tests only ran the no-redis chain. The new test builds the redis chain via a fetch and asserts a subsequent invalidate reaches the same store instance without a rebuild |
||
|
Yassin Kortam
|
bcd52754de
|
feat(rate_limit): support per-tag rpm limiting on a single key (#31502)
Add a tag_rpm_limit field to virtual keys so each request tag gets its own independent RPM counter on the v3 rate limiter. A key configured with per-tag limits tracks each tag/group separately, and requests whose tag has no configured limit fall back to the key-level limit. Includes the dashboard UI to manage per-tag limits on key create and edit. Resolves LIT-3147 |
||
|
tin-berri
|
db2402754a
|
feat(mcp): let users select the entra_obo token_exchange profile in the UI and API (#32144)
* feat(mcp): let users select the entra_obo token_exchange profile in the UI and API
The backend token_exchange arm supports two wire dialects via token_exchange_profile
("rfc8693" default, or "entra_obo" for Microsoft Entra's On-Behalf-Of, the RFC 7523
jwt-bearer grant), but it could only be set through config.yaml. This surfaces it to the
create/update REST API and the dashboard so an admin can create an entra_obo server there,
completing the parity started in the parent PR for the other token-exchange fields.
token_exchange_profile becomes a dedicated column on LiteLLM_MCPServerTable, mirroring the
sibling fields: it is added to the request models, read column-first in
build_mcp_server_from_table with the credentials-blob as a back-compat fallback and a
default of rfc8693, and carried through both runtime-to-table builders so registry
round-trips preserve it. It is a non-secret dialect selector, so it is not scrubbed from
non-admin or virtual-key responses.
In the dashboard a Profile dropdown (RFC 8693 vs Microsoft Entra OBO) is added to the
token-exchange section. Entra OBO carries the target resource in the scope, so selecting it
makes the scope required and hints the api://<app-id>/.default form, while audience and
subject_token_type (which that dialect ignores) are hidden.
* fix(mcp): extend the blob-to-column lift and non-admin scrubbing to token_exchange_profile
token_exchange_profile gets the same storage contract as the other three
token-exchange settings: the column is authoritative, a blob copy is the legacy
shape — lifted into the column on every write and stripped from the stored
blob — and switching auth_type away from token exchange clears it
(_AUTH_FLOW_SCOPED_FIELDS). Both restricted-view sanitizers scrub it for
uniformity, and the edit form's auth-switch payload nulling includes it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(mcp): assert every token-exchange setting is configurable via config.yaml
Pins the config surface: token_exchange_endpoint, audience, subject_token_type
and token_exchange_profile load from top-level config keys onto the built
server and through to the resolver spec; omitted keys resolve to their
documented defaults (RFC 8693 subject token type, rfc8693 profile), and
token_exchange servers need no oauth2_flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
tin-berri
|
ff6dc33291
|
feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard (#31772)
* feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard OAuth 2.0 Token Exchange (RFC 8693, a.k.a. OBO) for MCP servers could previously only be configured through config.yaml; the create/update REST API and the dashboard had no way to express it. This wires token_exchange_endpoint, audience, and subject_token_type end to end. These three are persisted as dedicated columns on LiteLLM_MCPServerTable, mirroring how token_url and oauth2_flow are stored, so the edit form prefills them and they are unaffected by the credentials-blob clearing on auth_type change. build_mcp_server_from_table reads the columns first and falls back to the credentials blob so servers persisted before the columns existed still load. client_id and client_secret continue to ride the existing encrypted credentials path. On the dashboard, "OAuth Token Exchange (OBO)" is a distinct auth-type option with its own field section. The McpOAuthMode classifier gains a token_exchange arm keyed off auth_type; the previous catch-all oauth2 mode was renamed from "obo" to "authorization_code" so the two on-behalf-of mechanisms are no longer conflated. The token-exchange IdP endpoint and audience are scrubbed from non-admin and virtual-key responses, matching how token_url is treated. * fix(ui): gate the MCP list-401 re-auth on authorization_code, not token_exchange The renamed 401 gate keyed off isTokenExchange, but that condition exists for authorization_code: when a stored per-user credential is present yet the tools/list 401s (the backend's refresh could not mint a token), the user must re-authorize via the browser flow. token_exchange has no gateway-side authorize step, so the Authorize gate never applied to it. The prior isObo flag was undefined (a compile error) and, per this file's convention and its tests, meant authorization_code; renaming it to isTokenExchange changed the behavior and broke the mcp_tools auth-gate test for an authorization_code server whose token expired with no usable refresh. Gate on isAuthorizationCode instead and drop the now-unused isTokenExchange * fix(mcp): clear flow-scoped endpoint config when a server's auth_type changes Switching an existing oauth2 server to oauth2_token_exchange left the old flow's token_url on the row. The OBO resolver treats token_exchange_endpoint or token_url as the configured exchange endpoint, so the stale value both suppressed the RFC 9728/8414 discovery this PR adds and sent the exchange grant (client credentials plus the user's subject token) to the previous flow's token endpoint update_mcp_server now mirrors its existing stale-credentials rule for the flow-scoped columns (authorization_url, token_url, registration_url, oauth2_flow, token_exchange_endpoint, audience, subject_token_type): when auth_type changes, each one is cleared unless the same request explicitly provides it, so a deliberate override in the switch request still wins. Updates that keep the auth_type never touch these columns, which keeps legacy OBO rows that use token_url as their exchange endpoint working The edit form sends explicit nulls for the previous flow's fields on an auth type switch; antd preserves unmounted field values by default, so without this the old token_url would be re-sent verbatim and read as an explicit override. Transitions are detected against the persisted auth_type, so saves that keep the auth type send nothing extra Reported by Cursor Bugbot on the PR * fix(mcp): lift legacy blob token-exchange settings into their columns on every write The three token-exchange settings live in dedicated columns but also exist on MCPCredentials as the pre-column REST shape. Writes now lift incoming blob values into the columns (an explicit top-level value wins, including an explicit null) and strip them from the stored blob; the same-auth credentials merge migrates legacy rows the same way. The read-time column-or-blob fallback then only ever serves rows current code has never written, so clearing a column to re-enable RFC 9728/8414 discovery can no longer be silently undone by a stale blob copy. Also asserts the auth-switch clearing fires on the external fields_set path (PUT /v1/mcp/server). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(mcp): single source for the RFC 8693 default subject_token_type The default was applied at four egress build sites plus two model defaults, each with its own copy of the literal. All sites now share DEFAULT_SUBJECT_TOKEN_TYPE from litellm.types.mcp. A DB-level DEFAULT is deliberately not used: Prisma writes explicit values on insert, so a column default would rarely apply, and NULL-means-RFC-default keeps existing rows correct. Also documents two review decisions in place: the audience column keeps the RFC 8693 parameter name (RFC 8707 resource indicators are already a separate concept named resource in the v2 egress types), and the migration's out-of-order timestamp is safe under prisma migrate deploy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: fix import sort order in outbound_credentials/types.py (I001 strict budget) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): purge legacy blob copies when a token-exchange column is written without credentials The migrate-on-write in the credentials merge lifts blob values into null columns, which is correct for legacy rows but could repopulate a column an admin had cleared in an earlier no-credentials update (that path never touched the blob, so the stale copy survived to be lifted later). An explicit token-exchange column write (set or clear) now migrates the row even when the update carries no credentials: untouched null columns are lifted, every blob copy is stripped, and unrelated blob keys stay as-is. A cleared column can then never be resurrected, because no write path leaves a blob copy behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(mcp): state the blob-to-column lift contract on the legacy credential keys The three token-exchange keys on MCPCredentials are the pre-column REST shape (the only REST shape from 2026-05 until this PR). Document on both the blob type and the request models that the dedicated columns are authoritative and that writes lift blob values into them and strip the stored copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): scrub subject_token_type in the non-admin and virtual-key sanitizers The other two token-exchange fields were cleared while subject_token_type was left visible. It is a public RFC 8693 URN with no disclosure value, but the sanitizers' rule is that these views receive no token-exchange config at all — cleared for uniformity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
tin-berri
|
7ce573e6e8
|
fix(mcp): stop 'Team doesn't exist' warnings for UI dashboard sessions (#32348)
UI session tokens carry the virtual team_id litellm-dashboard (UI_TEAM_ID), which is never persisted. The MCP team-permission helpers passed it to get_team_object anyway, so every dashboard MCP listing raised a 404 per lookup that was swallowed into per-server 'Failed to get allowed tools for server' warnings (plus the sibling 'allowed MCP servers for team' and 'MCP access groups for team' warnings) and wasted DB queries. The 404 also escaped past the key-level permission handling in get_allowed_tools_for_server, dropping key tool restrictions for such sessions. Short-circuit the virtual team before the DB lookup in the three helpers, mirroring the existing UI_TEAM_ID handling in agent_permission_handler. Also reject /team/new with the reserved team_id, since a real row would bind its budget and permissions to every UI session |
||
|
Yassin Kortam
|
68f997dd09
|
feat(budget): throttle keys after spend limit instead of revoking access (#31300)
Add an opt-in mode so a key that exceeds its own max_budget is throttled to a globally configured percentage of its TPM/RPM instead of being blocked entirely. A new litellm_settings global, budget_exceeded_throttle_percentage, sets the fraction (e.g. 0.1 = 10%). A per-key throttle_on_budget_exceeded flag (stored in key metadata via the existing management-endpoint metadata routing) opts the key in. When both are set and the key is over budget, the budget check records the percentage on a request-scoped budget_throttle_pct instead of raising, and the rate limiter scales the key's configured TPM/RPM by it. Keys without the flag keep hard-blocking; team/user/org budgets are unaffected. The throttle is recomputed from the key's original limits on every request and the decision is cleared before the auth object is cached, so it never compounds across requests. Both the budget read-time check and the budget reservation path honor the opt-in, and both the v3 and legacy rate limiters apply the scaling. Enabling throttle_on_budget_exceeded is proxy-admin only. It converts an admin-imposed hard budget block into a soft throttle that keeps spending past max_budget, so a non-admin must not be able to self-opt-in and bypass their own spend cap. Both /key/generate and /key/update reject a non-admin setting it to true (update only gates the transition to enabled, so a non-admin can still edit other fields and turn the flag off). This matches the feature being wholly proxy-admin operated: the global percentage is admin-only too. A key that opts in but has no TPM or RPM limit has nothing to scale, so it stays hard-blocked rather than serving unlimited requests past its budget (fail-safe). The global budget_exceeded_throttle_percentage is configurable from the admin UI (Settings -> General Settings), persisted through litellm_settings so it survives a restart, not only from config.yaml. Resolves LIT-3894. Scope for LIT-3893. |
||
|
yuneng-jiang
|
4851cba831
|
Merge pull request #32075 from BerriAI/litellm_/admiring-sammet-3b1831
feat(cache): back the Redis URL and Database Index UI fields end-to-end |
||
|
tin-berri
|
76eeaf2381
|
feat(mcp): persist oauth2_flow explicitly on create instead of inferring it at read time (#32288)
* feat(mcp): persist oauth2_flow explicitly on create instead of inferring it at read time The UI create payload never carried oauth2_flow, so every UI-created oauth2 server persisted a null flow and relied on _resolve_oauth2_flow's field-shape inference at registry build. That inference cannot tell a DCR-registered interactive server (client creds + token_url, no persisted authorization_url) from an M2M server unless endpoint discovery succeeds first, and the dashboard cannot reproduce it at all because credentials are redacted in responses The create form now persists the selected flow for oauth2 servers: authorization_code for Interactive (PKCE), client_credentials for M2M. The REST create endpoints stamp an omitted oauth2_flow server-side with the same discriminator the legacy inference uses, run at write time where the payload carries plaintext credentials, so the decision is made once with full information and stored. Applied to the admin create, the BYOM submission, and the temporary session-server endpoints The edit form derives its flow display from oauth2_flow instead of token_url presence (token_url is present on authorization_code servers too, so it cannot distinguish M2M) and deliberately never writes oauth2_flow: it has no flow selector, so a write from edit could only erase an explicit value, including the authorization_code stamp the DCR flow persists. Regression tests pin all of this down Second step of persisting oauth2_flow at every write site so the legacy inference can eventually be deleted; the backfill for existing null rows lands next * refactor(mcp): name the create-time flow stamp for its fallback-only contract stamp_omitted_oauth2_flow with a dedicated explicit-value early return and the shape check renamed to has_m2m_shape, so the precedence (caller's oauth2_flow always wins, inference only fills an omitted field) reads directly off the code |
||
|
tin-berri
|
2e38da6b3e
|
feat(mcp): add entra_obo profile to the token_exchange (OBO) arm (#31983)
* feat(mcp): add entra_obo profile to the token_exchange (OBO) arm
Microsoft Entra On-Behalf-Of uses the RFC 7523 jwt-bearer grant rather than RFC 8693, so the existing token_exchange arm cannot mint tokens against Entra. This adds an entra_obo profile on TokenExchangeConfig that switches the request form to Entra's jwt-bearer OBO dialect (the inbound token as assertion, the target resource carried in scope, and requested_token_use=on_behalf_of), while reusing the shared caching, single-flight, TTL, and fail-closed machinery. The exchanger is renamed from Rfc8693TokenExchanger to OboTokenExchanger since it now serves both dialects
The profile is threaded through the config-load path, the DB credentials blob, and the MCPCredentials request schema, so an operator can select it from YAML or the management API. It rides in the existing credentials JSON blob, so there is no new auth_type and no DB migration
Resolves LIT-4163
* feat(mcp): propagate the Entra Conditional Access step-up challenge on the OBO 401
An entra_obo exchange that the IdP rejects for Conditional Access returns a 4xx with
error=interaction_required and a claims blob the client must satisfy to step up. The arm
dropped both and emitted a static RFC 9728 challenge, so a CA-protected Entra upstream was
unreachable through the gateway. The provider now reads the RFC 6749 error code and the claims
string off the rejection body (error_description is still never carried; it can leak IdP
internals), threads them through SubjectTokenRejected -> CredError.unauthorized, and the
challenge builder folds them into WWW-Authenticate: the machine error only when it is a plain
OAuth token (guards against header injection from a hostile body) and the claims base64-encoded
in a claims parameter, the convention MSAL-family clients decode. With neither field the header
is byte-identical to the static challenge. The multi-server aggregate still absorbs a
step-up 401 to an empty listing; only single-server routes surface it
* fix(mcp): use error=insufficient_claims for the Entra step-up challenge
Per Microsoft's claims-challenge format, a WWW-Authenticate carrying a claims challenge must set
error=insufficient_claims (the value MSAL-family clients key on to recognize the challenge and
replay the claims), not the raw token-endpoint code. The challenge now sets insufficient_claims
whenever a claims blob is present and keeps invalid_token otherwise, with a step-up-accurate
error_description in the claims case. The presence of claims now drives the error value, so the
raw oauth_error no longer needs threading from the provider through CredError to the edge; that
plumbing is removed (the provider still reads the error code for its gateway-fault classification).
Both the error value and the base64 claims are fixed-alphabet, so nothing from the IdP body reaches
the header unescaped. Cross-checked field-by-field against Microsoft Learn; a bogus jwt-bearer OBO
POST to the real login.microsoftonline.com/common endpoint confirmed Entra recognizes the grant and
returns the error shape the parser reads. Follow-up: also emit authorization_uri alongside
resource_metadata for strict non-MCP MSAL clients (RFC 9728 resource_metadata already serves MCP)
* fix(mcp): filter blank scopes on the config-load path so entra_obo fails closed
The DB-build path already runs YAML/DB scopes through _extract_scopes (which drops blanks), but the
config-load path read server_config["scopes"] raw. A YAML `scopes: [""]` therefore reached the
exchanger as a ("",) tuple: non-empty, so the entra_obo `not config.scopes` precondition skipped its
fail-closed misconfigured path and the form builder POSTed an empty scope to the IdP instead of
failing before any network call. Config-load now filters blanks the same way, so an all-blank list
normalizes to None and the precondition fails closed. Regression test loads a blank-scope entra_obo
server and asserts the exchange returns misconfigured without POSTing
|
||
|
Yuneng Jiang
|
b86c01c624
|
fix(cache): mask inline url credentials and drop discrete username under url precedence
Two follow-ups from review of the url/db work. A Redis/Valkey url can embed a password (redis://:secret@host:6379/1), but _CACHE_SENSITIVE_FIELDS only masked the discrete password and sentinel_password, so a stored password-bearing url came back in plaintext from every GET /cache/settings. Add url to the masked set so it gets the same masked-on-read treatment as password. The url-precedence resolver dropped host/port/db/password but not username, even though a url can encode a username too (redis://user:pass@host). Left in, the discrete username rode along and could contradict the url. Add username to the overridden set and update the Redis URL help text to list it among the fields url takes precedence over. Tests: GET masks a password-bearing url (secret never returned verbatim) while a non-credential field is untouched, and the resolver drops a discrete username when a url is present. |
||
|
Yuneng Jiang
|
611b8dee18
|
feat(cache): back the Redis URL and Database Index UI fields end-to-end
The typed cache-settings form already renders a Redis URL and a Database Index field, but the backend never defined them, so GET /cache/settings could not round-trip a saved value into the form and the "URL takes precedence over Host/Port/Password/Database Index" help text the UI shows was not actually enforced anywhere. Add the url and db entries to CACHE_SETTINGS_FIELDS so the endpoint knows about them, and add _resolve_cache_url_precedence: when a non-empty url is present it wins and the discrete host/port/db/password fields are dropped before the settings are tested or persisted, matching how litellm._redis resolves the connection at runtime (redis.Redis.from_url ignores them). Cluster mode is exempt because it authenticates via the discrete fields rather than a url. Both test and save paths go through the resolver so the stored config is unambiguous. This finishes LIT-3996: operators can now isolate the cache into a logical database (e.g. redis://host:6379/1) entirely from the Admin UI instead of hardcoding REDIS_URL in the environment. |
||
|
Krrish Dholakia
|
28ddad271e
|
feat(proxy): add key-level budget_fallbacks to reroute requests when a per-model budget is exceeded (#31783) | ||
|
tin-berri
|
15ff389eb4
|
fix(mcp): persist DCR client_id so interactive OAuth token refresh works (#31912)
* fix(mcp): persist DCR client_id so interactive OAuth token refresh works Interactive authorization_code MCP servers register an OAuth client via Dynamic Client Registration (RFC 7591) during the authorize flow, but the minted client_id and the discovered token_url were returned to the caller and never written to the server row. The autonomous refresh_token grant reads client_id, client_secret and token_url off the server, so an expired access token could not be refreshed; the user was bounced back to re-authorize and tools/list returned zero tools Persist the DCR client_id (plus client_secret and token_endpoint_auth_method when the registration returns them) and the discovered token_url onto the server row, reusing the encrypt_credentials write that client_credentials and token exchange already use, then refresh the in-memory registry so the value is live at refresh time. Both the v1 refresher and the v2 AuthorizationCodeRefresher read those same fields, so egress needs no change * fix: reuse persisted MCP DCR clients * fix: reuse persisted MCP DCR clients --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
yucheng-berri
|
b61fae397b
|
fix(proxy): admin-gate permissions on _process_single_key_update (LIT-4137) (#32002)
The bulk-update entrypoints `/key/bulk_update` and `/team/key/bulk_update` route through `_process_single_key_update`, not through `_validate_update_key_data`, so the `permissions` gate LIT-4092 wired into the single-key path never fires on bulk. Currently safe by construction: `BulkUpdateKeyRequestItem` doesn't declare `permissions` (Pydantic silently drops it), and `KeyUpdateFields` uses `model_config = ConfigDict(extra="forbid")` (Pydantic 422s at parse time). Neither structural barrier is enforced by tests on the field itself; a future widening of either allowlist to include `permissions` would reopen the class silently. This wires `_check_permissions_caller_permission` into `_process_single_key_update` right after `_validate_max_budget`, before `prepare_key_update_data`. Zero behavior change today for any caller routing through the current bulk request models; a defense-in-depth gate for the class. Tests: - test_process_single_key_update_non_admin_permissions_rejected - test_process_single_key_update_non_admin_permissions_explicit_empty_rejected Both mutation-killed against removing the gate. Full mapped test file (341 tests) green. |
||
|
yucheng-berri
|
b426925824
|
fix(proxy): admin-gate allowed_routes presence on /key/update and /key/regenerate (LIT-4139) (#31987)
`_check_allowed_routes_caller_permission` previously keyed its admin-only rule on truthiness. The refactor adds an `allowed_routes_was_provided` keyword param that raw-body call sites populate from `"allowed_routes" in data.model_fields_set`, so a caller that omits the field (default flows through) is distinct from one that sends any explicit value. Four raw-body call sites now pass `allowed_routes_was_provided=...`: `_common_key_generation_helper`, `generate_service_account_key_fn`, `_validate_update_key_data`, and `regenerate_key_fn`. Two derived-value call sites keep the pre-fix shape: the post-`handle_key_type` recheck at `_common_key_generation_helper` and the mirror in `regenerate_key_fn`. Both pass values produced by `handle_key_type` (not by the request body), so `allowed_routes_was_provided` stays False and the `allow_safe_presets=True` carve-out continues to accept the `llm_api_routes` / `info_routes` presets. In `regenerate_key_fn` the gate runs before the `premium_user` license check, matching the LIT-4092 ordering. `test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate` pins the ordering; it fails on a swap of the two gates. Tests in `tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py` under `TestAllowedRoutesCallerPermission`: - test_non_admin_generate_key_explicit_empty_allowed_routes_rejected - test_non_admin_update_key_explicit_empty_allowed_routes_rejected - test_non_admin_update_key_explicit_null_allowed_routes_rejected - test_non_admin_regenerate_key_explicit_empty_allowed_routes_rejected - test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate - test_helper_accepts_derived_safe_preset_for_non_admin - test_helper_rejects_derived_unsafe_preset_for_non_admin - test_helper_rejects_when_provided_and_none_without_typeerror The four attack-vector tests fail on the pre-fix HEAD and pass on this commit. Three helper-level tests pin the derived-value branch and the load-bearing None guard; each is mutation-killed against a targeted change to the frozenset or the guard. Full mapped test file (347 tests) green. |
||
|
yucheng-berri
|
6dcbac88b8
|
fix(proxy): admin-gate permissions on /user/new and /user/update (LIT-4138) (#31998)
`NewUserRequest` and `UpdateUserRequest` inherit `permissions` from
`GenerateRequestBase`. `/user/new` passes the field into
`generate_key_helper_fn` which persists it on the auto-created key,
so an org admin who lands on `/user/new` (the route allowlist accepts
org_admin callers when the request body names an org where they hold
that membership) can mint a key with proxy-wide capabilities such as
`get_spend_routes`.
This wires the existing `_check_permissions_caller_permission` helper
into `new_user` and `_update_single_user_helper`. The helper's presence
check keys on `data.model_fields_set`, so an omitted field flows
through untouched and an explicit `{}` / `null` from a non-admin is
rejected 403 the same as any other value.
`_update_single_user_helper` is shared by `/user/update` and
`/user/bulk_update`, so both paths inherit the gate.
Tests in
`tests/test_litellm/proxy/management_endpoints/test_internal_user_endpoints.py`:
- test_new_user_non_admin_permissions_non_empty_rejected
- test_new_user_non_admin_permissions_explicit_empty_rejected
- test_new_user_non_admin_omits_permissions_succeeds (control)
- test_new_user_admin_can_set_permissions (control)
- test_update_single_user_non_admin_permissions_rejected
- test_update_single_user_non_admin_permissions_explicit_empty_rejected
The four attack-vector tests fail on the pre-fix HEAD and pass on this
commit. Full mapped test file (79 tests) green.
|
||
|
tin-berri
|
c370503091
|
fix(mcp): gate OAuth authorize/token/register/discovery on auth_type=oauth2 (#31736)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* fix(mcp): gate OAuth authorize/token/register/discovery on auth_type=oauth2 A non-oauth2 MCP server (notably auth_type=none, access-group gated) has no client_id and no authorization URL, yet the gateway OAuth endpoints did not check auth_type. authorize() raised "client_id is required" before the auth_type was ever examined, and the .well-known discovery builders always advertised authorization_servers / authorization_endpoint / token_endpoint / registration_endpoint, so spec-compliant MCP clients were pointed at an OAuth flow that can never succeed. Add an auth_type != oauth2 guard to the authorize, token, register, protected-resource and authorization-server paths (covering the internal UI OAuth endpoints too). The discovery guard sits after the OAuth pass-through branch so genuine pass-through servers keep proxying their upstream metadata. oauth2 servers are unaffected. * fix(mcp): accurate non-oauth2 message; 404 unknown discovery names to close enumeration oracle Address review feedback on the auth_type gate. The 400 message no longer claims access is governed by access groups, which is only true for auth_type=none; it now states that the gateway runs the OAuth client_id/authorize/token/register flow only for oauth2 servers and that the server is reached using its configured auth_type, which is accurate for every non-oauth2 type (api_key, oauth2_token_exchange, etc.). The discovery gate previously 404'd a named non-oauth2 server but still returned 200 metadata for an unknown name, which both serves a broken document for a typo and lets an unauthenticated caller enumerate non-OAuth server names by comparing 404 vs 200. A named discovery request now returns 200 only when it resolves to an oauth2 server; unknown (or hidden) and non-oauth2 names return the same 404. Root discovery and pass-through servers are unaffected. * Apply suggestions from code review Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
f5f8ba93fa
|
fix(mcp): tighten role-based visibility on /v1/mcp/server/submissions (#31932)
Route non-full-admin callers through _sanitize_mcp_server_list_for_non_admin, matching the pattern the fetch and list handlers adopted. Replace the two regression tests that pinned the old partial-blank behavior with a sanitize/full-admin pair mirroring the fetch/list coverage. Resolves LIT-3929 |
||
|
Sameer Kankute
|
b96f1aa686
|
fix(mcp): byom visibility, preview UX, and admin settings gating (#31809)
* fix(ui): show info message when MCP tool preview returns 403
Internal users submitting MCP servers hit an admin-only preview endpoint; replace the red connection error with a clear review notice while leaving other failures unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): let BYOM submitters see their approved servers
Approved user-submitted MCP servers defaulted to no access groups and allow_all_keys=false, so submitters could not see them after admin approval. Grant creator visibility for active submissions in get_allowed_mcp_servers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Improve dialogue box
* fix(security): restrict MCP semantic filter settings to proxy admins
Add an explicit PROXY_ADMIN check on PATCH /update/mcp_semantic_filter_settings
and hide Semantic Filter and Network Settings tabs from non-admin users in
the MCP Servers UI.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): use list[str] instead of List[str] to satisfy UP006 budget
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(mcp): cache BYOM submitter server lookup with 60s TTL
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: fix ruff format and prettier formatting
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: preserve approved BYOM server visibility
* fix(mcp): keep no-mcp-servers opt-out absolute and gate BYOM union by key scope
The autofix in
|
||
|
yucheng-berri
|
99c65ea6dd
|
fix(proxy): admin-gate permissions on /key/update and /key/regenerate (LIT-4092) (#31810)
The `_check_permissions_caller_permission` helper introduced in #31469 was only wired into `_common_key_generation_helper`. This change wires it into `_validate_update_key_data` and `regenerate_key_fn` so the three write paths share the admin gate, and refactors the helper to accept the full request model so it can key on `"permissions" in data.model_fields_set` rather than truthiness. The presence check keeps the model-level omit default flowing through unchanged while treating any explicit value (including `{}` / `null`) as an admin-only write. In `regenerate_key_fn` the gate is placed before the `premium_user` license check so the rejection is consistent across premium and non-premium deployments. That ordering is pinned by `test_regenerate_key_non_admin_permissions_rejected_before_enterprise_gate` Tests in tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py: - test_update_key_non_admin_permissions_non_empty_rejected - test_update_key_non_admin_permissions_explicit_empty_rejected - test_update_key_non_admin_permissions_explicit_null_rejected - test_update_key_non_admin_omits_permissions_succeeds (control) - test_update_key_admin_can_set_permissions (control) - test_regenerate_key_non_admin_permissions_rejected - test_regenerate_key_non_admin_permissions_explicit_empty_rejected - test_permissions_explicit_empty_rejected_for_non_admin_on_generate - test_regenerate_key_non_admin_permissions_rejected_before_enterprise_gate Mutation-killed against gate removal on either wire, against reverting the helper to a truthiness check, and against reordering the gate past the enterprise-license check |
||
|
Krrish Dholakia
|
cca71a07c2
|
feat(mcp): add mcp_tool_search virtual tools for large tool catalogs (#31777)
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
|
||
|
yucheng-berri
|
bfb8ffccb8
|
feat(proxy): audit remaining system-wide settings updates (#31754)
* feat(proxy): audit remaining system-wide settings updates Extends the audit logging framework introduced in the parent PR to the rest of the LiteLLM_Config writers and the two adjacent settings tables: /config/update (general, environment_variables, litellm_settings, router_settings sections), /config/field/update, /config/field/delete, /config/callback/delete, /update/default_team_settings, /update/mcp_semantic_filter_settings, /add/allowed_ip, /delete/allowed_ip, /update/sso_settings, /update/ui_theme_settings, /update/ui_settings. Each writer records the actor, action, the affected config section, and a redacted before/after snapshot. SSO and UI settings rows use their own table_name (LiteLLM_SSOConfig, LiteLLM_UISettings). The /config/callback and /update/sso_settings audits fire BEFORE the proxy reload and the env cleanup step respectively, so a failure in either leaves the audit row intact. The audit-actor parameter on _update_litellm_setting is now required rather than optional; the chokepoint covers default_team and mcp_semantic_filter for free, and a future caller that forgets the actor fails loudly instead of silently skipping the audit. The two direct-calling tests pass a dummy actor. The environment_variables section redacts every value rather than relying on key-name matching, because it carries credentials under non-secret-looking uppercase keys (e.g. DATABASE_URL). * fix(proxy): capture redacted SSO before-snapshot in audit log Greptile review of #31754 flagged update_sso_settings as the one endpoint where before_value is permanently None, so the LiteLLM_SSOConfig audit trail has no pre-change state. An auditor reviewing a secret-rotation event could see what the SSO settings were changed to but not what they were before. Read the existing SSO row before the upsert, decrypt it via proxy_config._decrypt_db_variables, and pass it as before_value. create_config_audit_log's secret-name redaction then masks the *_client_secret fields, so neither the old nor the new plaintext secret lands in the audit row. Add a regression test asserting the before-snapshot reflects the pre-change values for non-secret fields (google_client_id) and is redacted for secret fields (google_client_secret). Mutation-checked against reverting to before_value=None. The pre-existing SSO tests now also mock litellm_ssoconfig.find_unique since the endpoint reads it; the read returns None for tests that do not care about the before-state. * fix: remove committed zero init migration * refactor(proxy): audit config writes via asyncio.create_task everywhere PR A's chokepoint audit call was refactored from a blocking await to asyncio.create_task so that a post-save audit-log failure could not surface as a 500 to the caller. The 12 other audit call sites added in this PR were still using await, reintroducing the exact 500-after-commit exposure at every sibling endpoint. Wrap them all in asyncio.create_task to match the model_management_endpoints / key_management_endpoints / hooks / config_override_endpoints / team_callback_endpoints / cache_settings_endpoints house pattern, so the codebase tells one story. The two direct-invocation tests (test_update_config_general_settings and test_delete_config_general_settings, which call the handler in-process rather than via TestClient) yield with `await asyncio.sleep(0)` after the handler returns so the scheduled audit task runs before the assertion. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Krrish Dholakia
|
846dbecbf2
|
feat(proxy): support object_permission in default_key_generate_params (#31776)
* feat(proxy): support object_permission in default_key_generate_params default_key_generate_params filled in a fixed whitelist of scalar fields plus a full-replace for models/metadata, but never touched object_permission, so admins had no way to set a default (e.g. mcp_tool_search_enabled, vector_stores) applied to every new key. Merge object_permission field-by-field instead of replacing it wholesale, so a caller-supplied field (e.g. mcp_servers) is preserved alongside defaulted fields the caller left unset. * ci: retrigger proxy_pass_through_endpoint_tests (suspected flake, unrelated to this PR's diff) * fix(proxy): apply default object_permission after team-scope validation Injecting the default before validate_key_vector_stores_against_team / validate_key_search_tools_against_team ran meant a default containing a team-scoped field (e.g. vector_stores) looked like a caller-requested permission, turning ordinary non-admin personal key creation into a 403. Merge the default into data_json after those checks instead, and guard against a non-dict default value. |
||
|
ryan-crabbe-berri
|
3dce3daff6
|
feat(proxy): type Customer Management response_model for OpenAPI coverage (#31043)
* feat(proxy): type Customer Management response_model for OpenAPI coverage Add response_model to the five remaining untyped /customer operations (block, unblock, new, update, delete) so the generated OpenAPI schema documents a concrete response body. new/update reuse the canonical LiteLLM_EndUserTable (matching info/list); block, unblock, and delete get small dedicated models in litellm/types/proxy/management_endpoints/customer_endpoints.py. Together with the already-typed info/list/daily-activity routes this brings the Customer Management group to full response_model coverage. Regression tests assert each public /customer/* route declares the expected response_model and that /customer/new surfaces a typed schema in app.openapi(), so dropping a response_model fails CI. * fix(proxy): keep budget_id in typed customer responses Address review feedback on the Customer Management response_model typing. Greptile flagged that response_model=LiteLLM_EndUserTable on /customer/new and /customer/update silently drops fields the raw Prisma model_dump() echoed. Checking the schema, budget_id is the only such scalar column that was missing from the Pydantic model (created_at/updated_at/tpm_limit do not exist on litellm_endusertable), so add budget_id to LiteLLM_EndUserTable. This restores budget_id on new/update and also fixes the pre-existing gap where /customer/info and /customer/list (already typed) dropped it, which the UI Customer type expects. A regression test pins budget_id surviving the response_model filter on /customer/update. Also document UnblockUsersResponse.blocked_users via a Field description: it holds the users that remain blocked after the call. The key name predates this PR and is kept to avoid a backwards-incompatible rename on a beta route. * fix(proxy): keep nested budget fields in customer responses response_model=LiteLLM_EndUserTable nests the budget as the narrow write allowlist LiteLLM_BudgetTable, which silently drops the server-managed fields the customer endpoints used to return (budget_reset_at, created_at). Introduce CustomerResponse, a thin response model that nests LiteLLM_BudgetTableFull (the repo's budget response model), and apply it on /customer/new, /customer/update, /customer/info and /customer/list. list also builds CustomerResponse so its budget isn't narrowed at construction time. created_by/updated_at/updated_by remain omitted, matching how budgets are returned elsewhere. The shared LiteLLM_EndUserTable is left untouched: it's constructed in many places that pass narrow budget instances, and pydantic v2 won't coerce a budget instance into a wider nested model. Typing only at the response boundary (where the handler hands FastAPI a dict) sidesteps that. A regression test pins budget_reset_at + created_at through the filter and asserts the internal audit fields stay out. * test(proxy): add golden-master characterization tests for customer responses Lock the exact JSON body each customer-object endpoint (info/list/new/update) and delete return today, so the upcoming type-safety refactor of the handlers is only allowed to land if it reproduces these byte for byte. Pins null-field inclusion, the nested budget shape (server fields kept, audit fields dropped), and object_permission reverse-relation stripping. Green against current code. * refactor(proxy): make the customer response flow type-safe Replace the untyped dict + bolt-on response_model pattern on the customer object endpoints with explicit typed construction. A single mapper, _to_customer_response, validates a DB row into CustomerResponse at one Any -> typed seam; new/update/info/list now return it (or a list of it) and carry real -> CustomerResponse / -> List[CustomerResponse] return annotations, and delete returns DeleteCustomersResponse. basedpyright now verifies the handlers' return shapes instead of a runtime filter doing it silently. This also deletes the four copy-pasted object_permission reverse-relation cleanup loops: pydantic's extra=ignore drops those undeclared fields during validation, so the loops were dead code (proven by the golden-master tests, which stay byte-for-byte green). basedpyright errors on the file drop from 140 to 116, all from removed dict plumbing. CustomerResponse stays a thin subclass of LiteLLM_EndUserTable so it inherits the existing validators/config unchanged (behavior preservation); only the nested budget type is widened. * refactor(proxy): annotate customer response mapper param as BaseModel Address review nit: the mapper's untyped `record` added an ANN001 violation. The incoming rows are pydantic v2 models, so type the param as BaseModel rather than object (object has no model_dump, which would just move the problem to basedpyright). This clears the ANN001 and also drops three basedpyright unknown-type violations the untyped param was adding. * style(test): ruff format customer endpoint tests * test(proxy): give customer budget test update mocks a valid model_dump The type-safe response refactor validates the update result via _to_customer_response (CustomerResponse.model_validate(record.model_dump())). These budget tests mocked the end-user update to return a bare MagicMock, so model_dump() yielded a MagicMock that fails validation. Give each update mock a minimal valid dict; the tests assert on the prisma calls, not the body. * chore(ui): regenerate API types from proxy OpenAPI spec * fix(ui): make generated API types stable across Python versions Python 3.13 strips a docstring's common leading indentation at compile time while 3.12 keeps it, so app.openapi() emits differently-indented description strings depending on the interpreter. The dashboard type generator ran locally on 3.13 and in CI on 3.12, so schema.d.ts drifted and the "Verify schema.d.ts matches the proxy OpenAPI spec" check failed Normalize every description through inspect.cleandoc in the spec dump so the output is identical regardless of interpreter, then regenerate |
||
|
yuneng-jiang
|
494d04c2a2
|
Merge pull request #31471 from BerriAI/litellm_veria_218_personal_key_metadata
fix(proxy): reject team-scoped object_permission on personal keys for non-admins |
||
|
ryan-crabbe-berri
|
e195532c14
|
fix(proxy): count only active users toward license seat limit (#31227)
* fix(proxy): count only active users toward license seat limit SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected * fix(proxy): floor billable user count at zero count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call Addresses Greptile P1 on the PR * refactor(proxy): count teams via TeamRepository in available_users * style: ruff format changed files at line-length 120 |
||
|
Cursor Agent
|
5b029ecd08 | fix: preserve normalized mcp permissions on key regenerate | ||
|
yucheng-berri
|
971a1bedc7
|
fix(proxy): hard-reject CLI session token personal-key budget_limits (#31631)
Mirror the scalar `max_budget` guard in `_common_key_generation_helper` for the per-window check: a CLI session token caller (carrying `max_budget=None`) cannot set `budget_limits` on a personal key. Pass `team_table` into the helper so it can detect the personal-key shape; reject before the `delegation_ceiling is None` early return. Four new regression tests cover the personal-key reject, the team-key happy path, the team-key over-team-budget path, and the proxy-admin exemption. |
||
|
yucheng-berri
|
be6b28f25e
|
fix(proxy): reject non-finite budget_limits windows on /key/generate (#31630)
Enforce that every `budget_limits[*].max_budget` is a finite number; applies to every caller including proxy admin and runs before the role / ceiling checks. Six parametrized regression tests cover NaN / +inf / -inf for both non-admin and admin callers. |
||
|
yucheng-berri
|
7a1ba958f8
|
fix(proxy): gate non-admin /key/generate budget_limits and permissions (VERIA-392) (#31469)
/key/generate validated the caller's delegation ceiling against
data.max_budget only. The per-window entries in data.budget_limits
bypassed the check, so a non-admin caller could mint a key whose
1-day window vastly exceeded their own max_budget. The data.permissions
dict also went unvalidated for non-admin callers, so they could
self-grant capabilities like allow_pii_controls (and on Enterprise,
get_spend_routes).
Both gates now live in _common_key_generation_helper, covering
/key/generate and /key/service-account/generate. The existing empty
{} default on permissions still passes for non-admin callers.
|
||
|
michelligabriele
|
d7654d07ab
|
feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration (#31215)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration * test(proxy): add behavior scenarios for credential migration endpoints * fix(proxy): scan covered tables in encryption check, fix CI lint and route types * fix(proxy): migrate callback_settings credentials, clear CI lint/recursion gates, add encryption endpoint+CLI tests * fix(proxy): correct dry-run/real-run migrated vs residual-legacy counters in config and SSO walkers * fix(proxy): make callback-vars residual detection gate-independent in encryption check |
||
|
yucheng-berriai
|
2a5790fe55 |
fix(proxy): reject team-scoped object_permission on personal keys for non-admins
Non-admin callers could create or update a personal key (no team_id) with arbitrary access_group_ids, mcp_toolsets, vector_stores, or search_tools in object_permission. The server persisted the values without ownership validation; runtime authorization then trusted the IDs because they were stored on the key, allowing cross-tenant access to other teams' restricted models, MCP toolsets, and vector stores. The personal-key gate now mirrors the team-key path. enforce_member_can_assign_access_groups raises 403 for non-admin teamless callers. validate_key_mcp_servers_against_team rejects non-empty mcp_toolsets on personal non-admin keys. A new validate_key_vector_stores_against_team enforces the same rule for vector_stores. validate_key_search_tools_against_team gains the same gate for search_tools. The four validators are wired into /key/generate, /key/update, and /key/regenerate. Proxy admins keep their existing carve-out across all fields; team keys are unaffected. Endpoint-level regression coverage lives in tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py (six new parametrised cases through generate_key_fn and _validate_update_key_data) and helper-level coverage in tests/test_litellm/proxy/management_helpers/. Deleting any of the validator calls in _common_key_generation_helper or unmoving the enforce gate in _validate_update_key_data breaks the suite. |
||
|
ryan-crabbe-berri
|
234263fdda
|
fix(router): persist global retry_policy via /config/update (#29540)
* fix(router): persist global retry_policy via /config/update (LIT-3152)
The Admin UI Model Retry Settings tab POSTs
{router_settings: {retry_policy: {...}}} to /config/update, but the
field was dropped on two write-side layers so it never reached the
router. UpdateRouterConfig did not declare retry_policy, so
dict(exclude_none=True) stripped it before the DB upsert. And even when
fed directly, Router.update_settings had no "retry_policy" entry in
_allowed_settings, so the assignment was a silent no-op. The DB row
stayed at {"model_group_alias": {}}, llm_router.retry_policy stayed
None, and the UI fell back to defaultRetry = num_retries = 2 on refresh.
Declare retry_policy on UpdateRouterConfig as a plain dict, and add a
retry_policy branch to update_settings that coerces dict payloads to
RetryPolicy before setattr, mirroring Router.__init__. get_settings
already lists retry_policy, so reads work once writes land.
* fix(router): guard retry_policy type in update_settings
Mirror Router.__init__ semantics in update_settings: only assign
retry_policy when it is None or a RetryPolicy (after dict coercion).
Previously a non-dict, non-RetryPolicy value (e.g. a YAML typo like
retry_policy: 5 flowing through /config/update) was stored verbatim,
deferring the failure to request time in get_num_retries_from_retry_policy
instead of being dropped at write time.
* refactor(ui): harden Model Retry Settings flow and validate retry_policy at the boundary
Types UpdateRouterConfig.retry_policy as RetryPolicy and model_group_retry_policy as Dict[str, RetryPolicy] so /config/update validates the payload and rejects malformed counts instead of silently persisting them; the apply path in update_settings keeps coercing the stored dict back to RetryPolicy
Makes the Model Retry Settings tab the single owner of retry_policy and model_group_retry_policy so the generic Router Settings page no longer renders or writes them, replaces the fire-and-forget save with a react-query mutation that only shows the success toast after the write resolves, surfaces real errors, disables Save while in flight, and re-reads authoritative state on success, and sends both the global and per-group policies atomically so edits in the inactive scope are no longer dropped
Decouples the retry-scope selector from the All Models filter and defaults it to Global, seeds the displayed default from num_retries (falling back to 2), and gives per-group rows real inherit semantics so an empty input shows the global value as a placeholder with a Reset control, keeping 0 ("no retries") distinct from inheriting the global value
* fix(keys): align router_settings examples with typed RetryPolicy and resync UI artifacts
model_group_retry_policy is now Dict[str, RetryPolicy], so the {"max_retries": 5} sample in the key-generate test and the /key/generate and /key/update docstrings no longer validate; they now use a valid {"gpt-4": {"RateLimitErrorRetries": 5}} shape.
Regenerated eslint-metrics.json (no-explicit-any drifted 2027 -> 2026) and schema.d.ts (new RetryPolicy schema, retry_policy field, model_group_retry_policy value type) so the UI build and api-types-sync checks pass
* test(router): pin retry_policy persistence end to end (LIT-3152)
The existing retry_policy tests exercise UpdateRouterConfig and Router.update_settings in isolation, so they would all still pass if a regression flipped ConfigYAML.router_settings back to a loose dict or stopped add_deployment from applying the stored row. This drives the real handler chain an Admin UI save triggers: update_config writes the LiteLLM_Config row, the apply path forwards it to the live router, and get_config serializes it back, pinning retry_policy across persist, apply, and read-back.
* fix(teams): use valid model_group_retry_policy example in router_settings docstring
Same stale {"max_retries": 5} example the key endpoints carried; model_group_retry_policy maps a model group to a RetryPolicy, so the team /team/new and /team/update docs now show {"gpt-4": {"RateLimitErrorRetries": 5}}. Regenerated schema.d.ts to match.
* fix(ui): load retry settings via deferred fetch to satisfy set-state-in-effect
The Model Retry Settings effect called loadRetrySettings synchronously; eslint-plugin-react-hooks (react-hooks/set-state-in-effect) traces into it and flags the setState calls, failing frontend-lint. Split the loader into fetchRouterSettings + applyRouterSettings and run the fetch in an inline async IIFE with a cancellation flag, so state is applied in the post-await callback rather than on the effect's synchronous path. Behavior is unchanged and onSuccess still refreshes via loadRetrySettings.
* fix(ui): match CI rendering of RateLimitError 429 docstring in generated schema
gen:api run on a dev env (python 3.13 / newer fastapi) rendered the RateLimitError response description with 4-space indentation, but CI regenerates it with 8-space under its frozen python 3.12 toolchain, which is the canonical committed form. The Check UI API Types Sync job regenerates and diffs, so restore that block to the CI rendering; verified byte-identical to the pre-existing committed version.
* fix(ui): pin RateLimitError 429 docstring to CI's frozen schema rendering
Base #29619 regenerated schema.d.ts on a newer FastAPI that renders the RateLimitError response description at 4-space indent, but the Check UI API Types Sync job regenerates under the frozen python 3.12 toolchain, which renders 8-space. Merging base pulled in the 4-space form; restore the 8-space rendering so the generated types match what CI produces (verified byte-identical to the pre-#29619 committed form), which also corrects the base drift once this PR merges.
|
||
|
ryan-crabbe-berri
|
ac56320f26
|
fix(agents): show an agent's attached virtual key in the UI (#29619)
* fix(agents): show an agent's attached virtual key in the UI
The A2A agent detail view never surfaced which virtual key was attached to
an agent, so after assigning a key during agent creation there was no way to
see it again. Surface the attached key(s) in the agent detail view, derived
from the key table's agent_id foreign key the same way spend is already
joined into the agent response.
Backend adds an agent_id filter to /key/list (mirrors team_id) and enriches
GET /v1/agents and GET /v1/agents/{id} with a non-secret key summary (alias,
masked key_name, hashed token id). The frontend renders a Virtual Keys
section in the agent detail view that lists the agent's keys and links
through to the key detail, and the list view drops its fetch-500-keys-and-
filter-client-side workaround in favor of the enriched response. The orphaned
AgentCard and AgentCardGrid components, left behind when the agent list
switched from a card grid to a table, are removed
* fix(agents): redact attached virtual keys for non-admins
_attach_keys_to_agents joins keys onto the agent response by agent_id with
no caller scoping, but _redact_sensitive_agent_fields never cleared the new
keys field. A non-admin able to view an agent therefore received the alias,
masked name, and hashed token of every key attached to it, including keys
owned by other users or teams; the old client-side path used the scoped
key list, so this was a visibility regression. Clear keys in the redaction
path so only admins see attached-key metadata.
Adds an endpoint-level regression test asserting keys is populated for admins
and null for non-admins, and a list-view test covering the Active vs Needs
Setup badge that lost coverage when the agent card tests were removed.
* fix(agents): satisfy strict lint and resync key/list types
- use builtin list/dict generics in the new agent key helpers to stay
under the UP006 strict-rule ceiling
- swap @tremor/react for antd Typography in agent_virtual_keys (tremor is
being phased out; the new component was the only unsuppressed import)
- regenerate schema.d.ts so the /key/list agent_id query param is typed
* style(agents): prettier-format key hook test and agent_info
|
||
|
Yassin Kortam
|
b2e708d5ae
|
feat(prometheus): add per-team litellm_team_members_metric gauge (#31506)
Emit litellm_team_members_metric on every team member add and delete, labelled by team and team_alias and set to the team's authoritative member count. Because it is set from the current membership rather than incremented or decremented, it tracks the count up and down, never goes negative, and self-corrects on the next change after a proxy restart. Bulk member add is covered for free since it delegates to team_member_add, and the helper no-ops when the Prometheus callback is not registered. Resolves LIT-3082 |
||
|
Yassin Kortam
|
f55d13ebba
|
fix(team): persist budget_duration on /team/member_add member budgets (#31443)
/team/member_add could not set budget_duration on an individual member budget. add_new_member created the budget row with only max_budget and allowed_models, and TeamMemberAddRequest had no budget_duration field, so a member added with an explicit per-member budget while the team ran a recurring member budget got a lifetime cap instead of a recurring allowance. Thread budget_duration from TeamMemberAddRequest through _process_team_members into add_new_member, and pull the member-budget resolution into a helper that writes budget_duration plus a computed budget_reset_at. When only a budget_duration is supplied and the team has a default member budget, the default is cloned and its reset window overridden so the member keeps the default's max_budget rather than becoming uncapped; a duration with no team default creates a window-only budget. Invalid durations are rejected with a 400 before any DB write, symmetric with /team/member_update. The available-team self-join bypass only grants the ability to join, so reject per-member budget and model controls (max_budget_in_team, budget_duration, allowed_models) for non-admin self-join callers in _validate_team_member_add_permissions, before any DB write. Otherwise a self-joining non-admin could set their own cap, reset window, or model scope past the team default; admins, team admins, and org admins are unaffected and a clean self-join still inherits the team default budget. Resolves LIT-4052 |
||
|
Sameer Kankute
|
687a62e561
|
fix(cli): mint per-session agent credential on lite login (#31072)
* fix(cli): mint per-session agent credential on lite login
The `lite login` command was producing a shared UI session token that broke agent use in three ways: a $0.25 budget cap (from max_ui_session_budget) that killed agent sessions in minutes, a fixed identity "cli-jwt-token" shared across every user preventing per-session spend attribution, and auth gated behind EXPERIMENTAL_UI_LOGIN so the token was rejected on default deployments.
This fixes all three. Each login now generates a unique cli-session-{uuid} token with no per-key budget cap (enforced via shared team/user counters instead), and the decrypt path activates for any non-sk- token without requiring EXPERIMENTAL_UI_LOGIN.
* fix(cli): address review feedback on EXPERIMENTAL_UI_LOGIN gate and e2e test
Restore EXPERIMENTAL_UI_LOGIN=false as an explicit opt-out: operators who set it to false keep the old boundary; unset (new default) and true both attempt NaCl decryption, which fails closed for non-blob tokens.
In the e2e test: replace the silent Redis fallback with pytest.skip so a missing Redis instance is explicit rather than silently degrading to a directly-minted token. Write the seeded flow back as JSON (proxy reads it via json.loads on cache fetch) instead of Python repr, and build the updated flow immutably.
* fix(key-management): cap CLI session token delegation budget to team ceiling
A CLI session token intentionally carries max_budget=None to avoid a per-session LLM spend cap. The key-generation delegation check (GHSA-q775-qw9r-2r4g) previously skipped non-admin callers with max_budget=None, treating them as having unlimited delegation authority. This allowed any internal user with a lite login session to mint virtual keys with arbitrary budgets.
Adds is_session_token=True to UserAPIKeyAuth for CLI session tokens and uses the caller's team budget as the delegation ceiling in that case, so the effective limit is min(requested_budget, team.max_budget) rather than unbounded.
* chore: regenerate dashboard OpenAPI types
The is_session_token field added to UserAPIKeyAuth cascades to the
dashboard schema. Regenerate types from the updated OpenAPI spec.
* fix(key-management): block personal key budget delegation from CLI session tokens
When team_table is None (personal key, no team_id in request), the personal key
has no team-budget enforcement at request time. A session token therefore cannot
delegate any explicit max_budget for a personal key -- that would open a budget
bypass path. Block the request with a clear 400 directing the caller to use a
team_id instead.
* test(auth): add unit coverage for non-admin CLI session token production path
* fix(type-check): use model_validate in _return_user_api_key_auth_obj to fix reportArgumentType gate
UserAPIKeyAuth(**user_api_key_kwargs) spread triggers a basedpyright
reportArgumentType error for each named field in UserAPIKeyAuth because
the dict's inferred value type (str | Span | LitellmUserRoles | Unknown)
is not assignable to each field's specific type. Adding is_session_token:
bool introduced +2 more such errors, breaching the gate cap.
model_validate accepts an untyped dict without per-field argument checking,
which eliminates the +2 new errors and also ratchets down the pre-existing
333 errors at those call sites. basedpyright-code-budget.json is updated
to reflect the new lower baseline (1814, down from 1934).
* fix(type-check): ratchet down reportArgumentType baseline only
The previous lint-budget-update captured all baselines from the local
environment, raising many ceilings vs the merge-base and failing the
non-gating budget_ratchet_check. Restore staging's values for every
rule and only lower reportArgumentType (1934 -> 1814) to reflect the
reduction from switching to model_validate in _return_user_api_key_auth_obj.
* fix(auth): set max_budget on CLI session token to enforce max_ui_session_budget
CLI session tokens were missing max_budget, so _virtual_key_max_budget_check
had no per-session ceiling to enforce. Operators relying on max_ui_session_budget
could be bypassed for the full token lifetime. Mirrors the existing UI token path.
* revert(auth): remove max_ui_session_budget from CLI session token
max_ui_session_budget defaults to $0.25 and is sized for the UI chat
pane (10-min sessions). CLI sessions are 24-hour tokens for real work;
capping them at that ceiling would throttle users under their actual
user/team budget. Budget enforcement for CLI sessions is via the shared
user and team counters as originally intended.
* fix(auth): cap CLI session at max_ui_session_budget only when user and team have no budget
When neither the user nor their team has a budget configured, CLI sessions
were fully uncapped. The poll endpoint now looks up the real user and team
objects from DB; if both have no max_budget, it passes litellm.max_ui_session_budget
as the token's per-key ceiling. Users or teams that already have a budget
configured are unaffected and continue to rely on the shared counters.
* fix(auth): fix black formatting and update test mock for cli_poll_key budget lookup
The get_user_object and get_team_object async calls in cli_poll_key were
not mocked in the existing test, causing MagicMock await errors. Patch
both functions at the auth_checks module level. Also apply black formatting
to ui_sso.py which CI rejected.
* fix(auth): skip fallback budget cap when team lookup fails for cli session token
* test(auth): pin cli session budget cap to user/team budget presence
The session_max_budget fallback in cli_poll_key only applied
max_ui_session_budget when neither the user nor the resolved team had a
budget. The existing coverage exercised only the team-lookup-failure
branch. Add two regression tests: a user with a configured budget must
not receive the fallback cap, and a session with no user and no team
budget must fall back to max_ui_session_budget. Mutating either guard
out of the branch now fails these tests.
* fix: remove CLI poll session budget cap
* revert(auth): restore CLI session fallback budget cap
Bugbot autofix (
|
||
|
Sameer Kankute
|
7eacdd5258
|
chore: litellm oss staging 250626 (#31305)
* fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes #30926) * style: format common_utils.py with black * fix(anthropic): extract api_base from litellm_params in batches/files validate_environment * fix(anthropic): scope Bearer key check to custom api_base endpoints * fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives The Anthropic streaming protocol emits `message_start.usage.output_tokens=1` as a placeholder cursor; the real cumulative output count only arrives in the final `message_delta` event. When a stream is cancelled before `message_delta` lands (common for thinking models on long-tail prompts), ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left completion_tokens stuck at 1. Because 1 is truthy, the `completion_tokens or token_counter(text=...)` fallback in calculate_usage() never fired, and requests were billed for 1 output token even when several thousand tokens of text had actually streamed. Fix: track whether any chunk's completion_tokens exceeded 1 (saw_non_cursor_completion). If the only update we saw was the cursor, reset completion_tokens to 0 so the text-based fallback estimates from the real completion content. Legitimate 1-token completions (model returns "Yes." etc.) are unaffected in practice — token_counter on a 1-token completion_output also yields ~1, so billing stays approximately correct. Tests: - TestAnthropicCursorBug (6 cases) — pins the post-fix behavior - TestNonAnthropicStreamingIntact (2 cases) — guards against regression on providers without the cursor pattern All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests still pass. * fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival Addresses both Greptile P2 threads on PR #30420: CLASS A — Anthropic-specific heuristic was applied globally ============================================================ The `completion_tokens == 1 and not saw_non_cursor_completion` reset lived in provider-neutral `streaming_chunk_builder_utils.py`. Any non-Anthropic provider that legitimately reports completion_tokens=1 in a single usage chunk (perfectly normal for short OpenAI / Bedrock / Vertex single-token replies with stream_options.include_usage=true) would have its value silently rewritten to 0 and re-billed via token_counter — producing a different number than what the provider actually charged. Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved from the first chunk's `_hidden_params` (the same field set by streaming_handler.py:722 on the live path). Unknown / missing provider is treated as non-Anthropic and skips the reset, so newer providers and custom plugins are also safe by default. CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies ============================================================ Previous condition was `usage_chunk_dict["completion_tokens"] > 1`, which never fires for an Anthropic stream where the model legitimately emits exactly one output token (e.g., "Yes."). Anthropic still sends message_start (output_tokens=1, the cursor) AND message_delta (output_tokens=1, the real value) — same value, but two distinct usage events. The old check couldn't tell that apart from a cancelled stream where only message_start landed. Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion` when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR (2) we've seen >=2 completion-bearing usage events (positive evidence that message_delta arrived). Cancelled cursor-only streams still have exactly one event and still hit the reset; cache chunks with completion_tokens=0 don't count toward the threshold. Tests ============================================================ - _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default "anthropic") so the gate is exercised by every existing test — none of them needed assertion changes besides the legitimate-single- token case, which now expects exactly 1 (was a fuzzy 0..3 range). - New: test_anthropic_cache_only_chunks_after_message_start_still_resets - New: test_non_anthropic_provider_completion_tokens_one_not_reset - New: test_unknown_provider_completion_tokens_one_not_reset 11/11 tests pass. * chore: add Co-authored-by trailer for attribution Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> * fix(anthropic): preserve messages cache usage * style(anthropic): format messages cache usage helper * fix(anthropic): accept integral float cache token counts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(anthropic): accept integral float cache token counts * test(anthropic): cover cache usage edge cases * fix(gemini): preserve thoughtSignature for server-side tool responses When Gemini API returns toolCall and toolResponse parts, they might have different thoughtSignatures. Previously, LiteLLM merged them into a single dict, overwriting the response's thoughtSignature with the call's. This fix extracts them separately and re-injects them correctly. TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * fix(gemini): address PR comments on thoughtSignature handling - Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature - Add missing assertions in existing tests - Add new unit tests for orphan-response signature handling TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * feat(mcp): include server alias and server_id in mcp_info response - Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint - Update rest_endpoints.py to surface alias from server config - Add test coverage in test_mcp_server.py and test_rest_endpoints.py Fixes #31015 * fix(proxy): reject non-finite spend via validate_finite_spend A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a shared finite-value guard, defined above the litellm.proxy.* imports to avoid the module-level cyclic-import warning. * fix(proxy): require admin for any /key/update spend, reject non-finite Gate the admin check on the presence of `spend` (not a value diff): the DB spend lags the live cross-pod counter, so an "unchanged" spend on the non-admin path let a key owner / team member overwrite the live counter below real usage. Also reject NaN/+-inf spend before the DB write. * fix(proxy): invalidate spend counter on /user/update spend change A direct spend change on /user/update wrote the DB row but left the warm cross-pod counter at the stale value, so enforcement kept reading the old spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB), and reject non-finite spend before the write. * fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244) The semantic cache's embedding model is a proxy Router alias whose AWS credentials (aws_role_name, aws_session_name) live only in the Router deployment's litellm_params. The sync embedding paths called litellm.embedding() directly, bypassing the Router, so they could neither resolve the alias nor assume the configured role; cross-account Bedrock semantic caching failed with "bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding during cache construction, while llm_router is still None. Fix A: make the sync paths mirror the already-correct async paths. A shared, dependency-injected helper (litellm/caching/_embedding_router.py) decides whether to route through llm_router.embedding(...) when the model is a Router deployment, else fall back to direct litellm.embedding(...). Redis and qdrant sync set_cache/get_cache now precompute the embedding and pass vector= to the backend, exactly as the async astore/acheck already do. Both async _get_async_embedding methods are unified onto the same helper and now forward the caller's full metadata instead of a hand-picked subset. Fix B (Redis only): defer redisvl index construction from __init__ into a lazy, memoized llmcache property, so the dimension-probe embedding fires on first cache use, after llm_router is wired. A failed build is not memoized, so a transient outage recovers on the next request. Known limitation: resolve_embedding_router gates on an exact model-name match (same as the shipped async path); wildcard/alias/team-public routes still fall back to direct embedding. Tracked as a follow-up. * fix(cache): harden embedding-router and shrink Any surface (review) Address review feedback on the semantic-cache aws-role fix (#28244): - resolve_embedding_router now skips deployment entries missing model_name instead of raising KeyError on a malformed model_list (Greptile P2); add a regression test that fails on the old direct-key access. - Replace the `**kwargs: Any` passthrough on the four cache _get_embedding / _get_async_embedding helpers with an explicit, typed `metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only ever consumed kwargs["metadata"], so this is behavior-preserving, makes the forwarded field obvious at the call site, and removes three bare-Any annotations (keeps the strict-rule ANN401 budget within ceiling). - Note in _build_llmcache that redisvl's dimension-probe embedding adds one extra billable embedding on the first cache request (Greptile P2). * fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist" Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash * test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression Adds the regression coverage codecov flagged on the two responses to completion bridge guard lines and the bedrock route-prefix helper. The handler tests drive both the sync and async fallback paths with litellm.completion and litellm.acompletion mocked, and assert the forwarded kwargs carry _skip_responses_api_bridge=True, so dropping either flag line fails the suite. The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids still do, exercising both branches of _model_has_route_prefix. Also aligns update_messages_with_model_file_ids model_id to Optional[str], matching its Responses API sibling, so the defensive model_info fallback no longer introduces a new reportArgumentType in completion(); the file-id lookup narrows model_id before the dict get * chore(ui): sync generated OpenAPI types for optional test_connection mode The test_model_connection mode body param default changed from chat to None so the mode is auto-detected from model capabilities, which makes the field optional in the proxy OpenAPI spec. Regenerate the committed schema so the dashboard types match: mode becomes optional and the description and default JSDoc follow the spec, keeping the Check UI API Types Sync gate green * refactor(bedrock): match all explicit route prefixes at path-segment boundary Migrates the remaining substring route checks to the existing _model_has_route_prefix helper so every explicit route token matches only as a leading path segment, consistent with get_bedrock_route and the mantle route. Covers _explicit_converse_route, _explicit_claude_platform_route, _explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route, _explicit_converse_like_route, _explicit_async_invoke_route and _explicit_openai_route. This also stops invoke/ from substring-matching async_invoke/. Route precedence and order are unchanged, and a note on the segment invariant is added to the helper docstring * test(bedrock): cover explicit route prefix segment matching Exercises all eight migrated _explicit_*_route helpers (converse, converse_like, invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each matches its token as a leading path segment and rejects the token glued to a preceding segment, so reverting any method to the old substring check fails the suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete improvement of the segment-boundary migration * test(proxy): assert negative spend is allowed (one-time grant use-case) Negative spend is intentionally permitted so admins can grant extra allowance for the current budget period only, without raising the recurring budget ceiling. Cover it explicitly in validate_finite_spend and via the /user/update invalidation test. * fix(google_genai): forward native generateContent top-level fields Google's native generateContent REST body carries safetySettings, toolConfig, cachedContent and labels at the top level as siblings of generationConfig. The proxy's :generateContent endpoint spread them into agenerate_content as loose kwargs and then dropped them, so callers had to wrap them in extra_body for them to take effect; safetySettings, for instance, was silently ignored The provider config now exposes the native top-level field names and setup_generate_content_call collects whichever are present, merging them into the outgoing request body through the existing extra_body merge so they reach Google verbatim. An explicit extra_body still wins on conflict. The sync generate_content_stream path now also forwards systemInstruction, matching the other three entry points Fixes #12671 Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK * fix(proxy): resolve env refs for DB-stored models * fix(proxy): restrict DB env ref resolution * fix(proxy): block team DB env ref resolution * fix(lint): resolve ANN401/UP045/C901 strict-gate violations - Replace Optional[X] with X | None (UP045) in 8 files - Replace Any return/param types with concrete types or object (ANN401) - Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix Users who pass a key already prefixed with "Bearer " get Authorization: Bearer. All other keys continue to use x-api-key, preserving backward compatibility with custom api_base endpoints that expect x-api-key rather than Authorization. Also consolidates get_auth_header to reuse _make_api_key_auth_header helper, eliminating the duplicated custom-endpoint routing logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base The backwards-compat change broke existing tests that verify the intentional Bearer-for-custom-base behavior (Fixes #30926). Restore original logic while keeping the _make_api_key_auth_header helper for code deduplication. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag Previously the auth-header switch from x-api-key to Authorization: Bearer applied unconditionally for non-sk-ant- keys on a custom api_base, silently breaking existing deployments that proxied to gateways expecting x-api-key. Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header, get_anthropic_headers, and get_auth_header. validate_environment reads it from litellm_params so callers can opt in per-model without any API surface change. Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981) DEL was the only Redis cache operation that skipped check_and_fix_namespace, so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM error on deployments with an ACL restricting DEL to the litellm:* pattern, and a silent no-op on all other deployments since the un-prefixed key was never stored. * style(anthropic): reformat common_utils.py with Black (--target-version py312) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve cache metadata and spend counters * style: apply ruff format to streaming_iterator.py * refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate Extract Anthropic message_start cursor reset into _reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter invalidation into _invalidate_user_spend_counter_if_changed, keeping both _calculate_usage_per_chunk and _update_single_user_helper under the max-complexity ceiling. Use builtin generics in the new signatures so no new UP006 violations are introduced. Behavior unchanged. --------- Co-authored-by: rupak-eng <rupakji99@gmail.com> Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com> Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com> Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com> Co-authored-by: Andrii Butko <booandrew23@gmail.com> Co-authored-by: Kent <kingdooo@gmail.com> Co-authored-by: kunal2002 <k.nayyar2002@gmail.com> Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com> Co-authored-by: jesco-absolut <team@srswti.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Matt Hill <mhill@dataminr.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
tin-berri
|
360adbe765
|
fix(mcp): resolve config-defined servers in per-user credential and env-var endpoints (#31171)
The per-user BYOK, OAuth (OBO), and env-var management endpoints resolved the target MCP server through a DB-only lookup (get_mcp_server / get_all_mcp_servers_for_user). A server defined in config.yaml lives only in the in-memory registry and never gets a row in LiteLLM_MCPServerTable, so those endpoints raised 404 "MCP Server <id> not found" (or 403 for non-admins) before any credential could be stored, leaving config-server users unable to connect and forced to re-authorize forever. Route all three through a single registry-aware resolver: DB first, then the in-memory registry (built into LiteLLM_MCPServerTable via _build_mcp_server_table, the same fallback fetch_mcp_server already uses), then the canonical get_allowed_mcp_servers authorization the MCP gateway enforces on tool calls. Admins get a 404 for an unknown id; non-admins get 403 for a missing-or-forbidden server so server ids stay non-enumerable. This also closes a gap where the two store endpoints performed no per-server authorization at all. |
||
|
yucheng-berri
|
c2e06890ad
|
fix: tighten role-based visibility of config and MCP fields (#30587)
* fix: redact config and MCP secrets in read-only admin views GET /config/field/info and the MCP server list/detail endpoints returned secret-bearing fields to any caller with an admin view, including read-only admins. They now return those fields in full only to a full PROXY_ADMIN; every other caller gets the reduced, non-admin view, while non-sensitive fields remain readable. Regression tests cover the role-based visibility on both endpoints, including that a full admin still sees everything needed to populate the edit form. * fix: redact nested secrets in config field info for non-admins /config/field/info returned structured general_settings fields verbatim to any admin-view caller, so a view-only admin reading database_args received the nested aws_web_identity_token (a DynamoDB role-assumption credential) in plaintext. Recurse into dict/list field values and redact secret leaves for non-PROXY_ADMIN callers, leaving non-secret siblings and full-admin reads unchanged * fix: redact secret config values in /config/list for non-admins /config/list shared the same _user_has_admin_view gate as /config/field/info but returned each field value unredacted, so a view-only admin reading the list received pass_through_endpoints upstream Authorization headers verbatim. Route every general_settings value through a shared role-aware redactor (extracted from /config/field/info) covering the top-level and nested field paths, so non-PROXY_ADMIN callers get secret-bearing fields redacted while full-admin reads stay unchanged * chore(ci): allowlist _redact_secret_values_in_obj in recursive_detector The config secret redactor recurses over JsonValue, which is acyclic, and its depth is bounded by the operator-authored general_settings schema. Add it to the recursive_detector ignore list alongside the other bounded nested-redaction helpers (mask_dict, _redact_sensitive_litellm_params) * proxy: cap recursive secret redaction depth at 10 Match the cap on _redact_sensitive_litellm_params (the closest analog in the proxy, also recursive, key-name driven, returns a sentinel). The previous justification — bounded by operator-authored schema depth, JsonValue acyclic — is true today but is a property of the threat model, not an enforced invariant of the function. If a code path is ever added that pipes external input into general_settings (config import, migration tooling, JWT-driven settings, …) the assumption silently breaks. A local cap makes the invariant local. The cap branch fails closed: at _REDACT_SECRET_MAX_DEPTH the whole subtree is replaced with 'REDACTED' rather than returned verbatim. A future refactor that flips this to fail-open would let a deeply nested credential leak; the new regression test test_redact_secret_values_in_obj_fails_closed_at_max_depth guards against that. Updates the recursive_detector ignore-list rationale to point at the numeric cap rather than the structural argument. * test: actually exercise the depth cap in fails-closed test The previous fixture stored the leaf under the secret-named key 'aws_web_identity_token', which the recursor's key-name short-circuit redacts regardless of the cap — so the test passed both with and without the cap in place. Empirically confirmed: under an uncapped mutant the old fixture still hides the secret (key-name catches it), the new fixture leaks it (only the cap can stop it). Swap the leaf key to a non-secret name so the cap is the only redaction path exercised, making the test fail on mutation as advertised. |
||
|
Yassin Kortam
|
e70f7e2d7a
|
fix(ui): resolve user_id to email in Spend Per User usage chart (#30992)
The Usage dashboard "Spend Per User" chart rendered raw UUIDs (and default_user_id) instead of emails. /user/daily/activity passed entity_metadata_field=None, so every user entity in the breakdown carried empty metadata; the chart could only fall back to the user_id. The frontend resolved labels from a separately paginated user list, so any spender not on a loaded page showed as a UUID. Resolve the email/alias for the user_ids actually on the page (mirroring how api key metadata is already resolved) and attach it to the entity metadata, so the chart labels each spender with their email and falls back to the UUID only when no email is on file. get_daily_activity gains an optional resolve_entity_metadata hook so the user endpoint can do this page-scoped lookup without loading the whole user table. Resolves LIT-3889 |