* fix(auth): resolve bare model names against wildcard deployments in model access groups
* test(e2e): cover model access group permission checks on keys and teams
An internal user who administers an organization saw an empty
Organization Usage dashboard and had to be promoted to proxy admin to
see any of it.
Two independent gates were closed on them. The route layer rejected
GET /organization/daily/activity with 401 before the handler ran, since
the route belonged to no list a non-proxy-admin can reach, and the
handler's own org-admin scoping was therefore dead code. In the
dashboard, viewOrganizationUsage was granted by session role alone, and
an org admin's session role is internal_user, so the Organization Usage
option never rendered and its data fetch stayed disabled.
The route now sits in self_managed_routes, where the handler restricts
results to organizations the caller is ORG_ADMIN of and 403s on any
other org, and viewOrganizationUsage joins the existing per-capability
org-admin allowance that already covers viewDeletedTeams.
A caller who administers no organization resolves to an empty id list
rather than to None, so the organization-alias lookup is scoped by that
same list instead of reading the whole table.
The Usage page falls back to the global view when org-admin membership
is revoked while it is open, so the selector never keeps a value it no
longer offers.
* fix(proxy): cache tag-name registry so unregistered request tags skip Postgres
Request tags are free-form attribution labels, so most have no LiteLLM_TagTable
row. get_tag_objects_batch never cached that absence: every tagged request ran
a find_many that came back empty, and under Prisma pool contention those
per-request queries queued for minutes inside user_api_key_auth.
Cache the bounded set of registered tag names under one aggregate key with the
management-object TTL. Uncached request tags are filtered against it before any
per-tag DB fetch, so unregistered tags cost zero DB reads on a warm path. An
empty registry is cached as a valid answer; DB errors are not cached and fall
back to the per-tag lookup; tables past TAG_REGISTRY_MAX_SIZE cache an overflow
sentinel that disables filtering. Tag create/update/delete endpoints now evict
the registry and per-tag keys and publish cross-worker invalidation (they
previously evicted nothing). The per-tag write-back also gains the management
TTL it was missing, and the hand-built tag:{name} key strings are replaced with
a shared builder.
* fix(proxy): skip per-request end-user DB reads via restricted-id registry
Every request carrying a user id ran get_end_user_object, and with high-cardinality
auto-created end-user rows (hundreds of thousands of ids, all restriction fields
NULL) the per-pod cache missed on nearly every request, so each one paid a Postgres
find_unique that queued behind the Prisma pool during background-job bursts. True
misses were never cached, and unknown ids paid the read twice per request.
Cache the bounded set of end-user ids that carry any restriction (blocked, budget,
region, default model, or object permission) under one aggregate key with the
management-object TTL. When an id misses the per-id cache and is absent from a
usable registry, get_end_user_object returns None with zero DB reads; restricted
ids keep today's fetch-and-cache path. The skip is bypassed whenever
litellm.max_end_user_budget_id is set (default budgets make unrestricted rows
behaviorally distinct from missing rows), validate_end_user_id_in_db is on
(existence checks need the row), or the token carries end_user_max_budget from
custom auth (the row's recorded spend seeds the budget counter). Empty registries
cache as a valid answer, DB errors are never cached, and oversized tables cache an
overflow sentinel that disables filtering. Customer create/update/block/delete now
evict the registry and per-id keys and publish cross-worker invalidation (they
previously evicted nothing), and the per-id write-back gains the management TTL it
was missing so Redis entries no longer live forever.
* refactor(proxy): single generic registry loader with error sentinel and single-flight
Code review follow-ups on the two registry caches. Registry DB errors now cache
the overflow sentinel for a short REGISTRY_ERROR_NEGATIVE_CACHE_TTL window and
log at warning, so a degraded Postgres stops paying the failing registry scan on
every request on top of the per-id fallback. Cold registry loads are single-flight
per worker behind per-registry locks with a recheck after acquire, so a TTL expiry
no longer fans out one full-table scan per in-flight request. The tag and end-user
loaders collapse into one _load_bounded_registry with per-entity fetch closures,
and the triplicated evict-then-broadcast protocol becomes one evict_and_broadcast
helper beside publish_auth_cache_invalidation, shared by the tag, customer, and
project eviction paths.
* chore(lint): suppress fail-safe registry excepts and ratchet BLE001 budget
* docs(proxy): trim registry cache commentary to single-line why docstrings
* fix(lint): move tag fetch return to else block to satisfy TRY300 budget
This reverts commit ab2333b6c4.
Every Admin UI login mints its session key against the sentinel team_id
`litellm-dashboard`, and no LiteLLM_TeamTable row is ever created for it.
That lookup is therefore a provably-absent row on every UI request, which
#36837 turned into a hard refusal with no override, so the whole dashboard
404s.
Reverting restores the token-derived fallback. The model-access widening
#36837 closed is reopened and needs a re-land that exempts the UI sentinel
team.
When get_team_object fails, the centralized auth gate rebuilds the team
from the token's own fields. A token whose team row was missing when the
key was read carries team_models=[] and team_blocked=False, and the
model-access check reads an empty model list as every model, so the
rebuilt team grants more than the real team ever did.
get_team_object reported a deleted team and a database that would not
answer as the same 404, so the fallback could not tell a definitive
answer from a degraded read. Raise a TeamNotFoundError subclass, still a
404 with the same detail so every other caller is unaffected, only when
the database answers and the row is absent.
A team that is provably gone now refuses, and no setting overrides that.
Otherwise the grant is merely unknown: a token carrying one may vouch,
since replaying a recorded grant cannot widen it, and a token carrying
none may not. allow_requests_on_db_unavailable still opts back out there,
and is only consulted once the failure is known to be a degraded read.
CLI session tokens minted by /sso/cli/poll set team_id and team_alias but
never team_models or team_model_aliases, so the token carried a team with
none of that team's grants. /v1/models bails out to "unrestricted" when both
key_models and team_models are empty and listed the whole proxy, and team
model aliases never resolved because both can_team_access_model and the
pre-call rewrite read team_model_aliases off the token.
The team data was not close at hand: _fetch_cli_sso_team_details projected
full team rows down to team_id and team_alias before they reached the mint.
Widen that projection to include the team's models and its joined alias
table, and populate both fields at mint time.
Also stop writing the user's personal allowlist into the key models slot
when a team is bound, matching virtual-key semantics where a team-bound
credential is governed by the team grant.
Because an empty team grant is itself a real value meaning unrestricted, a
team whose grants cannot be resolved must not be minted as empty: that is
the same "unrestricted" bail-out this fix exists to close. The poll now
refuses to mint when the selected team has no complete cached detail.
That refusal is only safe because a login can no longer be pinned to a team
whose grants will never resolve. Deleting an organization drops its team
rows but leaves the memberships behind, so the login now offers only teams
whose rows still exist, and a lookup that fails outright fails the login
rather than caching a session that silently drops every team.
* fix: never price a strategy-router alias
A strategy-router alias (auto_router/complexity_router/<name>) is never the
deployment that gets called or billed, but custom pricing configured on it was
being treated as real pricing in two places:
- registered in litellm.model_cost under the alias deployment id, so an
explicit zero made _is_cost_explicitly_configured() report the group as a
genuinely free model and every budget check was skipped, while the request
routed to a paid deployment and accrued real spend
- copied onto request_kwargs by the alias-params merge, so the routed
deployment got re-registered at the alias price and the request billed 0.0
Both are fixed at the writer, so config, /model/new and price-map reload all
take the same path
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: annotate filtered cost-map copy for the mutable-collection gate
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(proxy): expand config-defined model access groups when resolving team models for /v2/model/info
Teams whose only model grant is a config-defined access group (a model_info.access_groups
name listed in team.models) got an empty /v2/model/info?include_team_models=true result.
_add_team_models_to_all_models passed each team.models entry straight to
llm_router.get_model_list(model_name=...), which never matches an access-group name, so
the group's member deployments were dropped. Runtime auth and /v1/models were unaffected
because they expand team.models through get_team_models first.
Resolve team.models through the same get_team_models resolver before iterating, reusing the
exact path runtime auth and /v1/models trust so the two can't drift again. The get_model_names
and get_model_access_groups accessors are hoisted above the team loop so they run once.
* fix(proxy): keep a literal model whose name collides with an access-group name in listings
A grant string that names both a deployed model and a config access group grants
BOTH at runtime (_check_model_access_helper unions them), but the listing resolver
dropped the literal and substituted the group members, hiding a callable model from
/v1/models and /v2/model/info. Keep the literal when it is also a deployed model so
listings match runtime access exactly. Pure-group names (no collision) are still
replaced by their members. Also rewrites _get_models_from_access_groups to build
its result without mutating the input list.
Addresses the Greptile P1 on this PR.
* fix(proxy): type proxy_model_list param as Sequence to satisfy LIT001 budget
A request whose body never parses is rejected in auth, before the endpoint
runs, so nothing downstream fires the failure hook that writes the spend log
row Request Logs reads. The caller sees a 400 that leaves no trace.
Auth now records that rejection through the same post_call_failure_hook the
endpoints use, keyed to the caller it already authenticated. Logging is
best-effort: a logging failure is swallowed so the 400 the caller sees is
unchanged. The path where the key is also rejected is left alone, since the
auth failure handler already logs that request.
* fix(proxy): treat SAML as configured in UI SSO detection
_has_user_setup_sso only checked OAuth client IDs, so SAML-only setups
left /.well-known/litellm-ui-config sso_configured=false and the login
button gray even when SAML IdP metadata was set. Include
SAML_IDP_METADATA_URL / SAML_IDP_METADATA_XML so UI discovery matches
the login redirect path.
* chore: adhere to comment policy
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(otel): mark v2 server spans as failed for pre-call errors (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): authenticate malformed-body requests before rejecting them (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): cover malformed-body rejection when auth error is recovered
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): skip authorization for a request whose body never parsed
Deferring the parse failure ran the full auth phase, including budget reservation, whose reserved amount is only released by the endpoint's post call path; the endpoint never runs, so malformed requests leaked reservations and locked a budgeted key out. Authorization now runs only when the body parsed, and a parse failure with a rejected key keeps returning the 400 it returned before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve entity access groups in the model listing endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): reuse the fetched team object when listing models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover key-level access group resolution in model listing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add apply_user_budget_to_team_keys opt-in
PR #32005 made a user's personal max_budget apply to their team-scoped keys
too, and PR #35271 reverted the whole thing (behavior plus the
skip_user_budget_on_team_key opt-out) because that flipped the default for
everyone. This brings the behavior back the other way round: default is
unchanged, and general_settings.apply_user_budget_to_team_keys opts a
deployment into charging the key owner's personal budget on team keys.
The flag reaches all three personal-budget gates so an opted-in deployment
enforces consistently: the read-time check in common_checks, the optimistic
reservation counter in _get_budget_counters, and the _PROXY_MaxBudgetLimiter
pre-call hook. It is also in the /config/list allowed args and, unlike the
reverted flag, in the _update_general_settings propagation allowlist, so the
Admin UI General Settings toggle actually takes effect at runtime; an explicit
YAML value still wins over the DB value on reload.
get_config_list's allowed_args moves to a module-level frozen mapping of
field name to type string, dropping 18 LIT002 violations and rebuilding one
less dict per request.
* style(proxy): drop explanatory comments from the budget flag paths
* fix(proxy): invalidate cached project object on /project/update and /project/delete
The auth path reads projects cache-first via get_project_object with a 60s
TTL and no freshness check, but no project write endpoint ever evicted the
project_id:{id} cache entry. A project cached before /project/update added a
model allowlist kept an empty models list in cache, so _run_project_checks
skipped can_project_access_model and project-bound keys could call team
models outside the project allowlist until the TTL expired. The same
staleness applied to blocked status and budget fields, and /project/delete
left the deleted project enforceable from cache.
Evict the cache entry after the DB write in update_project and
delete_project via a shared delete_cached_project_object helper, with the
cache key derivation shared with get_project_object.
* fix(proxy): broadcast project cache invalidation to all workers and make eviction best-effort
Single-worker eviction leaves every other worker serving its in-memory copy
of the mutated project until the 60s TTL expires, so a project allowlist
change was still bypassable on multi-worker deployments. Add a coordination
Redis pub/sub channel (litellm_proxy.auth_cache_invalidation): project
eviction publishes the cache key and a per-worker subscriber deletes the
local in-memory entry, with the next auth read refetching from the DB.
Subscriber starts on any deployment with a coordination Redis and falls back
to the TTL when none is configured.
Also wrap the eviction in a best-effort catch: the DB write has already
committed when eviction runs, so a cache backend error must not turn a
successful update into a 500 or abort the remaining ids in /project/delete.
* fix(lint): sort auth cache invalidation import and suppress best-effort shutdown catch
The strict-budget gate flagged the new import block as un-sorted (I001) and
the broad except in stop_auth_cache_invalidation_subscriber (BLE001); the
catch is intentional since a failing stop must not break proxy shutdown, so
it carries a named suppression instead of counting against the budget.
The aggregated route was missing from LiteLLMRoutes.self_managed_routes
while its paginated sibling /user/daily/activity was listed, so auth
rejected every internal user with a 401 before the handler ran. That
route backs the default "Your Usage" view in the dashboard, which left
the main Usage page broken for non-admin users.
The handler already self-scopes: it checks admin view first, then falls
back to require_caller_user_id_for_non_admin, defaults a missing user_id
to the caller's own, and returns 403 when a non-admin asks for someone
else's data. Listing the route restores reachability without widening
what a caller can read.
check_route_access matches exactly (plus explicit wildcards), so the
parent entry never covered the /aggregated sub-path.
* fix: rebuild models_by_provider in add_known_models so cost map reloads reach wildcard expansion
* fix: refresh models_by_provider in place so captured references survive reloads
Multipart callers express nested metadata as flat bracket-notation keys, which
reach the request-body check as literal keys rather than as a metadata dict.
The check now rebuilds them with the same helper the endpoints use, so both
encodings are handled identically and cannot drift apart.
BREAKING CHANGE: a multipart field such as `litellm_metadata[api_base]` is now
subject to the same request-body parameter rules as its JSON equivalent. Set
`general_settings.allow_client_side_credentials`, or the deployment's
`configurable_clientside_auth_params`, to keep passing these.
`is_database_connection_error` answered True for any `PrismaError` it did not
recognize, on the reasoning that an unclassified failure might be an outage and
the safer default was to keep serving. That default is inverted for faults that
never resolve. A query engine that is missing or version-skewed, a malformed
generated query, or a misused transaction all satisfied the predicate, so with
`allow_requests_on_db_unavailable` enabled the proxy would absorb one, boot
clean, and keep issuing fallback identities for as long as the process ran.
The predicate is now an allowlist: the httpx transport errors, prisma's
`EngineConnectionError`, and a `no_db_connection` ProxyException. That is what a
real outage produces, since the query engine is a local HTTP server and an
unreachable database surfaces as a transport failure against it, so the
high-availability path is unchanged. Anything unrecognized is now treated as
permanent and surfaces instead of being absorbed.
Deciding whether to serve without a database and deciding what to tell the
caller are different questions, so they no longer share a predicate.
`is_database_infrastructure_error` keeps the previous broad behavior and now
backs the reporting and recovery paths: service-unavailable classification, the
access-group endpoint's status mapping, and the health watchdog's reconnect
trigger. Their behavior is unchanged. Without that split, a permanently faulted
engine would have started reporting as an authentication failure, sending an
operator after a credential problem that does not exist.
The enterprise gate on the OAuth2 auth path raised a bare `ValueError`,
which the terminal handler in auth_exception_handler.py converts to a 401.
Every sibling enterprise gate answers 403, including `_premium_user_check`
and the SSO gate. A 401 tells the client its credential was wrong and to
retry with a better one, and no credential can satisfy that while the
install is unlicensed, so it invites a retry loop that can never succeed.
It now raises a 403 `ProxyException` shaped like the SSO gate. Two response
fields move with it: the `Authentication Error, ` prefix goes away, since
the catch-all built that around `str(e)` and a `ProxyException` is re-raised
unmodified, and `param` becomes `premium_user`, naming the condition an
operator has to clear.
The gate's own text also gains the sentence break it was missing. The
message concatenated straight onto `CommonProxyErrors.not_premium_user`,
rendering as "premium usersYou must be a LiteLLM Enterprise user".
* fix(proxy): give proxy_admin_viewer read parity with proxy_admin
Route-level checks already default-allow management GETs for the viewer
role, but ~15 handlers compared user_role to PROXY_ADMIN only, dropping
viewers into regular-user scoping (/key/list, /user/info, /model/info,
guardrails, prompts, agents, memory, workflows, MCP catalog, coordination
redis settings, credential migration check, enterprise projects). Swap
those read paths to user_api_key_has_admin_view; write gates unchanged.
The dashboard now presents the viewer session as Admin for all gating
(effectiveSessionRole) so every page fetches with admin visibility, with
userRoleLabel/isViewOnly preserving the account-menu label and the
playground cost guard. The server remains the write authority.
* refactor(agents): remove side-effectful health_check param from GET /v1/agents
Addresses a security review finding on the admin viewer read parity change:
listing agents with health_check=true made the proxy issue a server-side GET
to every agent URL, so a read-scoped caller could trigger request fan-out
beyond their object permissions. The list endpoint is now a pure read for
every role.
Removes the query param, the URL probing helper and its timeouts, the
AgentHealthCheck httpx provider tag, and the dashboard's Health Check
toggle. Requests still passing health_check=true get the full list back
with the param ignored.
* fix(proxy): keep credential encryption check proxy_admin only
The residual scan behind GET /credentials/migrate-encryption/check loads
every model, credential, MCP, team, and verification-token row and runs a
decryption attempt on each stored value. Extending it to proxy_admin_viewer
let a read-only account repeatedly trigger deployment-wide scans, so the
route keeps its original full-admin gate.
* fix(agents): restore health_check, keep list fast path proxy_admin only
Restores the agent health_check feature exactly as before this PR: the
query param, the URL probing helper, the httpx provider tag, and the
dashboard toggle all return, so existing callers keep the filtering
contract. The viewer expansion is instead reverted at its source: the
GET /v1/agents admin fast path stays PROXY_ADMIN only, so a
proxy_admin_viewer goes through the object-permission scoped branch as
before and cannot fan out health checks beyond their allowlist. The
viewer read of a single agent stays viewer-inclusive since it has no
side effects.
A three-segment token presented while `general_settings.enable_jwt_auth` is
unset is never treated as JWT-shaped, so it falls through to the virtual-key
path and is rejected for not starting with 'sk-'. That reads as a missing
key in the verification table and sends the operator off to inspect virtual
keys, when the real cause is one missing config line. The rejection now
names `enable_jwt_auth`, appended to the existing text so the Prometheus
invalid-key filter and the admin UI keep matching what they match today.
The hint claims only that the key is JWT-shaped. Segment count cannot tell a
JWT from any other dotted credential, so asserting the key IS a JWT would
swap one confident misdiagnosis for a narrower one.
The enterprise gate on that same path raised a bare `ValueError`, which the
terminal handler turns into a 401. Every sibling enterprise gate answers
403, and a 401 tells the client to retry with a better credential, which no
credential can satisfy while the install is unlicensed. It now raises a 403
`ProxyException` like the SSO gate does.
* fix(proxy): propagate user_email and bind api_key on JWT auth paths
Standard JWT auth built UserAPIKeyAuth with user_id but never user_email, and the first auto-registered request early-returned a key with token set but api_key unset, so spend-log attribution logged user_api_key_user_email and user_api_key_hash as null. Bind api_key to the token hash on the auto-registered key, copy user_email from the resolved user object on both the standard and auto-register JWT paths, and warn when enable_jwt_auth/litellm_jwtauth are placed at the config top level where they are silently ignored.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover misplaced top-level JWT config warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan <ryan@berri.ai>
* feat(ui): add admin-configurable user banner
Proxy admins can publish a markdown announcement that renders as a
dismissible banner on every dashboard page for all authenticated users,
editable from Admin Settings > UI Settings without a redeploy. Backed by
new /get/user_banner and /update/user_banner endpoints persisting to the
existing LiteLLM_UISettings table
* fix(ui): re-surface dismissed banner on identical republish
Stamp a server-side revision on every banner update and fold it into
the client dismissal signature, so unpublishing and republishing the
same message reaches users who dismissed the earlier run
* fix(ui): stamp banner revision as an opaque uuid instead of a counter
Two overlapping admin updates could read the same prior revision and
both persist the same incremented value, letting an identical republish
collide with a previously dismissed signature. A server-generated uuid
per update makes every publication identity unique by construction with
no read-modify-write
* refactor(ui): drop the server-side banner cache
Reads go straight to the single-row table; the dashboard already
throttles fetches client-side, so the cache only added staleness
windows under concurrent updates and multiple workers
* refactor(ui): move banner storage behind a domain repository and drop the store_model_in_db gate
UserBannerRepository owns the row shape instead of the endpoint
reaching through the generic .table bridge, and publishing no longer
depends on the unrelated STORE_MODEL_IN_DB flag; a connected database
remains the only requirement
* fix(proxy): backfill null user_email on existing users during JWT auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): guard mapped-key email backfill and make null update atomic
Resolve Greptile review on the JWT user_email backfill:
- only backfill when the mapped virtual-key owner is the JWT principal, so a
mismatched admin-created mapping cannot write one user's email onto another
- make the best-effort mapped-key enrichment non-fatal so a database outage on
a cached-key request no longer fails otherwise-valid authentication
- persist the backfill with an atomic null-guarded update_many so concurrent
writers cannot overwrite an already-populated email
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep cache coherent when a concurrent backfill wins the null-email update
* fix(proxy): cache DB-persisted email after JWT backfill, not the proposed value
Resolve the Greptile finding that a successful null-guarded backfill could
cache this request's proposed email even if a concurrent ordinary user update
wrote a different email first. The helper now always re-reads the row after the
atomic update and refreshes the cache from the value the database holds, so
cache-hit auth and attribution stay consistent with the persisted record.
Annotate the Prisma and model_copy dict literals to keep the LIT002 budget within its ceiling.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Keys created with key_type=llm_api get allowed_routes=["llm_api_routes"],
which covered /v1/models but not /v1/model/info, so a client could list model
names but not read pricing, mode, or max_tokens without a second key.
Adds both /model/info and /v1/model/info (same handler) to llm_api_routes only.
Membership there is not the same as RouteChecks.is_llm_api_route(), which is
what gates DISABLE_LLM_API_ENDPOINTS, global/virtual-key budget enforcement,
enforce_user_param and JWT team attachment; /guardrails/apply_guardrail already
sits in the group the same way. /v2/model/info stays out: it is the paginated
Admin UI listing, not model metadata a caller needs at request time.
public_routes moves from set([...]) to a frozenset literal to keep the LIT002
and ruff-strict ceilings from rising; both budgets ratchet down by one.
* feat(proxy): add a generic list contract for management/v1 entity lists
Paging, sorting, filtering and search for an entity collection, declared once
as a ListSpec and served by handle_list. The route injects a ListExecutor that
owns its table, so this module never imports Prisma.
The caller's scope is derived from the caller alone and ANDed with whatever
they filtered on, so a query parameter can only narrow what they may read.
This is the shared half of the budgets list; it lands here so the endpoint has
something to register against, and drops out when the framework arrives on its
own branch.
* feat(proxy): add GET /management/v1/budgets
The Budgets page reads /budget/list, which returns the whole table as a bare
array with no way to page, sort or filter it. A customer with enough budgets to
fill the page has no way to find one.
Registers LiteLLM_BudgetTable against the management/v1 list contract: sortable
on budget_id, max_budget, tpm_limit, rpm_limit and created_at, default order
newest-first with budget_id breaking ties, search on budget_id, and filters for
budget_duration, max_budget and created_at. budget_duration is deliberately not
sortable; the column holds "7d"/"30d" strings, so a lexicographic ORDER BY puts
"30d" ahead of "7d".
tpm_limit and rpm_limit are BigInt? in Prisma, so rows validate through a
pydantic model on the way out and serialize as JSON numbers.
A caller without admin view is refused 403 as a problem document rather than
served an empty page. /budget/list is untouched.
* fix(proxy): rework the budgets list onto the merged list contract
PR #35308 landed a different shape than this branch was written against: `where`
is a tuple of frozen predicates rather than a Prisma-shaped mapping, `ListSpec`
carries both the row and the wire type, and `where_sql` / `order_by_sql` render
for a raw-SQL executor. The budgets executor now queries through `query_raw` the
way the spend logs facet does, selecting only the columns it serves.
Also casts datetime binds in `where_sql`. They cross into the query engine as
JSON, so an uncast placeholder arrives as text and Postgres refuses
`timestamp >= text` outright; every `filter[created_at][gte|lte]` was answering
500. The cast reads the bind as an instant and drops it to naive UTC to match
Prisma's TIMESTAMP(3) column, the same one /spend/logs/ui applies.
* refactor(proxy): fold the predicate renderer instead of recursing
recursive_detector flags `_render_all`, and the flag is fair: it recursed once
per predicate, so the stack grew with the number of filters on the request for
no reason. Walking a predicate list is a running bind index, which is a fold.
`_render` still re-enters for `AnyOf`, but its clauses are plain comparisons
built by `?q=`, so that nesting is one level deep and no caller can drive it
deeper.
* fix(proxy): only enforce budgets on routes that can spend
Budget checks ran inside common_checks with no route filter, so an
over-budget user, team, organization or tag got a 429 on every
authenticated route, including the management calls the Admin UI makes
on load. An internal user who exhausted their budget could not open the
dashboard to see why, and a max_budget of 0 locked them out from the
moment the account existed.
Gate the scope budget checks on RouteChecks.is_llm_api_route, matching
the virtual key budget check, the reservation path and the global proxy
budget check, which already scope themselves this way. /health/services
keeps enforcing because it fires Slack, email and webhook sends.
The Admin UI is affected because a UI login mints a virtual key scoped
to the litellm-dashboard pseudo-team. That token was shielded from
personal budgets by the team-key exemption until #32005 removed it.
* fix(proxy): keep budget enforcement on provider-calling health routes
/health and /health/test_connection are not LLM API routes but both run
litellm.ahealth_check against real deployments, so exempting them let an
exhausted budget keep incurring provider spend.
Add them alongside /health/services in BUDGET_ENFORCED_SIDE_EFFECT_ROUTES
and cover all three with a regression test.
* chore(ui): drop env-dependent schema.d.ts regeneration from this PR
The regenerated diff was union-member reordering only, with no change to
the represented types, and the ordering differs between a local run and
CI. Keeping the committed file as-is lets the drift check pass and keeps
this PR to the auth change.
* chore(ui): restore schema.d.ts to the branch base
The previous commit restored it from the staging tip, which pulled in
unrelated merged changes. This PR changes no backend models, so the file
should be untouched.
Reverts #32005. Team-scoped keys are governed by the team and team-member
budgets only; the key owner personal max_budget no longer applies to them,
restoring the hierarchy that existed before that PR.
The skip_user_budget_on_team_key opt-out existed solely to turn the new
behavior back off, so it is removed along with the behavior: the
ConfigGeneralSettings field, the /config/list allowed_args entry that
surfaced it as an Admin UI toggle, and the argument threaded through
reserve_budget_for_request and _get_budget_counters.
Regression tests cover both enforcement points in the restored direction:
test_common_checks_personal_user_budget_skipped_for_team_key for the
read-time check and test_should_not_reserve_user_budget_counter_for_team_key
for the optimistic reservation path.
Add regression tests for the db-fetch paths whose converted construction
lines were uncovered: the auth_checks getters (default end user budget, end
user, team membership, access group, team by alias, org by alias, object
permission, managed vector stores, project), get_all_team_memberships and
list_available_teams in team_endpoints, and the proxy admin user info
helper. Each test feeds a mocked prisma row through the real function and
asserts the validated model's fields, so a bad model_validate conversion on
any of these paths now fails a test instead of only dropping coverage.
* fix(auth): route JWT default-team into memberships instead of the create payload
JWT auto-provisioning (get_user_object with user_id_upsert) merged
litellm.default_internal_user_params verbatim into the Prisma user create,
including a teams key. When a default team is configured through the Admin
UI it is stored as a list of NewUserRequestTeam objects, but the user
table's teams column is String[], so the create raised a Prisma type error
and every JWT-authenticated request 401'd with the user never created.
Mirror the /user/new path: strip teams (and available_teams) out of the
create payload, then route the configured default team through
check_if_default_team_set / add_new_user_to_default_team so provisioned
users get real membership rows. Reuse the synthetic PROXY_ADMIN
UserAPIKeyAuth pattern already used by the team-upsert path to satisfy the
membership permission gate, and import the helpers lazily to avoid the
auth_checks <-> internal_user_endpoints import cycle.
* fix(auth): propagate max_budget_in_team when adding users to default teams
* fix: use pipe union instead of Optional for UP045 budget
* fix(proxy): enforce global max_budget against the resettable proxy budget row
The global proxy budget check compared litellm.max_budget against
SUM(spend) from the MonthlyGlobalSpend view, whose window is hardcoded
to a trailing 30 days. litellm.budget_duration was stored and reset on
a user row that enforcement never read, and startup budgeted the admin
user's own row (default_user_id) instead of the litellm-proxy-budget
aggregate row the spend writer increments per request. Net effect: 1d,
7d and 30d all behaved as a trailing 30 day cap that never reset on the
configured duration.
Startup now upserts the budget onto the litellm-proxy-budget row (and
zeroes lifetime accrual when first putting a row on a reset schedule),
enforcement loads global spend from that row, and ResetBudgetJob drops
the cached global spend accumulator when it resets that row so the cap
unblocks immediately after each window.
Fixes https://github.com/BerriAI/litellm/issues/31292
* refactor(proxy): address review nits on global proxy budget fix
Drop the redundant litellm_proxy_budget_name parameter from
_upsert_proxy_budget_with_reset_at_backfill; its only caller always passed
LITELLM_PROXY_BUDGET_NAME, and any other value would write the budget to a
row enforcement never reads.
Introduce GLOBAL_PROXY_SPEND_CACHE_KEY in constants.py and use it at every
site that previously built the key from litellm_proxy_admin_name (auth
loads, spend-writer increments, startup warm, reset-job invalidation), so
the reader and invalidator can no longer drift apart. The literal key value
is unchanged. Also drop the now-pointless litellm_proxy_admin_name
parameter from _warm_global_spend_cache and the proxy_server import from
the reset-job helper.
* fix(proxy): attribute spend to org for team-linked keys minted without org_id
Keys attached to an org-linked team but minted without an organization_id
produced spend that was never credited to the org: the spend writer reads
user_api_key_dict.org_id with no team fallback, while the org budget check
resolves the org from the team. The check therefore ran against a counter
fed by almost none of the org's traffic and never tripped.
Backfill org_id from the freshly fetched team object in
_run_centralized_common_checks, per request only, so the spend writer and
the budget check read the same org. A key with an explicitly pinned org_id
always wins, and the cached key row is never mutated, so moving a team to
a different org takes effect on the next auth once the team cache
refreshes.
* test(proxy): cover CLI session-token org backfill from team
CLI session tokens from /sso/cli/poll are minted with a real team_id but
no org_id, and their auth path decrypts the blob without the combined_view
team join that fills org for DB keys. Spend from these tokens reached the
team but never the org, so org budgets never tripped. The regression test
mints a real CLI token, runs it through the centralized checks, and
asserts the credential leaves auth with the team's org.
* fix(proxy): reject request when budget reservation write fails under fail_closed_budget_enforcement
With general_settings.fail_closed_budget_enforcement set to true, the read-time
spend check already returns 503 when spend cannot be verified, but the atomic
pre-call reservation still failed open: reserve_budget_for_request swallowed
_CounterReservationUnavailable per counter and degraded to read-time-only
enforcement, so concurrent requests could all pass the same under-budget read
during a Redis outage and overspend past the configured budget.
Now the strict flag is threaded into reserve_budget_for_request and a failed
reservation write raises 503, releasing any counters that already reserved.
Default behavior with the flag absent or false is unchanged.
Fixes#33923
* fix(proxy): pass 503 budget-enforcement detail as plain string
* feat(proxy): add overwrite_user_with_key_hash to stamp outgoing user param with key hash
Adds a litellm_settings flag that forces the outgoing user param to the
authenticated key's hashed token before the request is forwarded to the
provider. The value overrides any caller-supplied user, so providers see
a stable, tamper-proof identifier they can rate-limit or ban on, and the
hash matches user_api_key_hash in spend logs for easy mapping back to
the key owner. Off by default
* fix(proxy): hash non-sk credentials before stamping user param
UserAPIKeyAuth only hashes sk-prefixed keys and JWTs; custom-auth
credentials stay raw on api_key, so stamping them directly would forward
auth material to the provider. Pass through the two known hashed forms
(sha256 hex, hashed-jwt-*) and hash anything else
* refactor(proxy): stamp only standard virtual keys, skip jwt and custom auth
A hashed JWT rotates on every token re-issue so it is useless as a
stable ban id, and custom-auth credentials arrive raw on api_key.
Instead of hashing whatever we hold, the stamp now applies only when
api_key is the sha256 hex digest of a standard virtual key; other auth
methods are explicitly out of scope until the stamped identifier is
configurable
* fix(proxy): gate user stamping on server-set virtual key provenance
Shape alone cannot distinguish a key hash from a raw custom-auth
credential that happens to be 64 hex chars. Adds via_virtual_key, a
server-only marker on UserAPIKeyAuth following the
mcp_admitted_user_subject pattern: stripped from all validated input so
handlers and claims cannot forge it, set by post-construction assignment
only at the DB virtual-key auth return. Stamping now requires the marker
and the hash shape
* test(proxy): prove db auth path sets via_virtual_key marker
The stamping unit tests set the marker manually, so deleting the
assignment in _user_api_key_auth_builder would pass every existing test;
this exercises the real builder path with a mocked identity store and
fails if the marker is not set
* fix(proxy): stamp master-key requests with the master key alias
Master-key auth substitutes LITELLM_PROXY_MASTER_KEY_ALIAS for api_key
so the key and its hash never propagate; that made master-key traffic
bypass stamping and pass the caller-supplied user through. The master
path now sets via_virtual_key and the stamp gate accepts the alias
alongside the sha256 shape, so admin traffic gets the same tamper-proof
id that spend logs already record for it
* fix(proxy): restore via_virtual_key marker on key-cache hits
Cached PROXY_ADMIN auth objects early-return before the marked DB and
master-key returns, and cache serialization drops the exclude=True
marker, so cached admin traffic bypassed stamping. Key-cache entries are
written only after the proxy validated a virtual key or the master key,
so the cache-hit boundary restores the marker; the UI-login JWT fallback
constructs its token from a decrypted blob, not this cache, and stays
unmarked
Admits a keyless SSO user (no virtual key) at the aggregate /mcp endpoint from a gateway DCR
session bearer, resolving team/org/SCIM/budget authorization fresh on every call.
- Aggregate DCR front door: stateless /register (sealed llm_dcrc_ client ids), SSO-backed
/authorize + /authorize/complete, and /token minting identity-only session tokens with PKCE,
single-use codes/flows, and rotating refresh tokens.
- Admission: a session-shaped Authorization at the aggregate scope opens via _admit_gateway_session,
reloads the live user, and runs the centralized policy gate; failures return the RFC 9728
invalid_token challenge. Gated on the un-forgeable, server-only mcp_admitted_user_subject marker,
so virtual-key and JWT auth are unchanged.
- Authorization model: an admitted subject is resolved as one plain UserAPIKeyAuth per grant source
(its own grants, plus each team it is a live roster member of), each answered by the SAME resolver
virtual keys use, then unioned. That branch is the FIRST statement of BOTH public resolvers, so no
single-credential prelude runs for it and a fault in a lookup it never uses cannot deny its grants. A source team counts only while it is a live grantor: roster membership, not
blocked, and neither the team nor its owning org over budget (enforced through the SAME
_team_max_budget_check / _organization_max_budget_check owners common_checks uses for keys).
Each team source carries that team's own org, so the existing org
ceiling caps it; for a keyless source the org list only ever intersects (a ceiling must not become
a grant) and an unresolvable ceiling denies rather than silently uncapping, on both the server and
tool axes. _roster_team_object is the single owner of "which teams count": a team whose roster no
longer lists the user neither grants servers nor throttles, in one place.
- Rate limits: the subject is bounded by its user rpm/tpm AND by the per-server mcp_rpm_limit of
the team a call is ATTRIBUTED to — the same single source billing charges, from the same owner. A key charges its one pinned team's bucket; a keyless
subject has no team_id, so admission stamps each granting team's limit map onto the auth
(server-only field, stripped from validated input like the marker) and the limiter emits that
team's mcp_per_team descriptor. Charging every granting team instead would let one cross-team user
drain several teams' SHARED buckets on a single call and block their other members; and a server
the user's OWN grant reaches charges no team bucket at all, because no team provided it. Per-KEY
MCP limits do not apply because there is no key.
- Wrapper channels: the manager-level union treats the admitted subject by the same grant model.
The admin-role short-circuit and the absolute no_mcp_servers early-return are key-credential
rules and never apply to it (a session bearer is a third-party client credential, not the
dashboard, and the subject's opt-out silences only its own source). Operator-open channels
(allow_all_keys, the user's own BYOM submissions) are owned by one operator_open_server_ids
helper that BOTH the server union and the admitted tool resolution consult (suppress-BYOM-when-
explicitly-scoped is a key-credential rule and never applies to the subject, whose user row
carries the DB-default empty mcp_servers), so an open-channel
server is default-open for tools instead of listable but uninvokable.
- Redirect URIs: one owner, validate_redirect_uri_shape, decides redirect-URI hygiene (bad scheme,
fragment, missing host, userinfo, backslash host) and resolves allowlisted native callbacks, shared
by DCR registration and the OAuth endpoints. Registration keeps a deliberately wider trust policy
than validate_trusted_redirect_uri: public dynamic registration accepts any https client, and its
controls are mandatory S256 PKCE plus the consent screen.
- Egress leak-defense: a gateway admission credential (session bearer / bridge envelope) is scrubbed
from EVERY egress header context, anchored to the credential shape, so it can never be forwarded
upstream and replayed.
- Single-use guard: auth-code, refresh and connect-flow claims resolve the proxy's cross-worker redis
cache themselves rather than trusting the cache passed in, and fail CLOSED on a Redis fault instead
of falling back to a per-worker count that a captured id could replay through another worker.
- Sign-in return_to: one shared, never-raising helper persists a safe return_to for every sign-in
branch (SSO/Okta/generic and username/password), and every branch RESUMES through the same
_sso_return_to_redirect the SSO callback uses, so however a deployment signs in the stored value
is honored identically (same-origin path directly; control_plane_url via the one-time login-code
handoff). A stale cookie is ignored rather than failing a completed sign-in.
- Budgets, both halves: ENFORCEMENT (an already over-budget team or its owning org stops being a
grantor, in the source gate) and ACCOUNTING (a team-derived tool call is billed to the granting
team and ITS org, so that budget accumulates and the right organization is charged). A server the
user's own grant reaches bills the user; when several teams grant one server the pick is the
lowest team_id, stable and auditable. Billing rides a COPY, so authorization still sees the full
union, and it is inert when the target server cannot be resolved from the tool name.
Deferred (tracked): client-selected server scoping of the session token (LIT-4680).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Greptile review: _cache_team_object runs after a successful DB fetch in
get_team_object and after every team mutation's DB write, but
DualCache.async_delete_cache propagates backend errors, so a Redis blip
during invalidation would turn a healthy team lookup into a 404 and a
committed /team/update into a 500. Both the internal usage cache delete
and the alias-key invalidation now log a warning and continue on failure,
matching how DualCache.async_set_cache already swallows write errors.
Worst case on failure is worker-local staleness bounded by the internal
cache's in-memory TTL, the same bound other workers already have
get_team_object consults proxy_logging_obj.internal_usage_cache before
user_api_key_cache, but _cache_team_object (the refresh every team
mutation goes through) only wrote user_api_key_cache. With
enable_redis_auth_cache both caches share one Redis, so any request
backfills the internal cache's in-memory tier with the team object and
that copy keeps shadowing the freshly written team until its TTL expires.
The auth builder then wrote the team object it had just read back into
the cache after check 6, clobbering the fresh Redis value with the stale
one, which made the staleness self-sustaining under traffic: keys with
models=["all-team-models"] kept getting 403 team_model_access_denied
for models added via /team/update, and kept access to removed ones.
_cache_team_object now deletes the internal usage cache entry before
writing the refreshed team, and the auth-time write-back is removed so
only authoritative writers (DB reads and team mutations) populate the
team cache, mirroring how key objects already handle this (see
test_auth_does_not_rewrite_cached_key_object_back_into_cache).
The LIT-4000 test pinning the removed write-back is deleted; its
concern (team object cached under the canonical key) is handled by
_cache_team_object inside get_team_object's DB path and pinned by
test_cache_team_object_writes_team_id_and_invalidates_team_alias
Resolves LIT-4391
JWT auth built UserAPIKeyAuth without user_email even though the resolved
user row and the JWT email claim were both available, so the user_email
label on Prometheus metrics and user_api_key_user_email in
StandardLogging/SpendLogs metadata were always None for JWT traffic.
Plumb user_email through JWTAuthBuilderResult: auth_builder returns the
user row email when set, falling back to the user_email_jwt_field claim
(covers the scope-based proxy-admin path where no user row is loaded).
The JWT branch now stamps it on the proxy-admin return, the standard
valid_token, and the auto-registered virtual key object.
Resolves LIT-4238
* refactor(auth): derive temp budget bump without mutation, tz-aware auth datetimes
_update_key_budget_with_temp_budget_increase mutated max_budget in place, so correctness depended on every resolution path handing it a fresh copy of the cached token; one future re-cache of a live token would compound the bump per request. Return a model_copy instead so no caller can leak an increased budget into shared state.
Also fixes the three remaining DTZ005 naive datetime.now() calls in user_api_key_auth.py (auth span start, builder start_time, service-log end_time; all consumers convert to epoch or subtract same-pair datetimes) and ratchets the DTZ005 strict budget 244 -> 241.
* test: pin non-mutation of the temp budget helper input
Adversarial mutation-testing showed reverting the helper to in-place mutation still passed every test: the cache's copy-on-read layer masks the mutation in the integration test and the direct unit test only inspected the return value. Assert the input object is left untouched and the result is a distinct object so the purity guarantee itself is load-bearing.
temp_budget_increase was only applied on the DB-fetch path of _user_api_key_auth_builder, so a key served from the auth cache reverted to its original max_budget and was wrongly blocked with BudgetExceededError once spend crossed the original budget while staying under the effective budget.
Move _update_key_budget_with_temp_budget_increase out of the DB-only branch so it runs for every resolved token regardless of source. The cache stores the original budget and each cache hit returns a fresh model_copy(), so this never double-applies.
Fixes#25760
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
