Previously, deleting a user via SCIM (`DELETE /scim/v2/Users/{id}`) or
marking them inactive (`PATCH active=false` / `PUT active=false`) only
touched the user row. Their virtual keys kept working because:
- `litellm_verificationtoken` was never updated.
- The auth path's combined-view query on the key never joined to the
user's active state.
- `get_user_object()` was wrapped in a silent `except` that set
`user_obj=None` when the owning user record was gone, so requests
proceeded normally.
Changes:
- Add `_set_user_keys_blocked(user_id, blocked)` in scim_v2.py that
flips only mismatched rows via `update_many` and invalidates each
affected token in the dual cache.
- Cascade SCIM lifecycle events to keys:
- `delete_user`: block all of the user's keys before deleting the
user row (preserves spend/audit while orphaning safely).
- `patch_user` / `update_user`: on `scim_active` transitions,
block (false) or unblock (true) the user's keys.
- Defense in depth in `user_api_key_auth`: reject the request when the
loaded `user_obj` has `metadata.scim_active == False`, even if a
cached key snuck past the per-key block.
- `transform_litellm_user_to_scim_user` now reflects the real
`scim_active` value instead of always returning `active=True`.
Tests:
- New `test_scim_key_deactivation.py` covering DELETE, PATCH
active=false, PATCH active=true, no-op patches, and the helper's
cache-invalidation contract.
- New `test_scim_deactivated_user_key_is_rejected` exercising the
auth-path defense.
- Existing PATCH tests updated with verificationtoken mocks for the
new code path.
* chore(auth): validate clientside api_base against SSRF guard; clear admin secrets on base override
Two related issues with how the proxy handles client-supplied
``api_base`` / ``base_url`` overrides on chat-completion requests:
1. **SSRF gate bypass** — ``check_complete_credentials()`` returned
``True`` for any non-empty ``api_key``, allowing the
``is_request_body_safe`` ``banned_params`` loop to admit ``api_base``
/ ``base_url`` values that point at private (RFC 1918), loopback,
link-local, or cloud-metadata addresses. Now: when the gate sees a
client-supplied ``api_base`` / ``base_url``, it runs the URL through
``litellm_core_utils.url_utils.validate_url`` (DNS-resolves, blocks
internal/IMDS/LL networks, defends against rebinding). Rejection
raises with a clear message.
2. **Admin-config leak on base override** —
``get_dynamic_litellm_params`` only carried the three clientside keys
(``api_key``, ``api_base``, ``base_url``) from request to upstream
call. Other admin-configured fields on ``litellm_params`` —
``organization``, ``extra_body``, ``extra_headers``, ``api_version``,
``azure_ad_token``, AWS / Vertex creds, etc. — flowed through
unchanged. With base redirected to a client-controlled server, those
admin secrets were sent to the attacker. Now: when ``api_base`` /
``base_url`` is in ``request_kwargs``, drop those admin-config
fields from ``litellm_params`` unless the caller re-supplied them.
Tests cover the SSRF-target rejection per URL field, the admin-secret
clearing on base override, the don't-clear case when only ``api_key``
is overridden (BYOK pattern), and the don't-overwrite case when the
caller resupplies fields like ``organization`` themselves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(vertex-batches): wrap api_base GET in safe_get for defense-in-depth
The vertex batches status-poll fetches an attacker-influenceable
``api_base`` URL with a raw ``sync_handler.get()``. The proxy auth gate
already validates clientside ``api_base`` before reaching this sink, so
the proxy flow is covered. This adds the per-sink wrap so SDK callers
and any future code path that bypasses the proxy gate pick up the same
SSRF defense from ``url_utils.safe_get``.
Operators with a legitimate private Vertex base can either allowlist
the host via ``litellm.user_url_allowed_hosts`` or disable validation
with ``litellm.user_url_validation = False``.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(auth): hoist url_utils import; derive admin-config field list from CredentialLiteLLMParams
/simplify pass:
- Move ``from litellm.litellm_core_utils.url_utils import SSRFError, validate_url``
to module top in ``proxy/auth/auth_utils.py``. CLAUDE.md prefers
module-level imports unless avoiding a circular dependency, and
there's no cycle here (``url_utils`` doesn't depend on ``proxy.auth``).
- Replace the hardcoded ``_ADMIN_CONFIG_FIELDS_TO_CLEAR_ON_BASE_OVERRIDE``
literal with ``_admin_config_fields_to_clear_on_base_override()`` that
derives the typed-field portion from
``CredentialLiteLLMParams.model_fields``. Adds three fields the
hardcoded list missed (``aws_bedrock_runtime_endpoint``,
``watsonx_region_name``, ``region_name``) and stays in sync as new
provider fields are declared on the model. The kwargs-only set
(``organization``, ``extra_body``, ``azure_ad_token``, ``aws_session_token``,
``aws_sts_endpoint``, ``aws_web_identity_token``, ``aws_role_name``, …)
remains explicit since those fields aren't on the typed model.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(auth): close field-echo bypass; gate URL check on toggle; cover async batch path
Three issues from review:
1. ``get_dynamic_litellm_params`` used ``if field not in request_kwargs:
pop`` to clear admin-set provider config when the caller redirected
``api_base``. A caller could *echo* any clear-list field name (with any
value, including an empty string) to skip the pop, leaving the admin's
value in ``litellm_params`` to be forwarded to the redirected upstream.
Fix: always pop, then write the caller's value back if they resupplied
the field.
2. ``check_complete_credentials`` called ``validate_url`` directly. That
helper doesn't itself consult ``litellm.user_url_validation``; the
toggle is honoured by ``safe_get`` / ``async_safe_get``. Mirror that
here so admins who explicitly disabled URL validation aren't blocked
at the proxy boundary.
3. ``VertexAIBatchesHandler._async_retrieve_batch`` still used a bare
``await client.get(api_base, ...)`` while the sync sibling was wrapped
in ``safe_get``. Wrap the async call in ``async_safe_get`` so SDK
callers on the async path get the same DNS-rebind / private /
cloud-metadata defenses as the sync path.
Tests:
- ``TestCheckCompleteCredentialsBlocksSSRF`` is now mock-only; an autouse
fixture flips the toggle on, ``validate_url`` is patched in the
parametrized blocking tests, and the positive path no longer makes a
real DNS call to api.openai.com.
- ``test_skips_url_validation_when_toggle_is_off`` documents the new
toggle-off behaviour and asserts ``validate_url`` is not called.
- ``test_caller_resupplied_value_overrides_admin_value_on_base_override``
replaces the prior test that asserted the buggy
preserve-admin-value-on-echo behaviour.
- ``test_field_echo_does_not_preserve_admin_value`` is a focused
regression test for the empty-string echo vector.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(auth): close provider-confusion credential exfil; expand banned-params; cover OCI
Three additions on top of the entry-point URL gate so the cluster is
fully closed against caller-supplied ``api_base`` redirection:
1. ``get_llm_provider_logic.py`` matched registered openai-compatible
endpoints against ``api_base`` with an unanchored substring search
(``if endpoint in api_base:``). A caller could pass an api_base like
``https://attacker.com/api.groq.com/openai/v1`` to coerce the proxy
into reading ``GROQ_API_KEY`` from the environment and forwarding it
as a Bearer credential to the attacker's host. Replaced with parsed-
URL semantics (hostname exact-match plus segment-bounded path-prefix)
in a new ``_endpoint_matches_api_base`` helper.
2. ``is_request_body_safe`` rejects ``api_base`` / ``base_url`` /
``user_config`` / a handful of AWS / vertex fields, but the list
omitted three other endpoint-targeting fields:
* ``aws_bedrock_runtime_endpoint`` — Bedrock endpoint redirect
* ``langsmith_base_url`` / ``langfuse_host`` — observability callback
hostnames; attacker-controlled values exfiltrate the entire request
payload (incl. message content) via the logging hook.
Added all three to the blocklist.
3. ``_admin_config_fields_to_clear_on_base_override`` derives its typed-
field list from ``CredentialLiteLLMParams.model_fields``, which does
not declare any of the OCI provider's auth fields. Added
``oci_signer``, ``oci_user``, ``oci_fingerprint``, ``oci_tenancy``,
``oci_key``, and ``oci_key_file`` to the kwargs-only fixed list so
they are cleared on caller-redirected ``api_base`` like the AWS /
Azure / Vertex equivalents.
Tests:
- ``TestEndpointMatchesApiBase`` — direct unit tests on the new
matcher: legitimate provider URLs (5 shapes) match; attacker
smuggling via path injection, suffix label, prefix label, userinfo
``@`` injection, and path-segment lookalikes (7 shapes) do not.
- ``TestGetLlmProviderRejectsAttackerSmuggledApiBase`` — end-to-end
invariant that ``GROQ_API_KEY`` is never read against an attacker-
controlled host while the legitimate ``api.groq.com`` path still
resolves the provider correctly.
- ``TestIsRequestBodySafeBlocksEndpointTargetingFields`` — parametrized
coverage that each of the three new banned-params raises a clear
rejection naming the offending field.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(auth): remove implicit api-key bypass + add posthog/braintrust/slack to blocklist
The historical ``check_complete_credentials`` clause inside
``is_request_body_safe`` was a third, *implicit*, *caller-controlled*
BYOK path: any caller that supplied a non-empty ``api_key`` caused the
entire banned-params blocklist to be skipped. That turned every missing
entry on the blocklist into an exploitable SSRF / credential-exfil hole
and is the root cause of the chain of api_base advisories that have
been re-discovered with each new integration:
* GHSA-jh89-88fc-qrfp (critical, triage) — env-var exfil via api_base
* GHSA-3frq-6r6h-7j64 (high, triage) — admin org / extra_body leak
* veria-admin Dv_m860l, b_yRJeQ5, stN90yjP, LBlyOAc8, U2TD78kg —
variations on "list X is missing field Y"
Two explicit, admin-controlled BYOK paths already exist and remain:
``general_settings.allow_client_side_credentials = true`` (proxy-wide)
and ``configurable_clientside_auth_params: [...]`` per deployment.
Removing the implicit bypass converts the failure mode of a missing
blocklist entry from "live credential leak" to "predictable 400 with
a clear remediation message," which is the structural fix.
Also adds the three remaining endpoint-targeting fields the dynamic
callback layer reads from request body: ``posthog_host``,
``braintrust_host``, ``slack_webhook_url``. ``slack_webhook_url`` in
particular was a direct exfil channel (caller-set webhook → proxy
mirrors every request to attacker's Slack).
Tests:
- ``test_api_key_does_not_bypass_blocklist`` — parametrized regression
asserting api_key=anything no longer skips the gate for any of the
five highest-risk fields.
- ``test_admin_opt_in_proxy_wide_still_allows`` — confirms the
documented BYOK opt-in still works.
- Extends ``test_endpoint_targeting_field_in_request_body_is_rejected``
to cover posthog / braintrust / slack.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(auth): block sagemaker_base_url, s3_endpoint_url, deployment_url
Provider-specific endpoint overrides surfaced by a wider audit of
``optional_params`` consumers in ``litellm/llms/``. Same threat as
``api_base``: a caller-supplied value redirects the outbound request
to an attacker host.
* ``s3_endpoint_url`` — read in ``litellm/llms/bedrock/files/transformation.py``
to build the S3 upload URL for Bedrock files. Caller redirects file
uploads to attacker-controlled S3.
* ``sagemaker_base_url`` — read in ``litellm/llms/sagemaker/{chat,completion}/*``.
Caller redirects SageMaker traffic. This is the primary vector
described in veria-admin mNqEBBtG.
* ``deployment_url`` — popped in ``litellm/llms/sap/chat/transformation.py``.
Caller redirects SAP deployment requests.
Tests parametrize ``test_endpoint_targeting_field_in_request_body_is_rejected``
to cover the three new fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Greptile flagged the unused ``from unittest.mock import patch``
left over from before the ``configure_proxy`` fixture refactor (the
fixture uses ``monkeypatch``, no ``patch`` calls remain). Also pruned
the now-stale "premium gate" paragraph from the module docstring
since that gate was removed in fbcfd59b1a.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Greptile flagged the ``premium_user is not True`` check as a hard
backwards-incompatible break for OSS users currently running
``enable_oauth2_proxy_auth=True``. They were right: unlike the
api_base case (where the docs already required admin opt-in), this
path was documented as available to OSS users. Adding the gate would
have closed a documented feature, not fixed a vuln.
Reframed the change:
* The **identity-only allowlist** (``ALLOWED_OAUTH2_PROXY_FIELDS`` =
``{user_id, user_email, team_id, team_alias, org_id, models}``) is
the actual security fix — it closes the privesc by rejecting any
mapping to a non-identity field at request time. This is unchanged.
* The **premium gate** was parity-with-siblings (a product decision,
not a security one). Removed. BerriAI can re-add it on their own
schedule with a proper deprecation cycle if they want enterprise-
only gating.
Tests: removed ``test_rejects_when_not_premium``; everything else
(allowlist enforcement, identity passthrough, attack-shape
regression) still passes — 14 tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Greptile flagged that the denylist was incomplete: ``user_max_budget``,
``user_tpm_limit``, ``user_rpm_limit``, and ``user_spend`` were not on
it. Inspection of the auth model showed dozens more privileged fields
across the ``LiteLLM_VerificationTokenView`` hierarchy (team / org /
end-user / region budget / spend / limit fields, plus
``allowed_model_region``, ``rpm_limit_per_model``, etc.) — a denylist
of "privileged fields" is unmaintainable here.
Inverted the model. ``ALLOWED_OAUTH2_PROXY_FIELDS`` is now an
identity-only allowlist: ``user_id``, ``user_email``, ``team_id``,
``team_alias``, ``org_id``, ``models``. Any mapping to a non-identity
field is rejected at request time. Default-secure: a future field
added to ``UserAPIKeyAuth`` is automatically blocked from
header-trust.
Use case for OAuth2-proxy auth is identity assertion from a trusted
upstream. Anything beyond that (privileges, budgets, rate limits) is
policy and should be authenticated with a signature, not a header —
operators who need this should switch to JWT auth.
Tests:
- ``test_refuses_to_map_non_identity_fields`` parametrized over 22
fields including all four ``user_*`` Greptile flagged, plus
team/org/end-user budget/limit fields, plus a fabricated field name
to confirm "anything not on the allowlist" is the rule.
- ``test_allowlist_is_identity_only`` locks in the allowlist's intent
so future additions of budget / role / permission entries are caught
in review.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two cleanups from the /simplify review pass:
* The header-mapping loop had a special-case ``if key == "max_budget":
auth_data[key] = float(value)`` branch. Since ``max_budget`` is now
in ``PRIVILEGED_OAUTH2_PROXY_FIELDS``, the denylist check rejects
the configuration before the loop runs — the float-conversion
branch is unreachable. Removed.
* Four tests independently called
``monkeypatch.setattr(proxy_server, "premium_user", ...)`` and
``monkeypatch.setattr(proxy_server, "general_settings", ...)`` with
almost-identical bodies. Replaced with a ``configure_proxy`` fixture
that yields a single callable —
``configure_proxy(premium=False)`` /
``configure_proxy(mappings={...})`` — so each test's setup is one
line. The previously-unused ``premium_proxy_settings`` fixture is
removed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
``handle_oauth2_proxy_request`` reads HTTP request headers per the
admin-set ``oauth2_config_mappings`` and constructs a
``UserAPIKeyAuth`` from the values. Two failure modes:
1. **Premium parity.** Sibling auth paths
(``enable_oauth2_auth``, ``enable_jwt_auth``) require
``premium_user``; this path did not, so any open-source deployment
could turn the feature on without realising it requires a hardened
reverse-proxy topology. Added the ``premium_user`` gate.
2. **Privileged-field denylist.** Without a denylist, an admin who
maps the wrong header to ``user_role`` (or whose reverse proxy
leaks the header from upstream user input) lets any caller send
``X-User-Role: proxy_admin`` and gain full admin access — Pydantic
coerces the string into the ``LitellmUserRoles.PROXY_ADMIN`` enum.
Mapping any field in ``PRIVILEGED_OAUTH2_PROXY_FIELDS``
(``user_role``, ``api_key``, ``token``, ``permissions``,
``allowed_routes``, budget/limit fields, ``metadata``) raises at
request time so the misconfiguration surfaces loudly rather than
as a silent privesc.
Operators who genuinely need a trusted upstream to assert one of
these privileged fields should switch to JWT auth (signature-validated)
rather than header-trust.
Tests:
- ``test_returns_auth_for_simple_user_id_mapping``: legitimate
identity-only mapping still works.
- ``test_rejects_when_not_premium``: open-source deployments get a
clear enterprise-feature error.
- ``test_refuses_to_map_privileged_fields``: parametrized over every
entry in the denylist — each is rejected at request time.
- ``test_user_role_header_forgery_attack_is_blocked``: end-to-end
shape of the GHSA-5c3m-qffq-4r9m attack; rejected before auth
object construction.
- ``test_safe_fields_still_pass_through``: documented usage
(``user_id``, ``user_email``, ``team_id``, ``models``) is
unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Use key-in-dict membership instead of truthy value lookup so explicitly
supplied empty/falsy payloads still trigger the permission check.
Adds parametrized regression coverage across all gated keys.
Two related changes to how the master-key auth path interacts with
downstream consumers of UserAPIKeyAuth.api_key:
1. The master-key auth branch in user_api_key_auth.py now sets
`valid_token.api_key` to a stable alias
(`LITELLM_PROXY_MASTER_KEY_ALIAS = "litellm_proxy_master_key"`) instead
of the raw master key. Downstream consumers — spend logging,
Prometheus metrics, audit trails, rate limiting, cost tracking — now
receive the alias instead of the master key (which they would
previously hash and propagate). Neither the raw master key nor its
hash flows past the auth layer.
2. `_is_master_key` in spend_tracking_utils.py is reduced to a strict
raw-only constant-time comparison. The hashed form is no longer
considered equivalent.
Side effects:
- The two hash-detection blocks in `get_logging_payload` are removed.
They were re-detecting the master key per spend-log write to swap in
the alias; that detection happens once at the auth layer now.
- The `disable_adding_master_key_hash_to_db` general setting becomes a
no-op. Operators can remove it from their config; existing config is
still accepted.
- Operator dashboards that filter Prometheus metrics by the master-key
hash will need to switch to the `api_key="litellm_proxy_master_key"`
label.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Updated instances of DualCache to UserApiKeyCache across multiple files to enhance cache handling for user API keys.
- Adjusted cache retrieval and storage methods to ensure proper serialization and deserialization of cached objects.
- Introduced a new UserApiKeyCache class to streamline caching logic and improve type safety.
- Updated relevant tests to reflect changes in caching behavior and ensure compatibility with the new cache implementation.
* fix(guardrails): apply team-level guardrails alongside global policy guardrails
Two bugs prevented team-direct guardrails from being automatically applied
when using a team-scoped API key:
1. Auth caching: `valid_token.team_metadata` was never refreshed from the
freshly-fetched team object at the "Check 6" step in
`_user_api_key_auth_builder`. Guardrails added to a team after the key
was first cached were therefore invisible to `move_guardrails_to_metadata`.
Fix: propagate `_team_obj.metadata` → `valid_token.team_metadata` after
every "Check 6" team fetch (user_api_key_auth.py).
2. Guardrail execution: `get_guardrail_from_metadata` checked
`data["litellm_metadata"]` before `data["metadata"]`. When a request
carried a non-empty `litellm_metadata` without a "guardrails" key, the
merged guardrail list written to `data["metadata"]` by
`move_guardrails_to_metadata` was shadowed and the guardrail received an
empty requested-guardrails list (custom_guardrail.py).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix merge conflict
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The asyncio.gather in `_run_centralized_common_checks` ran with
`return_exceptions=False` and a single bare `except HTTPException`
arm, so an HTTPException from any one fetch (the realistic case is a
404 from `get_team_object` when a token references a deleted team)
zeroed out the user, end-user, project, and global-spend contexts in
addition to falling back the team object. That silently skipped the
user budget, end-user budget, and project enforcement passes inside
`common_checks` for the unrelated contexts that had actually fetched
fine.
Switch to `return_exceptions=True` and apply per-fetch fallback
(matches the pre-refactor per-fetch try/except pattern in the builder):
- ProxyException / BudgetExceededError still propagate as authz failures.
- HTTPException on the team fetch reconstructs from the token; on the
other fetches it nulls only that one context.
- Successful fetches always reach `common_checks` intact.
Adds two unit tests covering the team-404 and user-404 cases to lock
the per-fetch isolation in. Drops the inaccurate `PROXY_ADMIN tokens
short-circuit` claim from the docstring — admin tokens still flow
through `common_checks`; admin status is only honored where the
underlying check exempts it.
The routes in `global_spend_tracking_routes` (e.g. /global/spend/report,
/global/spend/teams, /global/spend/keys) return spend aggregated across
every team, customer, and api_key in the proxy. They were included in
`internal_user_routes` and `internal_user_view_only_routes`, so non-admin
roles could read proxy-wide spend.
Drop them from both non-admin route lists. PROXY_ADMIN and
PROXY_ADMIN_VIEW_ONLY access is preserved through their existing branches
in route_checks.py, and the `get_spend_routes` permission opt-in
continues to grant access for keys that need it.
Updates two pre-existing test parametrizations whose expected results
flip from True to False, and adds parametrized coverage over every
route in `global_spend_tracking_routes` for: PROXY_ADMIN_VIEW_ONLY
allowed, INTERNAL_USER blocked, INTERNAL_USER_VIEW_ONLY blocked,
INTERNAL_USER + get_spend_routes permission allowed.
* fix(proxy): infer team from DB when JWT has no team and user has one team
- When team_id is unset after JWT auth but the user row has exactly one
team, set team_id, team_object, and team_membership from DB.
- Skip when zero or multiple teams (ambiguous).
- Add parametrized unit tests in test_handle_jwt.py.
Made-with: Cursor
* fix(proxy): JWT single-team DB fallback: catch errors, tests match get_team_object
- Wrap get_team_object + get_team_membership in one try/except; log and skip on failure (stale/missing team id no longer fails auth).
- Parametrize tests: HTTP 404/500, membership error; use side_effect not return_value=None for missing team row.
Made-with: Cursor
* refactor(jwt): extract single-team fallback into _resolve_single_team_fallback helper
Made-with: Cursor
User-configured pass-through endpoints with ``auth: false`` are
explicitly unauthenticated — the builder short-circuits at
check_api_key_for_custom_headers_or_pass_through_endpoints and returns
a fresh empty UserAPIKeyAuth() without an api_key, user_id, or role.
Pre-refactor, that empty token never reached common_checks. After the
centralization, it does — and common_checks rejects it as admin-only,
breaking every Langfuse / custom unauthenticated pass-through.
This is the same regression class as the public-routes one: a
builder fast-path whose return value cannot survive common_checks.
Honor the same contract here — when the matched endpoint config has
auth != True, skip the centralized gate. auth=True endpoints still
run the full gate (covered by a companion test).
No security regression: ``auth: false`` is the operator's explicit
opt-out from LiteLLM auth on this path. The original commit closed
seven authenticated bypasses; this exemption applies only to a path
the operator has already declared unauthenticated.
Two regressions introduced by 3737d6a1f3 (centralized common_checks):
1. Public routes (e.g. /health/readiness, /metrics) are exempted by the
builder fast-path but the wrapper then ran common_checks on the
synthetic INTERNAL_USER_VIEW_ONLY token, which has no user_id, no
team, no scopes — so common_checks rejected the request as admin-
only. This broke every k8s readiness probe when master_key is set
(helm chart job confirmed: pod never goes Ready, service has no
endpoints).
2. The admin user_object synthesis only triggered when
user_object is None. After any team-creation flow runs, the row
for litellm_proxy_admin_name (default "default_user_id") exists
in litellm_usertable with the default user_role=internal_user.
get_user_object then returned that row, the synthesis was skipped,
and master_key requests were demoted to internal_user — failing
/team/update, /team/block, etc. The token's user_role is the
source of truth for these paths (set inside the authenticated
master_key / JWT-admin builders); a stale DB row must not override
it.
Fix:
- Short-circuit _run_centralized_common_checks for routes already in
LiteLLMRoutes.public_routes (or general_settings.public_routes).
Same exemption surface the builder already trusts.
- When the token's user_role is PROXY_ADMIN, force the synthesized
admin user_object regardless of what get_user_object returned.
Preserves the spend value from the DB row.
Neither change reopens any of the seven bypasses the original commit
closed: OAuth2, JWT non-admin, DB-fallback, /user/auth, pass-through
headers, etc., still go through the gate. Only paths that were
already admin or already public skip it.
Adds two regression tests:
- test_centralized_common_checks_skips_public_routes
- test_centralized_common_checks_master_key_admin_overrides_db_user_role
Extract the admin team-header attachment into a helper so
auth_builder stays under the 50-statement lint threshold; apply
black formatting to the two files flagged on the prior commit.
No behavior change.
Scope the header-driven team fetch to LLM API routes so admin
management routes keep the pre-existing bypass behavior (no
phantom teams, no 404s on mgmt calls). Team context is threaded
onto UserAPIKeyAuth so spend logs, rate limits, and team_models
attribution are correctly applied when admins act on behalf of
a team via x-litellm-team-id.
MCP server CRUD endpoints (/v1/mcp/server*) were bundled with MCP
tool-call / passthrough endpoints under llm_api_routes, so setting
DISABLE_LLM_API_ENDPOINTS=true on admin-only nodes also blocked the
Admin UI from listing, adding, or attaching MCP servers.
Separate mcp_inference_routes (data-plane, gated by
DISABLE_LLM_API_ENDPOINTS) from mcp_management_routes (control-plane,
gated by DISABLE_ADMIN_ENDPOINTS). Keep mcp_routes as a union for
backward compat with allowed_routes=["mcp_routes"] virtual key configs.
Upgrade is_management_route to pattern-aware matching so
/v1/mcp/server/{path:path} resolves for concrete IDs.
The HTTPException arm in _run_centralized_common_checks assumed the
exception came from the team-object fetch, but asyncio.gather raises
the first exception from any of the five gathered coroutines. If
get_user_object / get_project_object / get_end_user_object raises
HTTPException on a token with team_id=None, the assert inside
_team_obj_from_token fires and the outer auth-exception handler
mishandles it.
Guard on team_id before calling _team_obj_from_token and default
team_object to None otherwise. (Greptile P1.)
get_end_user_object raises litellm.BudgetExceededError internally
when the end-user is over budget. The previous _safe_fetch in the
centralized gate swallowed it and returned None, which caused
common_checks to see end_user_object=None and skip the budget check
entirely — silently bypassing end-user budget enforcement.
Add BudgetExceededError to the re-raise list alongside HTTPException
and ProxyException (reported by Veria AI).
- Narrow _team_obj_from_token to require non-None team_id so mypy
passes.
- Preserve the no-auth dev-mode contract for deployments with
master_key unset AND no JWT/OAuth2 configured — the gate
short-circuits only in that specific combination. JWT or OAuth2
deployments without master_key still run the centralized authz.
- is_database_connection_error now enumerates data-layer PrismaError
subclasses (DataError, UniqueViolationError, ForeignKeyViolationError,
MissingRequiredValueError, RawQueryError, TableNotFoundError,
RecordNotFoundError) as False, and maps everything else (bare
PrismaError, connectivity subclasses, DB_CONNECTION_ERROR_TYPES) to
True. Known-safe-to-propagate errors don't trigger HA fallback;
unknown / generic PrismaError still falls back to preserve legacy
503 behavior.
- Update test_handle_authentication_error_db_unavailable_connectivity
to include PrismaError in the fallback list.
Multiple paths through _user_api_key_auth_builder returned a
UserAPIKeyAuth without running common_checks(): OAuth2 token validation,
OAuth2 proxy header hook, JWT admin shortcut, master_key path,
pass-through custom headers, the /user/auth route, and the
allow_requests_on_db_unavailable fallback. An operator-configured key
model-access list, max_budget, team_blocked flag, or team model scope
was therefore silently skipped on those paths. The HA-fallback token
was worse: it was a full proxy-admin synthetic, so a DB outage granted
full admin to every caller.
Fix three root causes (VERIA-18):
1. Centralize common_checks in the user_api_key_auth wrapper. The
builder paths no longer call it; the wrapper runs it once after the
builder returns, for every path. Introduces _run_centralized_common_checks
which gathers team/user/project/end_user/global_spend context in
parallel via asyncio.gather. Preserves the existing
custom_auth_run_common_checks opt-out for custom-auth deployments.
2. Narrow is_database_connection_error — drop the blanket PrismaError
catch that routed data-layer errors (UniqueViolationError, etc.)
into the HA fallback. Only real connectivity failures plus the
no_db_connection marker now qualify.
3. DB-unavailable fallback issues an INTERNAL_USER token with user_id
DB_UNAVAILABLE_FALLBACK_USER_ID instead of proxy-admin. An outage
can no longer escalate an anonymous caller.
JWT admin / master_key tokens still grant admin via a synthesized
admin user_object (so non_proxy_admin_allowed_routes_check in
common_checks recognizes them); other common_checks branches
(team_blocked, team_model_access) now apply uniformly.
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Has been cancelled
test_virtual_key_max_budget_alert_check_per_key_overrides_global asserted
override semantics but the implementation does additive merge. Renamed test
and updated assertion to match: per-key and global thresholds are unioned,
not replaced.
- Guard empty recipients in _handle_multi_threshold_max_budget_alert:
log warning and skip instead of falling through to old path error loop
- Widen max_budget_alert_emails type to Dict[str, Union[str, List[str]]]
to match _parse_email_list runtime behavior (accepts comma-separated strings)
- Pre-filter asyncio.create_task with min threshold check to avoid
unnecessary task allocation on every request when spend is below
all configured thresholds
- Add `default_key_max_budget_alert_emails` litellm_settings config as
global fallback for all virtual keys (per-key metadata takes priority)
- Fix crash when key has no user_id/user_email by passing recipient email
to _get_email_params (same pattern as team soft budget path)
- Use owner email for greeting, falling back to key_alias or token
- Rename setting from default_max_budget_alert_emails to
default_key_max_budget_alert_emails for clarity
Users can set metadata.max_budget_alert_emails as a JSON map of threshold
percentages to email recipients on virtual keys. When configured, the email
handler loops over each threshold, checks per-threshold dedup cache, and
sends to the configured recipients (auto-including the key owner's email).
When no map is set, the existing single 80% threshold behavior is preserved
unchanged. Teams support is out of scope for this v0.
Close three variant bypasses adjacent to VERIA-28 found during post-fix
variant audit:
1. _guardrail_modification_check had the same isinstance(dict) bypass
Veria-AI just flagged on the pre-call strip. A caller sending
`{"metadata": "{…}"}` as a JSON-encoded string (multipart/form-data
or extra_body) skipped the guard, got parsed to dict downstream, and
reached guardrail logic with bypass flags intact. Coerce strings via
safe_json_loads before evaluating.
2. The allow_client_tags strip only covered body metadata.tags and
litellm_metadata.tags — caller-supplied tags arriving via the
x-litellm-tags header or root-level data["tags"] bypassed it. Gate
add_request_tag_to_metadata's result on the same flag.
3. requester_metadata was deepcopied BEFORE the strip, so attacker
injections (user_api_key_metadata shadows, disallowed tags,
_pipeline_managed_guardrails) persisted in the snapshot. The PANW
guardrail (and any future consumer) trusting requester_metadata
would see forged values. Move the deepcopy to after the strip.
Regression tests added for each.
Per VERIA-28's secondary recommendation. The existing check only gated
metadata.guardrails. User-supplied values for disable_global_guardrails
(plural and the original singular typo variant) and opted_out_global_guardrails
are already silently ignored by _get_admin_metadata at read time, but the
silent-ignore makes diagnosis confusing and relies on one specific read
site catching them.
Reject at auth time with a 403 when any of:
- guardrails list (existing)
- disable_global_guardrails (new)
- disable_global_guardrail (new — historical singular-key variant)
- opted_out_global_guardrails (new)
are present in metadata, litellm_metadata, or at the request root, and the
caller's team lacks can_modify_guardrails. Defense in depth: the strip at
the pre-call layer still runs; this check fails loudly one layer earlier
so operators see an explicit 403 rather than a silent-ignore.
