Tag each key SCIM blocks with metadata.scim_blocked=True. On reactivation
unblock only those keys, leaving keys an admin blocked for unrelated
reasons untouched.
If atomic_check_and_increment_by_n returns overall_code=OVER_LIMIT but no
status entry matches a descriptor key the dynamic limiter dispatcher knows
how to translate into a 429 (`model_saturation_check` or `priority_model`),
the for-loop previously exited cleanly and execution fell through to the
priority-tracking increment + the data["litellm_proxy_rate_limit_response"]
write — silently admitting an over-limit request.
This is the fail-open path a future contributor would hit by wiring a new
descriptor type into enforced_descriptors without updating the dispatcher.
Refuse the request with a generic 429 carrying the offending descriptor
metadata so the operator can see what slipped past, and emit an error log
to surface the wiring gap.
Adds a regression test (test_dynamic_rate_limiter_v3_fails_closed_on_unknown_descriptor)
that drives the limiter with a synthetic OVER_LIMIT response carrying an
unrecognized descriptor_key and asserts a 429 is raised.
Tests: 65 passed (1 skipped), 0 regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: add expected_base_url origin check to get_litellm_gateway_api_key
* fix: scope get_stored_api_key and save base_url on login
* fix: pass base_url to get_stored_api_key in CLI entrypoint
* fix: scope ProxyClient stored key to base_url
* test: add expected_base_url coverage for get_stored_api_key
* fix: initialize self.http with resolved api_key not raw param
* fix: black formatting in client.py and test_auth_commands.py
- Lua script now reads time via redis.call('TIME') instead of a client-supplied
timestamp. Prevents window-reset divergence across replicas with skewed
wall-clocks, which could otherwise reopen the cross-replica TOCTOU window.
- Per-descriptor window_size is now plumbed through both the Lua ARGV layout
and the in-memory fallback. Previously the in-memory path used the global
self.window_size while Lua honored the per-descriptor override, so a
descriptor with a custom window would be enforced inconsistently between
Redis-available and Redis-unavailable code paths.
- Lua-failure fallback path now logs at error severity and explicitly
documents the in-memory ↔ Redis state divergence risk so operators can
alert on it. Prior `warning` log understated the impact.
- Coarse-granularity lock is now documented inline with the conditions under
which a per-descriptor sharded lock would be worth introducing.
- New regression test: zero-token batch consumes RPM only and is properly
capped by the RPM ceiling (validates the asymmetric quota path that arises
from `inc_amount <= 0: continue`).
Tests: 64 passed (1 skipped), 0 regressions. Multi-instance Redis loadtest
re-verified: chat 20/80 success @ RPM=20, batches 3/20 @ TPM=200.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: scope /health response to caller's models and tidy display fields
Two small consistency changes to the /health response:
1. health_endpoint() now narrows _llm_model_list to deployments whose
model_name is in user_api_key_dict.models, matching how other model
listing endpoints already scope their output. The same narrowing applies
to the cached health_check_results dict when background_health_checks is
enabled, via a new _filter_health_check_results_by_model_ids helper.
2. ILLEGAL_DISPLAY_PARAMS in health_check.py picks up api_base and
api_version, which are provider routing fields and not part of the
health response shape.
Tests in tests/test_litellm/proxy/health_endpoints/test_health_endpoints.py
pin both behaviors so future changes do not widen the response shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* address greptile review feedback (greploop iteration 1)
- tests: extend background-cache test with model_id on cached entries plus
positive assertions that model-a's deployment is the one returned, so
the test is no longer satisfied by an empty result.
- _health_endpoints.py: add a verbose_proxy_logger.debug line when a scoped
key has accessible model_names but the matching deployments have no
model_info.id, so the empty cache-result case is observable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* make background-cache test's non-vacuity explicit
Restructure test_health_endpoint_filters_background_cache_by_user_access
so the assertions positively pin the post-scoping result (one entry,
model_id == "id-a", api_base == https://example-a.test) and add fixture
sanity checks that confirm the source cache had two entries and every
cached entry carries a model_id.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* gate api_base in /health response on proxy-admin role
Replace the blanket strip of api_base / api_version with a role-aware
post-processor:
- api_base is now left in the cleaned per-deployment dict that
_clean_endpoint_data produces (api_version stays in the denylist).
- health_endpoint() removes api_base from each endpoint entry before
returning when the caller's user_role is not PROXY_ADMIN /
PROXY_ADMIN_VIEW_ONLY. The strip uses a copy so the shared
health_check_results cache still carries api_base for subsequent
admin reads.
Net effect: a proxy admin can still see which Vertex region or Azure
resource is healthy in the /health output, while non-admin keys (and
read-only keys) only see model / model_id / status fields.
Tests:
- test_health_endpoint_admin_sees_api_base_non_admin_does_not pins both
branches and verifies the cache is not mutated.
- test_clean_endpoint_data_strips_credentials_but_keeps_api_base
replaces the previous mask/drop tests now that the cleaning helper
no longer touches api_base.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* address review feedback: api_version symmetry, missing-id warnings, deprecation header
Three blockers raised in review:
1. api_version asymmetry — api_base was role-gated for proxy admins, but
api_version was unconditionally stripped via ILLEGAL_DISPLAY_PARAMS.
Move api_version out of the credential denylist and into a new
ADMIN_ONLY_HEALTH_DISPLAY_PARAMS tuple alongside api_base, so admins
keep both routing fields and non-admins lose both. Useful for telling
apart Vertex regions or Azure api-versions from the /health response.
2. Silent empty results when scoped key's deployments lack model_info.id —
raise the existing log from .debug to .warning, and add a structured
"warnings" field to the response so the caller can distinguish "no
deployments configured" from "deployments excluded due to missing
model_info.id".
3. Migration signal for the api_base / api_version removal — when a
non-admin caller hits /health, set a "Litellm-Health-Field-Notice"
response header so existing dashboards or scripts that parsed those
fields can detect the change programmatically rather than silently
seeing absent keys.
Tests adjusted: existing background-cache test injects a Response stub,
admin-vs-non-admin test now asserts both api_base and api_version are
gated and asserts the notice header. New test covers the warnings field
when a scoped key's deployments are missing model_info.id.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* defensive copies + clarifying comments in /health filter
- _filter_health_check_results_by_model_ids now shallow-copies each
retained endpoint dict before returning. The shared module-level
health_check_results cache should never be mutated by downstream
transforms, even though _strip_admin_only_fields_from_health_result
already builds new dicts today.
- Document the live (model_name) vs cache (model_id) scoping asymmetry
so future readers do not have to derive it from the warnings field.
- Document why _PROXY_ADMIN_ROLES includes PROXY_ADMIN_VIEW_ONLY (read-
only operators need routing fields to diagnose health).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Greptile follow-ups on the prior commit:
- (P1) ``is_request_body_safe`` recursed into ``litellm_embedding_config``
with no depth bound, so a request body 1000 levels deep could exhaust
Python's call stack and surface a 500 ``RecursionError``. Refactored
the check to be iterative (single-level descent into a fixed list of
nested-config keys) and extracted the per-dict banned-param scan into
a helper that's shared between the root and the nested call sites.
Also fixes the ``recursive_detector`` CI job that was triggered by
the recursive-by-name pattern.
- (P2) ``assert_same_origin`` error messages identified the mismatching
component but echoed the ``expected`` host and the candidate
hostname back to the caller. In the SSRF threat model the caller is
the attacker, so reflecting that information was a secondary leak of
operator infrastructure. Messages now identify only *which*
component mismatched (scheme / host / port) without naming names.
- (P2) ``_NESTED_CONFIG_KEYS`` was defined after the function that used
it. Hoisted the constant (and the new ``_BANNED_REQUEST_BODY_PARAMS``
tuple) above the function for readability.
Adds a 1000-level-deep nested config test that asserts no
``RecursionError`` and a hostname-leak test that asserts no operator
host appears in the rejection message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two SSRF findings were OPEN with no in-flight fix; both are closed
now using narrow defenses that key off existing trust boundaries.
VERIA-6 (Milvus ``litellm_embedding_config``):
``is_request_body_safe`` already blocks ``api_base`` / ``api_key`` /
``langfuse_host`` / ``s3_endpoint_url`` / etc. at the *root* of the
request body, gated by an admin opt-in (``allow_client_side_credentials``
or per-deployment ``configurable_clientside_auth_params``). The bug is
that the Milvus vector-store transformer unpacks
``litellm_embedding_config`` into ``litellm.embedding(**embedding_config)``,
so a caller can smuggle the same banned params in via nesting and bypass
the check. Fix: ``is_request_body_safe`` now recurses into a known list
of nested-config dicts (``litellm_embedding_config`` for now) and applies
the same banned-param check with the same admin opt-in. Admin-side
vector-store config flows through ``litellm_params`` rather than the
request body, so it's unaffected.
VERIA-51 (polling URLs returned by upstream APIs):
Azure DALL-E 2, Azure Document Intelligence, and Black Forest Labs
all blindly fetched a polling URL returned by the upstream and
attached the operator's API key to the request. A compromised upstream
or a future API contract change could redirect credentials anywhere.
New ``url_utils.assert_same_origin(candidate, expected)`` helper checks
scheme, host (case-insensitive), and port (with default-port
normalization). Applied at all five polling sites: Azure DALL-E
sync+async, Azure DI sync+async, BFL image generation sync+async, BFL
image edit sync+async. Cross-origin polling URLs now raise rather than
forward credentials. The Azure DALL-E ``Expected 'status' in response``
exception no longer reflects the raw response body — that path turned
Blind SSRF into Full-Read SSRF for the limited window before the
origin check fully closed it.
Tests: 7 ``assert_same_origin`` unit tests, 6 ``is_request_body_safe``
nested-config tests, 5 polling-site rejection tests + 1 same-origin
sanity check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CodeQL flagged the previous ``from litellm.types.router import
MockRouterTestingParams`` at module top-level — ``litellm.types.router``
indirectly imports back into proxy modules, so the dataclass may not
exist yet when ``route_llm_request`` is being imported.
Hardcode the three flag names instead, with a guard test
(``test_mock_testing_kwarg_names_matches_dataclass``) that asserts the
hardcoded list matches ``MockRouterTestingParams.fields`` so drift is
caught at test time rather than missed in production.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes that together prevent a caller from smuggling unauthorized
models past the API key's allowlist via per-request router overrides.
1. ``_enforce_key_and_fallback_model_access``: also walk fallback models
nested inside ``router_settings_override.fallbacks`` /
``context_window_fallbacks`` / ``content_policy_fallbacks``.
``route_llm_request.py`` promotes those to per-request kwargs after
auth, so without this they bypassed the model allowlist entirely.
New ``iter_router_fallback_model_names`` helper extracts leaf names
from both the simple top-level shape (str | {"model": str}) and the
nested router-config shape ({primary: [fallbacks]}). The two fallback
validation loops are unified — every name (top-level + override) is
deduplicated and validated once via ``can_key_call_model`` +
``is_valid_fallback_model``.
2. ``route_request``: strip router-internal ``mock_testing_*`` flags
from user-supplied data. These are testing-only flags that
deterministically force the router into fallback logic by raising a
synthetic ``InternalServerError`` etc. Combined with override
fallbacks they made the smuggling path trivially exploitable. Test
code that calls the router directly bypasses the strip and is
unaffected. The strip list is derived from ``MockRouterTestingParams``
so a new ``mock_testing_*`` flag added to that dataclass is
automatically covered.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /v2/model/info endpoint (used by the UI's Models + Endpoints page)
was not resolving access group names when filtering models by team.
When a team has models: ["Group-A"] where "Group-A" is an access group,
_filter_models_by_team_id() passed it as a literal model name to
get_model_list(), which found no deployments with that name. This caused
the UI to show all models instead of only team-accessible ones.
The request-time auth path (model_in_access_group in auth_checks.py)
correctly resolves access groups via get_model_access_groups(). This
fix applies the same resolution in _filter_models_by_team_id() for both
the in-memory router lookup and the database fallback query.
Tests added:
- test_filter_resolves_access_group_names
- test_filter_resolves_mix_of_access_groups_and_literal_names
- test_filter_excludes_models_from_other_access_group
- test_filter_db_fallback_receives_resolved_model_names
Greptile P1: Aim's ``_anonymize_request`` and Lakera v2's mask-PII path
both wrote redacted content only to ``data["messages"]``. The Responses
API backend reads ``data["input"]``, so when a request arrived via
``/v1/responses`` with a plain string ``input`` the hook would update
``messages`` (which the backend ignores) and leave ``input`` carrying
the original unredacted text. Net effect: anonymize/mask silently passed
PII through to the LLM.
Add ``apply_redacted_messages_back`` to ``_content_utils`` — it writes
the redacted messages back to ``data["messages"]`` AND, when present,
re-flattens the redacted content into ``data["input"]``. Aim and
Lakera v2 now route their mask writeback through this helper. List
``input`` (multimodal) is still handled by the upstream
block-on-multimodal guard.
Adds unit tests for the helper and regression tests asserting
``data["input"]`` is redacted for both hooks on Responses-API string
input.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply organization object_permission as a ceiling on allowed MCP servers
and tool permissions, consistent with vector store org checks.
Includes unit tests for org ceiling, intersection, and tool filtering.
Made-with: Cursor
Two more in-place rewrite paths exhibit the same regression as Lakera v2:
overwriting ``data["messages"]`` with text-only redacted versions silently
strips image/audio parts from multimodal requests.
- ``LassoGuardrail._run_lasso_guardrail``: when ``mask=True`` AND input
is multimodal/Responses-API list, fall back to the classify endpoint
(which raises on BLOCK actions but never overwrites the payload).
- ``AimGuardrail._anonymize_request``: when input is multimodal, raise
the standard 400 instead of replacing ``data["messages"]`` with the
text-only ``redacted_chat`` from Aim. The error message tells the
user to either send plain string content or rely on block-mode.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mask-in-place uses the offsets that Lakera returns for the inspection
payload. ``build_inspection_messages`` flattens multimodal content into
joined text before sending to Lakera, so the offsets refer to the
flattened representation. Writing those offsets back via
``_mask_pii_in_messages`` and overwriting ``data["messages"]`` would
silently strip image/audio parts from the original request — that is a
real functional regression for Lakera + mask mode + multimodal input.
Detect multimodal input (any list-format ``content`` or non-string
``data["input"]``) up front and skip the mask-in-place branch in that
case. The hook then falls into the standard block-on-detect path so PII
is still blocked but the multimodal payload is never silently rewritten.
Per-part masking that preserves multimodal structure is the right
long-term fix; tracking that as a follow-up.
Also: add ``has_non_string_content`` to ``_content_utils`` (with tests)
and a regression test that asserts multimodal+PII raises an HTTPException
instead of returning a flattened request body.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Greptile P2 follow-ups on _content_utils.py:
- Drop unreachable ``_resolve_messages``. The new
``_iter_inspection_messages`` walks ``messages`` AND ``input``
independently; leaving the old fallback-only variant around invited a
future maintainer to wire it back up and silently narrow coverage.
- Rename ``iter_user_text`` → ``iter_message_text``. The helper walks
every role (user, assistant, system); the old name implied user-turn
content only. Callers and tests updated.
- Close mixed-list coverage gap. When ``data["input"]`` was a list
mixing content-part dicts and bare strings, ``iter_message_text`` and
``build_inspection_messages`` only saw the dict parts while
``walk_user_text`` already inspected both. ``_iter_text_parts_in_content``
now treats bare strings inside a content list as text fragments, so
read and write helpers agree on coverage.
Adds two regression tests for the mixed-list shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Several guardrail hooks short-circuit when ``message.content`` is a list
or when the request uses the Responses-API ``input`` field instead of
``messages``. Centralise the content-walking logic in a shared helper and
update the affected hooks so list-format and Responses-API payloads no
longer skip inspection.
Also: Aim's ``async_post_call_success_hook`` now inspects every choice
(via ``asyncio.gather``) instead of only ``choices[0]`` — the prior
behaviour let ``n>1`` callers hide content in subsequent completions.
Hooks updated to use the new helper:
- aim, lakera_ai_v2, lasso (post a synthesised messages list to a remote
guardrail service)
- azure_content_safety, ibm_detector, banned_keywords, openai_moderation,
google_text_moderation (iterate text fragments locally)
- secret_detection (walk-and-rewrite to redact in place)
Drive-by fix: the legacy ``data["prompt"]`` list-handling path in
secret_detection rebound the loop variable instead of mutating the list,
leaving secrets unredacted on text-completion calls; corrected to index
back into the list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Greptile P2 follow-up: when a litellm_configoverrides row exists with a
NULL config_value (e.g. an earlier failed write left a stub), the audit
action was mislabeled "created" because we keyed off existing_decrypted
(which is only set when config_value is non-null). Key off existing_record
instead — a row is a row regardless of its value.
Also hoist asyncio + patch imports to module top in the test file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
