Streaming and pass-through requests could be logged with $0 cost or dropped from
SpendLogs entirely while the upstream provider still billed every token. This
closes the leak paths not already covered by #30160, #30787 and #30788.
- Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and
async). Large agentic tool-use / thinking streams can make assembly re-raise
as APIError from inside the except-StopIteration handler, where the sibling
except does not catch it, so it escaped __next__/__anext__ and dropped the
request; recover best-effort usage from the raw chunks instead
- Add a usage-only fallback for Anthropic streaming pass-through: when
stream_chunk_builder returns None or raises, rebuild usage from the
message_start / message_delta SSE events via AnthropicConfig.calculate_usage so
cache, web-search and geo tokens are priced instead of left at $0
- Decode buffered pass-through bytes with errors="replace" so a stream cut
mid-multibyte-sequence still logs the usage events already received
- Record response_cost into model_call_details on the pass-through success path
(it is read from there, not from kwargs), matching the gemini/cohere/openai
handlers
- Name the key (alias + masked key) in the virtual-key BudgetExceededError so
operators don't have to reverse-map spend back to a key
(cherry picked from commit b24b964e04)
Prerequisite for #31035 on this line, and a latent-bug fix in its own right.
#31035's usage-only fallback builds server_tool_use as a dict and prices it via
AnthropicConfig.calculate_usage, whose Usage(**model_dump()) round-trip drops it
back to a plain dict; without this Usage.__init__ coercion the recovered-cost path
does attribute access on a dict and raises, so #31035's web-search/server-tool
cost recovery is dead on arrival here. The same round-trip already affected the
pre-existing ChunkProcessor.calculate_usage path: every production consumer on
this line (litellm/llms/anthropic/cost_calculation.py,
litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py) reads
usage.server_tool_use.web_search_requests by attribute, so a dict there is a
latent AttributeError on streaming web-search cost. The coercion makes the value
a ServerToolUse, which all consumers expect.
Also updates the one test that pinned the old dict-subscript shape
(test_stream_chunk_builder_anthropic_web_search) to assert the ServerToolUse type
and attribute access, matching staging.
Content-verified present on litellm_internal_staging via aggregator
f49707bc66 (fix(otel) #30257), which carries both the coercion and the test
assertion update; this restores only those, not the rest of that aggregator. The
coercion also shipped to stable/1.89.x as 24e30b551f.
(cherry picked from commit 24e30b551f)
A streaming request that breaks mid-flight, for example on a mid-stream read
timeout, still bills the provider for the chunks already delivered, yet the proxy
recorded that interrupted request as a zero-spend failure. An earlier revision
logged the recovered partial usage through the success path, which mislabeled a
failed request as a success and produced a misleading spend row
This recovers the partial usage where the failure is actually logged. The
streaming handler assembles the usage from the chunks seen so far and stashes it,
with its cost, on the logging object before firing the failure handlers. The
proxy failure hook lifts that usage and cost onto request_data before the
non-serialisable logging object is popped, and the spend-log writer records the
real partial spend on the failure row instead of a hardcoded zero;
get_logging_payload honors the recovered usage for the token columns and
_failure_handler_helper_fn preserves the recovered cost so the non-DB failure
loggers stay consistent
A request that recovers via a successful fallback is unaffected: the failure hook
only fires when the whole request fails, so the fallback's combined-usage success
row stays the single source of truth and there is no double counting
Resolves LIT-3825
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
(cherry picked from commit 4847fa5dd5)
* fix(integrations): cap Anthropic cache_control injection at 4 blocks
Respect Anthropic's 4 cache_control breakpoint limit by counting client-supplied blocks, skipping messages that already carry cache_control, and stopping further auto-injection once the limit is reached.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(integrations): reserve cache slot for tool_config and short-circuit cap
Address review feedback on the cache_control cap: break out of the injection loop before resolving target indices once the limit is reached, and reserve one of the four breakpoint slots when a tool_config injection point is present so the cachePoint appended by the Bedrock transform does not push the total past Anthropic's limit.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit fc9d789d24)
Targeted subset of staging commit cfcdf8714a (#30202): only the
anthropic_passthrough_logging_handler.py hardening hunks and their four
tests are taken; the rest of that staging batch is intentionally excluded.
(cherry picked from commit cfcdf8714a)
(cherry picked from commit 973c7eb8d6)
* fix(proxy): expose Prisma idle/connect timeout + extra DB URL params
Operators have reported large numbers of idle Prisma connections that
never get closed. The proxy already forwards `connection_limit` and
`pool_timeout` to the DATABASE_URL, but had no knob for capping idle
or slow connections. Add three new `general_settings` keys that thread
through to the DATABASE_URL / DIRECT_URL query string:
- `database_connect_timeout` -> Prisma `connect_timeout`
- `database_socket_timeout` -> Prisma `socket_timeout` (the main
knob for closing idle connections from the LiteLLM side)
- `database_extra_connection_params` -> untyped passthrough dict for
any other Prisma URL param (`pgbouncer`, `statement_cache_size`,
`sslmode`, ...); keys here override LiteLLM defaults.
Refactors the duplicated DATABASE_URL/DIRECT_URL param dicts into a
single `_build_db_connection_url_params` helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update litellm/proxy/proxy_cli.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
(cherry picked from commit 2f9ac77b24)
* fix(router): use forwarded model_id for native Azure container IDs in _init_containers_api_endpoints
Azure code-interpreter containers return provider-native IDs (cntr_ + hex)
that carry no LiteLLM routing payload, so _decode_container_id returns
model_id=None. The router was falling through to call the handler directly,
bypassing _ageneric_api_call_with_fallbacks and leaving api_base=None for
Azure deployments. Fall back to the model_id forwarded from the proxy
ownership check so deployment credentials are always applied.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): strip /openai/responses path from api_base in AzureContainerConfig.get_complete_url
When a deployment's api_base is the responses endpoint URL
(e.g. .../openai/responses?api-version=...), AzureContainerConfig was
appending /openai/containers on top of it, producing the broken path
.../openai/responses/openai/containers. Azure returns 404 for that URL
while the correct path is .../openai/containers.
Strip any /openai/responses suffix from api_base before constructing
the containers URL so the resource root is always used as the starting point.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): prefer api-version from api_base URL over deployment's api_version
The deployment's api_version (e.g. 2024-08-01-preview) targets the chat/responses
API and is too old for the containers API, which requires 2025-04-01-preview.
The responses endpoint api_base already carries the correct api-version in its
query string. Extract it and use it for the containers URL, overriding the
stale deployment-level version.
Fixes DELETE and file-upload operations returning 404 due to wrong api-version.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(containers): pass params=None instead of params={} to httpx to preserve api-version
httpx erases a URL's query-string when params={} (empty dict) is passed,
silently stripping ?api-version=2025-04-01-preview from every container
POST/DELETE request. Azure's GET endpoints tolerate a missing api-version;
POST (upload) and DELETE are strict, so those returned 404.
Fix: use `params or None` in container_handler._async_handle and
llm_http_handler.async_container_delete_handler (and all sibling container
handlers) so that an empty params dict falls back to None, leaving httpx to
preserve the URL's existing query string intact.
Adds a regression test that directly documents the httpx behaviour.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): remove elif model_id branch from _init_containers_api_endpoints
Two reviewer findings addressed:
1. Truncated comment on the model_id fallback line — now complete.
2. Security: the elif branch that fired when container_id was absent allowed
any authenticated caller to supply model_id in a POST /v1/containers body
and route the request through an arbitrary deployment UUID, bypassing the
model-level access checks that only validate `model`. Removed the elif
branch; operations without container_id (create, list) route by the
caller-supplied `model` field as before. model_id forwarding is kept only
inside the container_id block, where the proxy ownership check has already
validated the container before forwarding the deployment ID.
Adds a regression test pinning the security boundary: no-container-id path
calls original_function directly even when model_id is in kwargs.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(containers): validate proxy-to-router model_id forwarding for managed IDs
Add test_regression_get_container_forwarding_params_sets_model_id_for_managed_id
to verify that get_container_forwarding_params (the proxy-side half of the Azure
routing fix) correctly extracts and forwards model_id from a LiteLLM-managed
encoded container ID.
This closes the gap identified by Greptile P1: the previous regression test
only injected model_id as a direct kwarg, validating the router in isolation.
The new test exercises the actual proxy-to-router data flow through
ownership.get_container_forwarding_params, confirming that kwargs["model_id"]
is populated before _init_containers_api_endpoints is reached.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): tighten endpoint-path strip to endswith match
Use path.endswith() instead of path.find() for _AZURE_ENDPOINT_PATHS so
the suffix strip only fires when api_base actually ends with one of the
endpoint-specific path suffixes. This is the more precise check greptile
flagged on the original find()-based implementation.
* Fix sync container handler to preserve URL query string
Mirror the async path fix: pass None instead of an empty params dict so
httpx does not strip the URL's existing query string (e.g.
?api-version=...), which is required for Azure container routing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(azure-containers): strip trailing slash before endpoint suffix match
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(containers): recover model_id from stored encoded id for native Azure container IDs
get_container_forwarding_params previously only set model_id when the
user-supplied container_id was a LiteLLM-managed encoded id. For native
upstream IDs (e.g. Azure 'cntr_<hex>') the decode fails and model_id was
never forwarded — making the router-side fallback in
_init_containers_api_endpoints unreachable in production.
Fall back to the stored 'unified_object_id' on the ownership row, which
is the encoded form captured at create time when the router selected a
specific deployment. Decoding that yields the deployment model_id and
restores router-based credential application (api_base, api_key) for
retrieve/delete and container-file operations on native IDs.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
(cherry picked from commit 7f563b2593)
* fix(proxy): authorize batch files using upload target_model_names (LIT-3593)
After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)
Restores the reverse-lookup for the JSONL body.model fallback path so that
legacy/pre-target_model_names managed files still map stripped provider IDs
back to proxy aliases before auth. Also cleans up redundant `or None`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)"
This reverts commit 30d2e96f77.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit 2cd7e87485)
Capture user_id and extra_info from metadata or litellm_metadata. The single-bag read dropped identity whenever a request carried a present litellm_metadata field (null or a user-supplied dict), since /chat/completions routes the authenticated identity into metadata while the guardrail read litellm_metadata first
(cherry picked from commit 1bbaf1c39d)
* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI
Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context,
128K output, adaptive thinking only) on the Anthropic API, Bedrock
converse (base, global, and us/eu geo inference profiles at the 10%
regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which
serves Fable 5 with the full 1M context window unlike Opus 4.8).
Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the
model in the setup wizard, and extends the reasoning effort e2e grid.
The Bedrock, Vertex, and Azure grid cells carry fail_reason markers
until the CI accounts are provisioned: Bedrock needs the provider data
sharing opt-in Fable 5 requires, and the Foundry resource needs a
claude-fable-5 deployment.
The first-party entry carries provider_specific_entry {us: 1.1} for the
inference_geo premium and deliberately no fast multiplier since Fable 5
has no fast mode.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Drop removed sampling params for Claude 4.7+ when drop_params is set
Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects
top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was
forwarding them even with drop_params enabled because the Anthropic and
Bedrock converse transformations passed temperature/top_p through
unconditionally.
Mirror the GPT-5/o-series handling: temperature=1 still passes through,
other values and any top_p are dropped when drop_params is set, and
without drop_params a clean client-side UnsupportedParamsError tells the
caller how to opt in, instead of surfacing the raw provider error.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Drive sampling param gating from the cost map and cover top_k
Greptile review follow-ups on the sampling param fix: the restriction for
Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false
on every affected cost map entry (perplexity excluded; that route is
OpenAI-compatible and maps sampling params upstream) and read back through
a tri-state map lookup, keeping the name check only as a fallback for
provider-routed ids whose hosted map entries predate the flag, the same
layering supports_adaptive_thinking uses. top_k bypasses map_openai_params
as a provider-specific kwarg, so it is gated at the shared
AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex,
Azure) and in the Bedrock converse _handle_top_k_value path, with
drop_params threaded through the converse transform helpers.
Also updates the reasoning effort grid cell count assertion for the four
Fable 5 rows added on this branch (29 x 11 cells).
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Declare supports_sampling_params in the cost map schema
The model map validation schema uses additionalProperties: false, so the
new flag must be declared for the 28 entries that carry it; this was the
one failing job (misc / Run tests) on the previous commit.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* fix(bedrock): gate top_k=0 on converse to match Anthropic boundary
Truthiness check let top_k=0 silently disappear on models that removed
sampling params, while AnthropicConfig.transform_request treats 0 as
present and raises UnsupportedParamsError (or drops when drop_params is
set). Switch to 'is not None' so converse, direct Anthropic, invoke,
Vertex, and Azure all behave the same for top_k=0.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix duplicate cost callbacks for anthropic streaming pass-through
Two bugs caused _PROXY_track_cost_callback to see stream=True +
complete_streaming_response=None on every streaming pass-through request,
making the dedup guard in dispatch_success_handlers permanently inactive:
1. pass_through_endpoints.py created the Logging object with stream=False
for all requests. _is_assembled_stream_success short-circuits on
self.stream is not True, so has_dispatched_final_stream_success was
never set and any second dispatch went through unchecked.
Fix: set logging_obj.stream = True after stream detection.
2. _create_anthropic_response_logging_payload set complete_streaming_response
inside the try block after litellm.completion_cost(), so a pricing error
caused an early return without setting it on model_call_details.
Fix: set complete_streaming_response before the try block.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix stream
* add stream to logging obj
* test(pass_through): give mock logging object a real model_call_details dict
The anthropic passthrough logging payload now records the assembled
response on model_call_details before cost calculation, which requires
model_call_details to support item assignment. In production it is always
a dict; the existing unit test stubbed the logging object with a bare Mock
whose attribute is not subscriptable, so the new assignment raised
TypeError. Use a real dict to match the production logging object.
* test(pass_through): cover streaming logging-obj stream flag
The streaming branch of pass_through_request that marks the logging object
as streaming (logging_obj.stream and model_call_details["stream"]) had no
unit coverage, so the patch coverage gate flagged it. Add a regression test
that drives a streaming pass-through request through pass_through_request and
asserts the logging object is flagged as a stream before dispatch.
* test(pass_through): cover SSE-response stream flag fallback branch
The auto-detected streaming branch of pass_through_request (when a request
that was not flagged as streaming returns a text/event-stream response) sets
logging_obj.stream and model_call_details["stream"] but had no unit coverage,
so the codecov patch gate failed at 60%. Drive a non-streaming pass-through
request whose upstream response is SSE through pass_through_request and assert
the logging object is flagged as a stream before dispatch.
* fix(pass_through): gate complete_streaming_response on stream flag
perform_redaction only scrubs complete_streaming_response when
model_call_details["stream"] is True. Setting it unconditionally for
non-streaming Anthropic pass-through responses left the assembled
response unredacted in model_call_details, which is handed to logging
callbacks as kwargs when message logging is disabled. Only record it for
actual streaming responses so redaction always applies.
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit 2bbdbfa5c3)
* fix(vertex): strip output_config.effort for models that reject it
Haiku 4.5 on Vertex AI does not support output_config.effort and 400s with
"output_config.effort: Extra inputs are not permitted". PR #27074 emptied
VERTEX_UNSUPPORTED_OUTPUT_CONFIG_KEYS so effort would forward for Opus/Sonnet
4.6+, but that made the strip unconditional across every Vertex Anthropic
model, including ones that don't support it. Claude Code injects effort into
its default Messages payload, so `claude --model claude-haiku-4.5` started
failing.
Make the sanitizer model-aware: drop output_config.effort for models that
don't advertise output_config support (or any reasoning effort level) while
forwarding it for those that do. The fix covers both the chat-completion and
Messages pass-through transformation paths since they share the helper.
* chore(vertex): log at debug when dropping unsupported output_config.effort
Operators pointing an unregistered Vertex Claude alias that does support
effort would otherwise see it stripped with no signal. Debug level keeps it
out of normal logs since Claude Code sends effort on every request.
(cherry picked from commit cc55662e5f)
* fix(key_generate): allow team members to create keys on org-scoped teams
When a virtual key is created for a team, enterprise logic inherits the
team's organization_id onto the key (add_team_organization_id). Since the
VERIA-55 org-IDOR fix, /key/generate then required the caller to be an
explicit LiteLLM_OrganizationMembership member of that org, returning
403 "Caller is not a member of organization_id=<uuid>". Admins normally
only add users to teams (not orgs), so self-serve key creation regressed
for any user on an org-scoped team (regression since v1.84.0-rc.1).
Skip the org-membership check when organization_id was inherited from the
key's team (organization_id == team_table.organization_id). Team-level
authorization already gates this path, so team membership is sufficient.
The membership check still runs when a caller assigns an organization_id
that did not come from the key's team, preserving the IDOR protection.
Adds regression tests covering both the team-inherited (allowed) and
foreign-org (still blocked) cases.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(key_generate): cover mismatched team org IDOR path on generate
Add test_generate_key_foreign_org_with_mismatched_team_still_enforces_membership
for the case where a team is present but request organization_id differs from
team_table.organization_id. Enterprise inheritance is no-op'd in the test so
the guard is exercised directly; membership validation must still run.
Addresses Greptile review on #29310.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit b11833c737)
* fix(proxy): map stripped batch body.model to proxy alias for auth
replace_model_in_jsonl rewrites JSONL body.model to the provider id before
upload; batch file access checks must resolve that id back to model_name
so keys granted the proxy alias are not rejected with 403.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): surface resolved proxy alias in batch file 403 detail
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 70d2748d80)
* fix(azure): preserve AD token refresh in v1 OpenAI client path
The /openai/v1/ code path (api_version in {"v1", "latest", "preview"})
constructs a plain OpenAI/AsyncOpenAI client, but only forwarded
`api_key` from `azure_client_params`. When `enable_azure_ad_token_refresh`
is set (or any AD-only auth), `api_key` is None and the client
constructor raised "The api_key client option must be set...", breaking
every Azure call with a v1 api_version.
The OpenAI SDK (>=2.20.0) accepts a callable for `api_key` and re-invokes
it on every request via `_refresh_api_key`, so we now forward
`azure_ad_token_provider` directly — preserving the per-request token
refresh behavior of the regular AzureOpenAI client and avoiding the
expiry hole that resolving the token once at client-creation time would
introduce. Static `azure_ad_token` strings fall through to `api_key`.
For the async path we wrap the sync provider returned by azure-identity
in an async function since AsyncOpenAI expects `Callable[[], Awaitable[str]]`.
Fixes#27945https://claude.ai/code/session_01UnzrDSFUUgp5T2wRoPMxq5
* fix(azure): offload sync token provider to thread in v1 async wrapper
* fix(azure): include AD credential identity in v1 client cache key
---------
Co-authored-by: Claude <noreply@anthropic.com>
(cherry picked from commit 96a2e8b16d)
* fix: stop use_chat_completions_api flag from leaking into provider request body
use_chat_completions_api is a LiteLLM control flag that forces the
/responses -> /chat/completions bridge. It was missing from
all_litellm_params, so get_non_default_completion_params treated it as a
model-specific param and forwarded it to the upstream provider. A
model-level "use_chat_completions_api: true" in the proxy config therefore
reached the chat-completions path and was rejected by strict providers
(OpenAI/Anthropic) with HTTP 400 for an unknown body field.
Register it as a known internal param so it is stripped on every path
(completion, the responses bridge that calls litellm.completion, and
filter_out_litellm_params).
Adds a regression test driving litellm.completion() with a mocked OpenAI
client that asserts the flag never reaches the request body.
* test: clarify extra_body assertion in use_chat_completions_api leak test
Replace the misleading 'not in ... or {}' precedence idiom with an explicit
parenthesized guard that also handles extra_body being None.
* fix(rate-limit): stop v3 limiter from leaking internal stash to provider body
PR #27001 (atomic TPM rate limit) introduced a reservation flow that
writes four LiteLLM-internal keys onto the request data dict:
_litellm_rate_limit_descriptors
_litellm_tpm_reserved_tokens
_litellm_tpm_reserved_model
_litellm_tpm_reserved_scopes
_litellm_tpm_reservation_released
These keys are forwarded as request body params to the upstream provider,
which rejects them as unknown fields:
OpenAI -> 400 'Unknown parameter: _litellm_rate_limit_descriptors'
(mapped by litellm to RateLimitError / 429, hiding the bug
behind a misleading 'throttling_error' code)
Anthropic -> 400 '_litellm_rate_limit_descriptors: Extra inputs are
not permitted'
Net effect: every chat completion against any real provider fails the
moment a virtual key has any tpm_limit / rpm_limit set — i.e. v3-enforced
key-level TPM/RPM limits are broken end-to-end. The v3 RPM/TPM check
itself still runs (raises 429 on over-limit), but the success path
poisons the upstream body.
Reproduced on litellm_internal_staging HEAD (410ce761dc) against
gpt-4o-mini and claude-haiku-4-5 with a 1-RPM/1-TPM key — first request
fails with the provider's unknown-field error.
Fix: the stash is metadata only.
- Add RATE_LIMIT_DESCRIPTORS_KEY constant and a _LITELLM_STASH_KEYS
registry so we have a single source of truth for stash keys.
- New helper _stash_value_in_metadata_channels writes to
data['metadata'] / data['litellm_metadata'] without touching the
top level.
- _stash_reservation_in_data and the descriptor stash now route
through that helper. _mark_reservation_released stops writing
top-level.
- _lookup_stashed_value also checks kwargs['metadata'] /
kwargs['litellm_metadata'] (raw request_data shape) in addition to
kwargs['litellm_params']['metadata'] (completion kwargs shape).
- async_post_call_failure_hook now reads descriptors via the unified
metadata lookup instead of request_data.get(top-level).
- Defense in depth: async_pre_call_hook strips any stash key that
somehow surfaced at the top level (stale cache, future refactor,
test fixture) before returning.
Tests:
- New regression test asserts no _litellm_* stash key is present at
the top level of data after async_pre_call_hook, and that the
metadata channel still carries the reservation + descriptors so
success / failure reconciliation works.
- Existing test_tpm_concurrent.py tests that asserted top-level
presence are updated to read from data['metadata'] — the location
is an implementation detail; the spec is that post-call callbacks
can resolve the stash.
Verified end-to-end against OpenAI gpt-4o-mini and Anthropic
claude-haiku-4-5 via /v1/chat/completions on a low-rpm key:
- With limits not exceeded: HTTP 200, valid completion response,
no leaked fields in body.
- With RPM exceeded: HTTP 429 from v3 enforcement
('Rate limit exceeded ... Limit type: requests').
- With TPM exceeded: HTTP 429 from v3 enforcement
('Rate limit exceeded ... Limit type: tokens').
Full v3 hook test suite passes (171 tests).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore(rate-limit): use RATE_LIMIT_DESCRIPTORS_KEY constant in test, trim noisy comments
Address greptile P2: test fixture now uses the imported constant.
Drop comments that re-explain what well-named identifiers already convey.
* fix(rate-limit): reject caller-supplied stash values to prevent TPM-refund abuse
Strip _LITELLM_STASH_KEYS from data top-level and both metadata channels at
the start of async_pre_call_hook. Without this, an authenticated caller can
inject _litellm_rate_limit_descriptors plus _litellm_tpm_reserved_tokens in
body metadata, trigger a proxy-side rejection, and cause
async_post_call_failure_hook to refund TPM counters against attacker-named
scopes (e.g. another tenant's api_key).
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter
ResetBudgetJob's batched update_data path shipped the full key/user/team
model on each reset. Prisma rejects object_permission_id and budget_limits
on the update input type, so any row carrying those fields detonated the
entire batch -- spend never reset, budget_reset_at never advanced. After
v1.84.0 started populating object_permission_id on UI-created keys, this
fires routinely.
_reset_budget_common also zeroed the cross-pod spend counter before the
DB write, so failed resets left enforcement reading 0 from the counter
while the DB still held the over-budget spend, admitting requests past
the cap until the counter naturally re-saturated from new reservations.
Switch the write to per-row narrow updates ({spend, budget_reset_at})
via db.batch_, and move the counter invalidation out of
_reset_budget_common so it only fires after the DB write commits. On
DB-write failure the counter is left untouched, enforcement continues
to block, and the next scheduler tick can retry without leaving a
bypass window.
Fixes#27730.
* fix(reset_budget): address Greptile review on #29358
- Strengthen the bypass-half regression test: replace the for-loop over
call_args_list (vacuously true when empty) with assert_not_called(),
so the test would actually flag a re-introduction of counter-zeroing
via any code path.
- Add the same explanatory docstring on _write_user_reset_updates and
_write_team_reset_updates that _write_key_reset_updates already has,
so all three helpers point future maintainers at #27730.
* test(reset_budget): update test_proxy_budget_reset for new batch-write path
Same shape as the previous test_reset_budget_job.py update: keys/users/teams
now write through prisma.db.batch_().<table>.update, not update_data, so the
tests need a batcher mock and updated assertions. Adds:
- _wire_batcher_for_test helper that returns a list which accumulates per-row
batch updates captured from prisma_client.db.batch_().
- _attrify helper that wraps dict fixtures so getattr(item, "token") works
alongside the dict item-access the fake_reset_* mocks rely on. The new
narrow-write helpers use getattr to pull out the row's id, and would
silently skip plain dicts otherwise.
- Updates 3 partial_failure tests to assert against the batch-call list
(rows by id, payload contains only {spend, budget_reset_at}) instead of
update_data.assert_awaited_once + data_list inspection.
- Updates test_reset_budget_continues_other_categories_on_failure: only
budget + enduser still flow through update_data; key/user/team go through
the batch path now.
- Wires the batcher mock into 3 service_logger_*_success tests so commit()
is actually awaitable and the success hook fires.
These tests were silently passing locally only because the editable install
in .venv pointed at the main repo, not the worktree — running pytest with
PYTHONPATH overridden to the worktree (matching CI) reproduces the failures.
* refactor(proxy/auth): normalize Bearer prefix in safe-hash helper
UserAPIKeyAuth._safe_hash_litellm_api_key now strips a leading
"Bearer "/"bearer " prefix before its existing sk-/JWT classification, so
the helper produces the same hashed output regardless of whether the
caller stripped the Authorization header prefix or passed the header
value through unchanged.
* refactor(proxy/auth): make Bearer-prefix strip case-insensitive
Per RFC 7235 the HTTP authorization scheme token is case-insensitive.
Replace the two-prefix loop with a single case-insensitive check so the
helper normalizes "Bearer ", "bearer ", "BEARER ", and any mixed-case
variant before classifying the remainder as sk- or JWT. The contract
test gains coverage of "BEARER " and "BeArEr ".
* test(mcp): align auth-handler test expectations with safe-hash helper
The two MCP auth tests asserted that UserAPIKeyAuth(api_key="Bearer ...")
retained the raw header bytes on the api_key field. _safe_hash_litellm_api_key
now normalizes that input — stripping the Bearer prefix and hashing the
resulting sk- key — so the expectations move to the normalized form:
the bare token in the parametrize case, and hash_token("sk-...") in the
backward-compat assertion. This matches what the real auth flow produces
(the builder strips Bearer and the DB stores the hashed token), so the
mocks now line up with production rather than with the un-normalized
validator output.
* chore(proxy): cherry-pick #28547 onto patch/v1.85.1
Backport of #28547 (`d480ffda3c`) onto the `patch/v1.85.1` branch.
Routes the remaining path-dependent call sites in auth, ACL, routing,
and audit-log decisions through `get_request_route(request)` so they
read from the ASGI `scope["path"]` instead of `request.url.path`. The
helper itself already exists on v1.85.1 (added by #27878); this PR
extends the helper's usage to the additional sites listed below.
Sites routed through get_request_route:
- _experimental/mcp_server/auth/user_api_key_auth_mcp.py
- management_endpoints/mcp_management_endpoints.py
- vector_store_endpoints/utils.py
- pass_through_endpoints/pass_through_endpoints.py
- auth/route_checks.py
- litellm_pre_call_utils.py
- spend_tracking/spend_management_endpoints.py
- common_utils/http_parsing_utils.py
- management_helpers/utils.py
- health_endpoints/_health_endpoints.py
Regression tests in tests/proxy_unit_tests/test_proxy_routes.py
construct a Request with scope["path"] set to a benign route and the
Host header crafted so url.path would resolve differently; each
site's decision is asserted against scope["path"].
Conflict resolution
-------------------
Cherry-pick applied cleanly with no conflicts. All 11 files plus the
test file are pure `request.url.path` → `get_request_route(request)`
swaps with the lazy auth_utils import (no feature drift).
* bump: version 1.85.1 → 1.85.2
* chore: uv lock after version bump 1.85.1 → 1.85.2
* fix(vertex_ai): omit function_call id on Vertex Gemini 3.5+ tool turns
Vertex AI rejects `id` on function_call/function_response parts; only Google AI Studio accepts it for Gemini 3.5+ strict tool matching.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(vertex_ai): forward custom_llm_provider in context caching
Pass custom_llm_provider through to _gemini_convert_messages_with_history
in the context caching path so Gemini 3.5+ tool-call `id` forwarding
behaves consistently between cached and non-cached completions on Google
AI Studio.
Co-authored-by: Claude <claude@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <claude@anthropic.com>
(cherry picked from commit fecf212d70)
* fix(spend_counter): seed Redis counter via SET NX to prevent cross-pod double-seed
Symptom
-------
Customers on multi-pod deployments see team `spend` jump to ~2x (or N x
the pod count) shortly after a Redis cache miss / TTL expiry, triggering
spurious "Budget Crossed" alerts and blocked requests until the value is
manually reset.
Root cause
----------
`SpendCounterReseed.coalesced` warmed the primary spend counter by
calling `redis.async_increment(key, value=db_spend, refresh_ttl=True)`,
which lowers to Redis `INCRBYFLOAT`. That is additive, not idempotent.
The per-counter `asyncio.Lock` only coalesces seeders inside one
process. With N pods sharing one Redis, on a cold key (cold start, TTL
expiry, manual delete) every pod independently passes its lock + Redis
re-check, reads the same `db_spend`, and issues `INCRBYFLOAT db_spend`.
Final value: N x db_spend.
Fix
---
Use `redis.async_set_cache(key, value=db_spend, nx=True)` for the seed.
SET NX is atomic across pods: exactly one writer initializes the key;
losers read the winner's value via `async_get_cache`. This is the same
idiom already used by `coalesced_window` in the same file, so the two
seed paths are now consistent.
Per-request deltas continue to use `INCRBYFLOAT` (correct - additive
behaviour is what we want for increments, not for initial seed).
Verification
------------
Live two-process repro against the same Postgres + Redis (DB
spend = 506):
Unpatched: 4/4 runs -> Redis counter = ~1012 (~2 x db_spend)
Patched: 12/12 runs -> Redis counter = ~506
Unit tests (`test_proxy_server.py`):
- New `test_primary_spend_counter_redis_concurrent_seed_does_not_double_seed`
patches `_get_lock` to return a fresh lock per caller (otherwise the
per-process lock masks the race), races two `coalesced` calls, and
asserts final = 506 with exactly one of two SET NX attempts winning.
- 4 existing tests updated for the new seed contract (SET NX for the
seed, INCRBYFLOAT only for the per-request delta).
- Full `spend_counter or reseed or budget` slice: 22 passed.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(spend_counter): make SET NX mock atomic so loser branch is exercised
Greptile flagged that `redis_set_cache` in
test_primary_spend_counter_redis_concurrent_seed_does_not_double_seed
placed `await asyncio.sleep(0)` AFTER the NX membership check. Both
concurrent tasks observed an empty `redis_store`, passed the guard, and
both returned True - so the loser branch (else: read back winner's value)
was never exercised.
Fix the mock to model real atomic Redis SET NX:
- Yield BEFORE the membership check so two concurrent callers interleave
the way real SET NX does (first to resume runs check + write atomically
and wins; second resumes after the key exists and loses).
- Track set_cache return values; assert sorted([loser, winner]) so we
know exactly one task wins and one loses.
- Track async_get_cache calls that happen AFTER at least one SET NX has
completed; assert at least one such read - that is the loser-path
fallback (`current_value = float(cached)` when seeded is False).
Verified by temporarily reverting the mock to the old order: the test
now fails with `expected exactly one SET NX winner and one loser, got
[True, True]`, exactly the failure mode Greptile described.
No production code change.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(spend_counter): mock async_set_cache to populate redis_store in concurrent read+write test
`test_concurrent_read_and_write_paths_share_one_db_query` mocks
`async_increment` to populate the in-memory `redis_store`, but did not
mock `async_set_cache`. After the SET-NX seed change in `coalesced()`,
the seed step writes via `async_set_cache(nx=True)` (default AsyncMock,
no `redis_store` write), so the simulated Redis stays empty after the
first reseed. The second `get_current_spend` then sees a clean Redis
miss, re-enters the DB read path, and the test fails with
`expected 1 DB query, got 2`.
Fix: add a `redis_set_cache` side_effect that updates `redis_store` on
`nx=True` (and rejects when the key already exists), matching the
pattern used by the four sibling tests fixed in this branch's first
commit. Pre-existing assertions are unchanged.
Full `tests/test_litellm/proxy/test_proxy_server.py`: 158 passed.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit 0fb710400f)
Provider validation errors (e.g. OpenAI RateLimitError carrying 178
pydantic errors each with their own 'input': [...]) were stored verbatim
in LiteLLM_SpendLogs.metadata.error_information.error_message via
str(original_exception), producing rows >12 MB.
Sanitize before metadata is serialized:
- redact 'input'/'messages' values in both error_message and traceback
when store_prompts_in_spend_logs is False (back-door leak paths)
- always apply the MAX_STRING_LENGTH_PROMPT_IN_DB size cap to
error_message and traceback (DB-storage safeguard)
Value scanning uses a parser-based balanced-bracket walk that respects
string quoting, so multi-modal payloads ('messages': [{'content': [...]}])
and user text containing literal brackets ("secret[123") are handled
correctly instead of leaking past a depth-1 regex.
Scoped to the spend-log path so OTEL/Datadog/etc. callbacks still
receive the untruncated error per LITELLM_TRUNCATION_DB_SAFEGUARD_NOTE.
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: patch Host-header auth bypass in get_request_route
Starlette reconstructs request.url from the Host header. A malformed
Host like `localhost/?x=1` causes Starlette to build the full URL as
`http://localhost/?x=1/health`, which url-parses to path="/". Since "/"
is in LiteLLMRoutes.public_routes, all protected routes became reachable
without authentication.
Fix: read scope["path"] (set by uvicorn from the HTTP request line,
not derivable from headers) instead of request.url.path. Sub-path
deployments are handled via scope["app_root_path"] / scope["root_path"],
mirroring Starlette's own base_url construction logic.
Affected variants confirmed fixed:
Host: localhost/?x=1
Host: localhost:4000/?x=1
Host: localhost/#test
Host: localhost:4000/#test
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: reduce comments in route fix
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block credential fields in RAG ingest vector_store options
Credential fields (vertex_credentials, aws_access_key_id, api_key, etc.)
in ingest_options.vector_store are now rejected at the API boundary with
a 400 error. Credentials must be configured server-side.
Previously any authenticated user could supply a vertex_credentials dict
with type=external_account pointing credential_source.file at an
arbitrary path (e.g. /proc/1/environ) and token_url at an
attacker-controlled server. google-auth's identity_pool.Credentials
refresh() would read the file and POST its contents to the attacker.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block /key/update self-escalation by assigned users
Non-admin users who were assigned a key (created_by != caller) could
update any non-budget field — models, rpm_limit, guardrails, etc. —
without admin authorization, allowing privilege self-escalation.
Gate: only the key creator (created_by == caller) may edit their own
key without admin check; budget changes always require admin regardless
of creator status. All other callers must pass _check_key_admin_access.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block user-controlled api_base in RAG ingest vector_store options
A user-supplied api_base in ingest_options.vector_store caused the server
to forward its configured provider credentials (Gemini, OpenAI) to an
attacker-controlled endpoint via SSRF.
Add api_base to the blocked credential params set alongside api_key and
the existing credential fields.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: restrict /utils/transform_request to PROXY_ADMIN and apply body safety check
Any authenticated internal_user could POST arbitrary provider config
(aws_sts_endpoint, api_base, etc.) to /utils/transform_request and have
the server forward its credentials to an attacker-controlled endpoint.
- Gate the endpoint on PROXY_ADMIN role (403 for all other roles)
- Call is_request_body_safe() to reject banned params even for admins
- Convert ValueError from safety check to HTTP 400
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: apply banned-param check to /utils/transform_request
Without is_request_body_safe(), any authenticated user could pass
aws_sts_endpoint, api_base, or aws_web_identity_token to
/utils/transform_request and have the server forward its configured
provider credentials to an attacker-controlled endpoint during SDK
credential resolution.
Applies the same banned-param blocklist already used by LLM endpoints.
Endpoint remains accessible to all authenticated users.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block SSRF via api_base in /prompts/test dotprompt YAML frontmatter
Any frontmatter key not in ["model","input","output"] flowed into
optional_params and was merged into the LLM call data dict, bypassing
is_request_body_safe. An attacker with any bearer key could set
api_base in YAML to redirect the outbound LLM request — including the
provider API key — to an attacker-controlled host.
Fix: call is_request_body_safe on the constructed data dict after
optional_params are merged, before invoking ProxyBaseLLMRequestProcessing.
ValueError from the banned-param check is surfaced as HTTP 400.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Update litellm/proxy/rag_endpoints/endpoints.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix: coerce nested config strings before banned-param check
_NESTED_CONFIG_KEYS descent used isinstance(nested, dict) which silently
skipped litellm_embedding_config when delivered as a JSON string via
multipart/form-data. Banned params (api_base, aws_sts_endpoint, etc.)
nested inside the stringified value were invisible to is_request_body_safe.
_NESTED_METADATA_KEYS already used _coerce_metadata_to_dict which parses
JSON strings before checking. Apply the same coercion to _NESTED_CONFIG_KEYS.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: replace substring match with prefix match in is_llm_api_route
mapped_pass_through_routes used `_llm_passthrough_route in route` (substring)
so any admin-only path whose URL contained a provider name (openai, anthropic,
azure, bedrock, etc.) was misclassified as an LLM API route and bypassed the
admin gate in non_proxy_admin_allowed_routes_check.
Confirmed live: non-admin key could GET /credentials/by_name/openai (read
masked provider API key) and DELETE /credentials/openai (delete credential).
Fix: use exact match or startswith(prefix + "/") — the same pattern used
everywhere else in RouteChecks — so only routes that actually start with a
passthrough prefix are allowed through.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: stabilize PR #27878 test failures
- key_management_endpoints: extend can_skip_admin_check to team keys so
team members with /key/update permission can update non-budget fields.
can_team_member_execute_key_management_endpoint already validates team
membership + permission and raises if unauthorized; reaching the admin
check on a team key means the caller was authorized.
- test: set created_by on mock key in
test_update_key_non_budget_fields_allowed_for_internal_user so
caller_is_creator resolves correctly (MagicMock default ≠ user_id).
- auth_utils.get_request_route: guard against non-dict request.scope
(e.g. MagicMock in unit tests) to prevent a MagicMock leaking into
UserAPIKeyAuth.request_route and failing Pydantic validation.
- ci: assign test_multipart_bypass_repro.py to the proxy-runtime shard
in test-unit-proxy-db.yml to satisfy the shard-coverage check.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(lint): add explicit str() cast in get_request_route for MyPy
scope.get() returns Any|None which MyPy cannot coerce to str implicitly.
Wrap both scope.get() calls in str() to satisfy the type checker.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: guard bare-/ root_path strip + make total_spend migration idempotent
auth_utils.get_request_route: when Starlette sets scope["app_root_path"]
to "/" (e.g. behind some middleware), the old stripping logic would
remove the leading slash from every path ("/team/new" → "team/new"),
breaking route matching and causing auth to misclassify protected routes.
Skip stripping when root_path is bare "/".
migration: add IF NOT EXISTS to total_spend ALTER TABLE so the migration
is safe to replay when a prior partial run already created the column.
Without this guard, prisma migrate deploy fails on CI DBs that were
partially migrated, causing all subsequent DB operations (including
/team/new) to 500.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: require creator still owns key for personal-key bypass in /key/update
caller_is_creator now requires both created_by == caller AND user_id ==
caller. Previously checking only created_by let a demoted admin who
originally created a key for another user continue editing non-budget
fields on it after reassignment, bypassing _check_key_admin_access.
Adds regression test: creator whose key was reassigned is blocked (403).
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: extract auth checks to fix PLR0915 + broaden max_budget assertion
internal_user_endpoints._update_single_user_helper exceeded 50 statements
(PLR0915). Extract authorization checks into _check_user_update_authz helper
to bring statement count under the limit.
test_validate_max_budget: assert "negative" (substring of both the local
"cannot be negative" and the CI "non-negative finite number" messages) so
the test is stable regardless of which exact wording the function uses.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
``test_azure_ad_token_is_in_banned_list`` only asserted tuple
membership of a name the parametrized test already exercises end-to-end
through ``is_request_body_safe``. Removed.
Tightened the admin-opt-in test comment.
``_NESTED_CONFIG_KEYS`` descent used ``isinstance(nested, dict)``, so a
caller sending ``extra_body`` as a JSON-encoded string instead of an
object (the same shape multipart/form-data clients use for
``litellm_metadata``) skipped the banned-key check entirely. Switched to
``_coerce_metadata_to_dict`` so the JSON-string path is parsed before
descent — mirrors the existing handling on ``_NESTED_METADATA_KEYS``.
``extra_body`` is the OpenAI-SDK passthrough container. Provider
modules read provider-auth fields out of it directly (Azure's
``extra_body.azure_ad_token``, Bedrock's
``extra_body.aws_web_identity_token``, etc.) without re-validating, so
the boundary check has to walk it the same way it walks
``litellm_embedding_config``. Adding it to ``_NESTED_CONFIG_KEYS``
extends single-level banned-key descent into the container — top-level
admin opt-ins (``allow_client_side_credentials`` /
``configurable_clientside_auth_params``) still apply.
``azure_ad_token`` was not in ``_BANNED_REQUEST_BODY_PARAMS`` despite
being the bearer-token field the Azure transformer resolves through
``get_secret`` (same shape as ``aws_web_identity_token`` on the
Bedrock STS path). Added so it can't be supplied per-request without
an admin opt-in.
* feat(mcp): support MCP access group names in URL-based namespacing
Extends dynamic_mcp_route to resolve /{name}/mcp requests where {name}
is an MCP access group tag or a comma-separated list of servers/groups,
matching what the documentation promised but the handler did not implement.
Resolution order: registered server alias → toolset → comma-separated
list → single access group tag (404 if none match).
Adds unit tests covering all four resolution paths plus 404 cases.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): address Greptile review comments on dynamic_mcp_route
- Move comma-separated check before toolset DB lookup so comma names
short-circuit without hitting the database
- Cache access-group DB lookups via user_api_key_cache to avoid a raw
find_many on every request (matches toolset caching pattern)
- Remove unused response_started variable from _forward_as_mcp_path
- Update tests to assert comma list skips toolset call and to mock cache
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(mcp): extract helpers to fix PLR0915 too-many-statements in dynamic_mcp_route
Extract _mcp_forward_as_path and _is_mcp_access_group_cached as
module-level helpers so dynamic_mcp_route stays under the 50-statement
limit. Update tests to patch the new module-level symbols directly.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Avoid caching missing MCP access groups
* fix(mcp): stream MCP responses via _stream_mcp_asgi_response instead of buffering
_mcp_forward_as_path previously accumulated the full response body in
memory before sending it. Replace the buffering custom_send pattern with
_stream_mcp_asgi_response, which uses an asyncio.Queue bridge so chunks
are yielded to the client as they arrive, preventing unbounded memory
growth on large or long-lived MCP responses.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): short-TTL negative cache for access-group existence lookup
An unauthenticated caller could repeatedly request /<unknown>/mcp and
force a fresh DB lookup for the access-group existence check on every
request (only positive results were cached). Cache negative results
for a short DEFAULT_MCP_ACCESS_GROUP_NEGATIVE_CACHE_TTL window (10s by
default) so the DB is shielded from flooding while a transient DB error
(which surfaces as an empty list) cannot hide a real group for long.
https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9
* fix(mcp): use plain int for access-group negative cache TTL
Drop the os.getenv wrapper around DEFAULT_MCP_ACCESS_GROUP_NEGATIVE_CACHE_TTL
to avoid the documentation_test_env_keys check failing on the new variable.
The negative-cache window is a small internal tuning constant, not a
user-facing knob, so a plain integer is clearer than an env override.
https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9
* fix(mcp): validate, dedupe, and cap CSV tokens in dynamic MCP route
For /{name1,name2,...}/mcp, validate every token resolves to a known
server alias or access group, dedupe case-insensitively, and cap at
DEFAULT_MCP_NAMESPACE_CSV_MAX_TOKENS=16 before forwarding.
- Bounds the per-request DB / cache fan-out an authenticated caller can
trigger by stuffing the path with tokens (raised by veria-ai).
- Returns 404 instead of forwarding when no token resolves, so the
downstream server filter cannot silently fall back to the full
allowed_mcp_servers list (raised by Cursor agentic security review).
- Forwards only the resolved subset, so unknown tokens cannot ride along
into the downstream filter.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(mcp): exact-match CSV token dedupe to preserve case-sensitive distinct tokens
Bugbot flagged that case-insensitive dedup on `MyGroup,mygroup` could
collapse to whichever case appeared first and silently drop the matching
casing if the downstream resolver is case-sensitive. Switch to exact-match
dedup so distinct casings survive; whitespace-only differences still
collapse via the .strip() before comparison.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: mateo-berri <mateo@berri.ai>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
A guardrail entry's ``callbacks`` list (v1: ``{name: {callbacks:[...]}}``,
v2: ``{guardrail_name, litellm_params: {callbacks: [...], guardrail:
"module.path"}}``) is iterated during config load and threaded through
``get_instance_fn``. A PROXY_ADMIN persisting
``litellm_settings.guardrails[*].callbacks: ["s3://..."]`` or
``litellm_settings.guardrails[*].litellm_params.guardrail: "s3://..."``
via ``/config/update`` was not covered by the previous scrub matrix.
Walk both v1 and v2 entry shapes and null out remote-URL callbacks /
module-path values before the merge. Adds four regression tests.
aws_sts_endpoint, aws_web_identity_token, and aws_bedrock_runtime_endpoint
in ingest_options.vector_store were passed directly to the Bedrock ingestion
class, which reads them into boto3 STS client construction. Any authenticated
caller could redirect AssumeRole calls to an attacker-controlled server,
leaking the proxy's instance profile credentials.
Calls is_request_body_safe() on ingest_options["vector_store"] before
forwarding to litellm.aingest(). Same banned-params list and admin opt-in
escape hatch (allow_client_side_credentials) as the /chat/completions path.
ValueError from the safety check is caught and re-raised as HTTP 400.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(prometheus): emit remaining_tokens/requests gauges for bedrock + vertex (LIT-2719)
Bedrock and Vertex AI never return x-ratelimit-remaining-* response headers,
so litellm_remaining_tokens_metric / litellm_remaining_requests_metric only
fired for OpenAI / Azure / Anthropic deployments even when tpm/rpm was
configured on the router.
Add a provider-agnostic fallback in PrometheusLogger.async_log_success_event
that asks Router.get_remaining_model_group_usage() for the same model_group
and emits the gauges with configured_limit - current_usage when the upstream
provider didn't populate the headers itself. Existing OpenAI / Azure /
Anthropic flows are unchanged because the fallback short-circuits when both
header values are already present.
Tests: 8 new tests covering bedrock + vertex emission, header short-circuit,
partial-header fill, llm_router=None, missing model_group, empty router
result, and router exception swallowing.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(prometheus): narrow except to ImportError, log router lookup failures via verbose_logger.exception
Address greptile review:
- The optional 'from litellm.proxy.proxy_server import llm_router' should
guard against ImportError specifically, not all exceptions, so that
unexpected errors (e.g. AttributeError from partially-initialized state)
stay visible.
- get_remaining_model_group_usage failures are now logged via
verbose_logger.exception (with traceback) instead of debug, matching the
PR description's intent and avoiding silent loss of router-cache errors
in production.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(prometheus): subtract in-flight delta in router-remaining fallback
The router's TPM/RPM counter is incremented by
Router.deployment_callback_on_success, which fires alongside this
prometheus callback in the success-log fan-out. Prometheus wins the
race, so get_remaining_model_group_usage returns the pre-decrement
counter for the current request — while vendor headers
(OpenAI/Anthropic/Azure) are already post-decrement.
That broke parity between providers on the same gauge: dashboards
plotting litellm_remaining_requests_metric showed Bedrock/Vertex
perpetually one request behind Anthropic for the same throughput.
Replay the in-flight increment before emit: subtract total_tokens
from remaining_tokens and 1 from remaining_requests.
* Revert "fix(prometheus): subtract in-flight delta in router-remaining fallback"
This reverts commit 001ce95ecdd952b4b5a23dd2b1e62c4562c932bc.
* fix(router): post-decrement router-derived ratelimit headers
Router.set_response_headers injects x-ratelimit-remaining-{tokens,
requests} for providers that don't return them natively (Bedrock,
Vertex). The values come from get_remaining_model_group_usage, which
reads the router's TPM/RPM counter — incremented post-response by
deployment_callback_on_success. So the headers reflected the counter
state before the current request was counted: pre-decrement.
Vendor headers from OpenAI/Anthropic/Azure are post-decrement (the
vendor counted the request before responding). Same metric name, two
semantics — dashboards plotting litellm_remaining_requests_metric
showed Bedrock/Vertex perpetually one request behind for the same
throughput, and the HTTP response headers exposed the same skew to
clients.
Subtract the in-flight delta before writing: 1 from
remaining-requests, response.usage.total_tokens from remaining-tokens.
Fixes both the response headers and (transitively) the prometheus
gauges that read from standard_logging_payload.additional_headers.
---------
Co-authored-by: cursor <cursor@example.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
A pass-through endpoint's ``target`` field is passed through
``create_pass_through_route`` into ``get_instance_fn`` during config
load. A PROXY_ADMIN persisting ``target: "s3://attacker/m.i"`` via
the DB-overlay ``pass_through_endpoints`` write path was not covered
by the previous scrub matrix, so the remote module load would still
reach the loader because the YAML-load chain has ``config_file_path``
set.
Walk each entry in ``general_settings.pass_through_endpoints`` and
null out any ``target`` that starts with ``s3://`` or ``gcs://``. The
entry itself is preserved so the path-registration helper can choose
how to handle a missing target (the existing code skips the route
when ``target is None``).
Adds two regression tests.
When ``ProxyConfig`` merges DB-persisted ``litellm_settings`` /
``general_settings`` on top of the YAML config, the merged dict is
later iterated by ``load_config`` which threads ``config_file_path``
(the YAML path) into ``get_instance_fn``. The runtime gate that
refuses ``s3://`` / ``gcs://`` modules when ``config_file_path`` is
``None`` therefore can't distinguish a YAML-sourced value from a
DB-sourced one: both look the same to ``get_instance_fn``.
Strip ``s3://`` / ``gcs://`` entries from the DB-overlay value for
every field whose contents reach ``get_instance_fn`` during config
load:
- litellm_settings: ``callbacks``, ``success_callback``,
``failure_callback``, ``audit_log_callbacks``, ``post_call_rules``,
``custom_provider_map[].custom_handler``
- general_settings: ``custom_auth``, ``custom_key_generate``,
``custom_key_update``, ``custom_sso``,
``custom_ui_sso_sign_in_handler``,
``litellm_jwtauth.custom_validate``
The YAML config-file load path is unchanged — the documented operator
flow (``callbacks: ["s3://bucket/module.instance"]`` in ``config.yaml``)
still works. Only DB-overlay writes (e.g. via ``/config/update``) are
stripped.
Adds 16 regression tests covering the scrub matrix.
Authenticated clients could supply CustomPricingLiteLLMParams fields
(input_cost_per_token, output_cost_per_token, etc.) in the request body.
These were forwarded to register_model() in main.py, permanently mutating
the shared global litellm.model_cost dict for all users on the instance.
Adds all CustomPricingLiteLLMParams fields to _BANNED_REQUEST_BODY_PARAMS
so is_request_body_safe() rejects them before they reach completion().
New pricing fields added to CustomPricingLiteLLMParams are auto-covered.
Admin opt-in via allow_client_side_credentials or
configurable_clientside_auth_params still works as before.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(fireworks_ai): strip thinking_blocks from chat messages before API call
Fireworks OpenAI-compatible ChatMessage schema uses additionalProperties:false
and rejects Anthropic-style messages[].thinking_blocks (e.g. Claude Code replays),
returning invalid_request_error. Remove the field in _transform_messages_helper
alongside provider_specific_fields.
Adds unit test test_transform_messages_helper_strips_thinking_blocks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(fireworks_ai): drop inline comments from message sanitization
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(fireworks_ai): explain why provider_specific_fields and thinking_blocks are stripped
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: strip Gemini thought-signature from tool_use.id in non-streaming path; example websearch config (#27873)
- adapters/transformation.py: mirror the streaming path and strip the
`__thought__<b64>` suffix off `tool_call.id` before building the
AnthropicResponseContentBlockToolUse. Base64's `+ / =` characters
violate Anthropic's `^[a-zA-Z0-9_-]+$` tool_use.id pattern, so when a
conversation that flowed through Gemini is later replayed to an
Anthropic-native provider (Bedrock or Anthropic API) the request 400s.
- example_config_yaml/websearch_interception_config.yaml: register the
interceptor under `callbacks:` not `success_callback:`. `success_callback`
does not run pre-request hooks, so the tool-conversion step never fires
on `/v1/messages` and the raw `web_search_20250305` tool is forwarded
to Bedrock, which 400s.
- adds a unit test pinning the non-streaming strip behavior and the
surviving `^[a-zA-Z0-9_-]+$` shape of the resulting id.
Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com>
* Fix/azure image edit auth header (#27863)
* fix(azure/image_edit): use api-key header instead of Authorization Bearer
Delegate `AzureImageEditConfig.validate_environment` to
`BaseAzureLLM._base_validate_azure_environment` so the image-edit route
follows the same auth resolution as every other Azure provider:
- prefer the Azure-native `api-key` header when an API key is available
- fall back to `Authorization: Bearer <azure_ad_token>` only for AAD auth
The previous implementation unconditionally set
`Authorization: Bearer <api_key>`, which is the OpenAI-direct convention
and is rejected by Azure OpenAI / APIM-fronted deployments with
`401 Access denied due to missing subscription key`.
Adds regression tests covering api_key kwarg, litellm_params.api_key, and
the AAD-token fallback path.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(azure/image_edit): pin api-key precedence semantics + add regression test
Address review feedback that the move to
``BaseAzureLLM._base_validate_azure_environment`` changed the relative
priority of the positional ``api_key`` kwarg vs. ``litellm_params["api_key"]``.
The new behavior — ``litellm_params["api_key"]`` wins, positional only fills
in when ``litellm_params["api_key"]`` is empty — is intentional and matches
every other Azure ``validate_environment``: ``AzureVideosConfig`` uses the
exact same merge logic, while ``AzureVectorStoresConfig`` and
``AzureResponsesAPIConfig`` don't accept a positional ``api_key`` at all.
The old ``or`` chain (positional wins) was the outlier and was part of the
same OpenAI-vs-Azure convention drift that produced the original
``Authorization: Bearer`` bug.
The only production caller (``llm_http_handler.image_edit``) sources both
values from the same ``litellm_params.api_key``, so this change is
behaviorally a no-op there. Document the precedence in the docstring and
lock it in with an explicit test so future refactors can't quietly
re-invert it.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Adam Kirstein <adam.kirstein@disney.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(azure/image_edit): expect api-key header instead of Authorization Bearer
PR #27863 fixed Azure image edit to use the Azure-native api-key header
instead of OpenAI's Authorization: Bearer convention, but did not update
test_azure_image_edit_litellm_sdk to match. The test still asserted
'Authorization' in headers, which now fails since the new code routes
through BaseAzureLLM._base_validate_azure_environment and emits
api-key when an api_key is provided.
Update the assertion to pin the correct Azure behavior: api-key header
present with the resolved key, and no Authorization header.
---------
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com>
Co-authored-by: Adam Kirstein <107421694+justalittleadam@users.noreply.github.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Adam Kirstein <adam.kirstein@disney.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* fix(ollama): Include provider in model list for ollama (#26135)
* Include provider in model names for ollama
* Fix unit tests
* fix(ollama): process both thinking and content in same streaming chunk (#26098)
* fix(health_check): skip max_tokens for image_generation mode (#26417)
* fix(health_check): skip max_tokens for image_generation mode
`_update_litellm_params_for_health_check` injected `max_tokens` for
every deployment. OpenAI `/v1/images/generations` strictly rejects
unknown fields, so health checks for dall-e-* and gpt-image-1 always
failed with `400 "Unknown parameter: 'max_tokens'"` even though the
actual image endpoint calls succeed. Skip the `max_tokens` injection
when `model_info.mode == "image_generation"`. `messages` still gets
injected (downstream `_filter_model_params` already strips it for
non-chat handlers).
* Switch to allow-list with per-deployment override
Per @krrishdholakia review: deny-listing image_generation only re-introduces
the same bug for every other non-chat mode (embedding, audio_*, rerank,
video_generation, ocr, search, moderation, ...).
Replace the single image_generation skip with `_MAX_TOKEN_SUPPORT_MODES =
{chat, completion, responses}`. Missing `mode` is treated as chat for
backward compatibility. New modes are safe by default.
Add `model_info.health_check_supports_max_tokens` as an operator escape
hatch — True forces injection on a non-listed deployment (operator wants
to bound probe tokens), False suppresses it on a chat-style deployment
behind a strict-schema provider.
Tests: parametrize over 3 chat-style + 10 non-chat modes, plus override
on/off and the no-mode legacy path.
* fix(http_handler): handle RequestNotRead in MaskedHTTPStatusError for multipart uploads (#26718)
Squash-merged by litellm-agent from dawidkulpa's PR.
* fix(ollama): guard against double 'ollama/' prefix in live model listing
Greptile flagged that Ollama servers can return names that already start
with 'ollama/'. Check the prefix before prepending so we don't produce
'ollama/ollama/...'. Adds a regression test.
* Fix Ollama empty reasoning stream chunks
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: James Myatt <james@jamesmyatt.co.uk>
Co-authored-by: VHash <225398745+vhash0@users.noreply.github.com>
Co-authored-by: hayden <sewhan.kim+@a-bly.com>
Co-authored-by: dawidkulpa <84176950+dawidkulpa@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: Fix Redis Sentinel client handling to solve authentication error with password protected sentinel (#25625)
* fix Redis Sentinel authentication handling
* test: cover Redis Sentinel auth routing
* refactor: align Redis Sentinel kwargs threading
* fix: avoid duplicate Redis Sentinel socket timeouts
* Address review comments
* refactor(_redis): return set from _get_redis_kwargs for O(1) lookup
Align _get_redis_kwargs() with the cluster helper by returning a set
instead of a list, so the sentinel connection-kwargs filter uses O(1)
membership tests. Addresses Greptile review feedback on PR #26302.
* fix(_redis): restore Azure-specific kwargs in cluster kwargs set
The set-literal refactor of _get_redis_cluster_kwargs dropped four
LiteLLM-custom Azure keys (azure_redis_ad_token, azure_client_id,
azure_tenant_id, azure_client_secret) that the prior list form had
explicitly appended. Because they are not in RedisCluster's argspec,
they were silently stripped, breaking Azure IAM auth on cluster
clients. Re-add them to the explicit include set.
---------
Co-authored-by: Kristin Cowalcijk <kristincowalcijk@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: krrish-berri-2 <krrish-berri-2@users.noreply.github.com>
Co-authored-by: claude <claude@anthropic.com>
External readiness probes consumed the legacy detailed payload's `db`
field to drive alerting and pod-rotation decisions. Stripping the body
to `{"status": "healthy"}` broke those probes silently — the HTTP code
still flipped to 503, but probes checking `body.db == "connected"`
treated the response as healthy.
Add `db` back to the unauthenticated payload. Keep the rest of the
diagnostic fields (litellm_version, callbacks, cache, log_level) gated
behind /health/readiness/details so the recon-leak gate from #26912
holds. Values match the legacy contract: "connected", "disconnected",
"Not connected".
