When AzureSentinelLogger is resolved from the string callback name
"azure_sentinel", it is constructed with no arguments, so audit_stream_name is
always None and resolved_audit_stream_name fell back to the standard
resolved_stream_name. Audit logs then ingested into the access-log DCR stream
whose schema is built from StandardLoggingPayload, so Azure Monitor Logs
Ingestion silently dropped the audit-specific columns and audit rows arrived
effectively empty.
Add an AZURE_SENTINEL_AUDIT_STREAM_NAME env var fallback in __init__, mirroring
the AZURE_SENTINEL_STREAM_NAME idiom already used for the standard stream, so
audit logs can target a separate DCR stream without a custom callbacks file.
* fix(mcp): persist DCR client_id so interactive OAuth token refresh works
Interactive authorization_code MCP servers register an OAuth client via Dynamic
Client Registration (RFC 7591) during the authorize flow, but the minted
client_id and the discovered token_url were returned to the caller and never
written to the server row. The autonomous refresh_token grant reads client_id,
client_secret and token_url off the server, so an expired access token could not
be refreshed; the user was bounced back to re-authorize and tools/list returned
zero tools
Persist the DCR client_id (plus client_secret and token_endpoint_auth_method when
the registration returns them) and the discovered token_url onto the server row,
reusing the encrypt_credentials write that client_credentials and token exchange
already use, then refresh the in-memory registry so the value is live at refresh
time. Both the v1 refresher and the v2 AuthorizationCodeRefresher read those same
fields, so egress needs no change
* fix: reuse persisted MCP DCR clients
* fix: reuse persisted MCP DCR clients
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(tinyfish): make search provider permissive, attribute errors
Reshapes the TinyFish search provider so LiteLLM mirrors the TinyFish
Search API surface instead of maintaining a parallel cherry-pick.
Request side:
- Drop misleading request TypedDict
- Stop sending max_results on wire (TinyFish ignores it); clamp to [1,10]
client-side via self-threaded state
- Guard non-numeric max_results from bare ValueError
- Auto-JSON-encode dict params; lowercase bool serialization for ux-labs
Response side:
- Drop both Pydantic response models; parse directly into SearchResponse
so per-result extras flow through via extra="allow"
- Default missing title/url/snippet to "" instead of failing the call
- Read top-level parameter_warnings and re-fire as verbose_logger.warning
(pre-wired for upcoming TinyFish-side rollout; no-op today)
Error handling:
- Attributed _wrap_error helper at 3 call sites in transform_search_response
("TinyFish Search: <msg>. See https://docs.tinyfish.ai/search-api for
details.")
- Dispatch non-2xx responses through _wrap_error (fixes pre-existing bug
where 4xx/5xx silently returned empty SearchResponse)
- Wrap json.JSONDecodeError on 200 bodies
- Wrap pydantic.ValidationError for envelope-shape mismatches
Bug fix worth flagging: 4xx/5xx responses now raise an attributed
BaseLLMException instead of silently returning SearchResponse(results=[]).
Follow-up to #30634.
* fix(tinyfish): apply ruff format; guard OverflowError in max_results clamp
- Run ruff format on the touched files (CI lint job rejected the prior
commit's formatting).
- Add OverflowError to the except clause in the max_results clamp so
callers passing math.inf (or other non-finite floats) get the same
warn-and-ignore behavior as other malformed values. Greptile spotted
this in the first-pass review.
- Add test_max_results_infinity_float_warns_and_skips covering the
inf case.
* fix(tinyfish): apply --line-length 88 ruff format to match CI
CI uses 'ruff format --check --line-length 88'; my prior format pass
used the default line length, leaving several lines unwrapped. No
behavior change — purely whitespace.
* fix(tinyfish): reduce transform_search_response complexity; sort imports
CI's ruff strict-rule budget rejected the prior commit with:
- C901: transform_search_response complexity 16 > 10 (cap exceeded by 1)
- I001: import sort violation (cap exceeded by 1)
Extract two module-level helpers from transform_search_response to drop
its cyclomatic complexity:
- _default_missing_result_fields: in-place title/url/snippet defaulting
- _emit_parameter_warnings: defensive parameter_warnings reader
Auto-fix the import sort via ruff --fix.
No behavior change; the 59 existing tests still pass.
* test(tinyfish): cover defensive branches in _default_missing_result_fields
Codecov flagged 97.61% patch coverage (2 lines missing). The uncovered
lines were the non-dict raw_json and non-dict per-result item early-exits
in _default_missing_result_fields. Add two unit tests on the helper
directly to bring patch coverage to 100%.
* chore(tinyfish): apply ruff format to fix lint after staging merge
---------
Co-authored-by: Chenlu Ji <jichenlulu@gmail.com>
* fix(azure_ai): preserve content, tables, and keyValuePairs in doc-intelligence /v1/ocr
Azure Document Intelligence analyzeResult.content, .tables, and
.keyValuePairs were dropped when normalizing to the Mistral OCR schema.
They are now passed through verbatim as top-level response fields, and
the duplicated sync/async response parsing is consolidated into one
pydantic-validated helper.
Also adds the Azure DI features query param (list[str] or
comma-separated string, e.g. features=keyValuePairs) which Azure
requires for keyValuePairs extraction.
* test(azure_ai): replace fastapi jsonable_encoder with model_dump in ocr unit tests
The bulk-update entrypoints `/key/bulk_update` and `/team/key/bulk_update`
route through `_process_single_key_update`, not through
`_validate_update_key_data`, so the `permissions` gate LIT-4092 wired
into the single-key path never fires on bulk. Currently safe by
construction: `BulkUpdateKeyRequestItem` doesn't declare `permissions`
(Pydantic silently drops it), and `KeyUpdateFields` uses
`model_config = ConfigDict(extra="forbid")` (Pydantic 422s at parse
time). Neither structural barrier is enforced by tests on the field
itself; a future widening of either allowlist to include `permissions`
would reopen the class silently.
This wires `_check_permissions_caller_permission` into
`_process_single_key_update` right after `_validate_max_budget`, before
`prepare_key_update_data`. Zero behavior change today for any caller
routing through the current bulk request models; a defense-in-depth
gate for the class.
Tests:
- test_process_single_key_update_non_admin_permissions_rejected
- test_process_single_key_update_non_admin_permissions_explicit_empty_rejected
Both mutation-killed against removing the gate. Full mapped test file
(341 tests) green.
`_check_allowed_routes_caller_permission` previously keyed its
admin-only rule on truthiness. The refactor adds an `allowed_routes_was_provided`
keyword param that raw-body call sites populate from
`"allowed_routes" in data.model_fields_set`, so a caller that omits
the field (default flows through) is distinct from one that sends
any explicit value.
Four raw-body call sites now pass `allowed_routes_was_provided=...`:
`_common_key_generation_helper`, `generate_service_account_key_fn`,
`_validate_update_key_data`, and `regenerate_key_fn`.
Two derived-value call sites keep the pre-fix shape: the
post-`handle_key_type` recheck at `_common_key_generation_helper`
and the mirror in `regenerate_key_fn`. Both pass values produced by
`handle_key_type` (not by the request body), so `allowed_routes_was_provided` stays
False and the `allow_safe_presets=True` carve-out continues to accept
the `llm_api_routes` / `info_routes` presets.
In `regenerate_key_fn` the gate runs before the `premium_user`
license check, matching the LIT-4092 ordering.
`test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate`
pins the ordering; it fails on a swap of the two gates.
Tests in `tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py`
under `TestAllowedRoutesCallerPermission`:
- test_non_admin_generate_key_explicit_empty_allowed_routes_rejected
- test_non_admin_update_key_explicit_empty_allowed_routes_rejected
- test_non_admin_update_key_explicit_null_allowed_routes_rejected
- test_non_admin_regenerate_key_explicit_empty_allowed_routes_rejected
- test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate
- test_helper_accepts_derived_safe_preset_for_non_admin
- test_helper_rejects_derived_unsafe_preset_for_non_admin
- test_helper_rejects_when_provided_and_none_without_typeerror
The four attack-vector tests fail on the pre-fix HEAD and pass on this
commit. Three helper-level tests pin the derived-value branch and the
load-bearing None guard; each is mutation-killed against a targeted
change to the frozenset or the guard. Full mapped test file (347 tests)
green.
`NewUserRequest` and `UpdateUserRequest` inherit `permissions` from
`GenerateRequestBase`. `/user/new` passes the field into
`generate_key_helper_fn` which persists it on the auto-created key,
so an org admin who lands on `/user/new` (the route allowlist accepts
org_admin callers when the request body names an org where they hold
that membership) can mint a key with proxy-wide capabilities such as
`get_spend_routes`.
This wires the existing `_check_permissions_caller_permission` helper
into `new_user` and `_update_single_user_helper`. The helper's presence
check keys on `data.model_fields_set`, so an omitted field flows
through untouched and an explicit `{}` / `null` from a non-admin is
rejected 403 the same as any other value.
`_update_single_user_helper` is shared by `/user/update` and
`/user/bulk_update`, so both paths inherit the gate.
Tests in
`tests/test_litellm/proxy/management_endpoints/test_internal_user_endpoints.py`:
- test_new_user_non_admin_permissions_non_empty_rejected
- test_new_user_non_admin_permissions_explicit_empty_rejected
- test_new_user_non_admin_omits_permissions_succeeds (control)
- test_new_user_admin_can_set_permissions (control)
- test_update_single_user_non_admin_permissions_rejected
- test_update_single_user_non_admin_permissions_explicit_empty_rejected
The four attack-vector tests fail on the pre-fix HEAD and pass on this
commit. Full mapped test file (79 tests) green.
_enforce_key_and_fallback_model_access and can_key_call_resolved_model
both unconditionally skipped key-level model checks whenever
all-team-models was present, without verifying the key actually belongs
to a team. PR #29746 fixed the listing path (get_key_models) but these
two call-path checks were left untouched, letting teamless keys call
any model via /chat/completions while seeing an empty model list.
Add team_id is not None guard to both bypass conditions so teamless
keys fall through to can_key_call_model, which already correctly
rejects unresolvable all-team-models sentinels
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): bound per-request budget metric emission with a timeout (#31632)
* fix(prometheus): bound per-request budget metric emission with a timeout
Wrap the per-request budget-metric gather in asyncio.wait_for so a slow Redis or DB lookup cannot consume the whole LoggingWorker watchdog and get the success-logging event cancelled. On timeout the emission is skipped in isolation; budget gauges are still refreshed by the periodic cron. The timeout is configurable via PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT and defaults to 5.0 seconds, falling back to the default on an invalid value instead of raising
* fix(prometheus): reject non-finite and non-positive budget-metrics timeout env
float() accepts 0, negatives, nan and inf, which bypass the fallback: a value <= 0 makes asyncio.wait_for time out immediately and skip every per-request emission, and inf reintroduces the unbounded wait the timeout was meant to bound. Validate the parsed value is finite and greater than zero before using it, otherwise fall back to the default
* fix: report the blocked LLM response's real token usage (#31217)
When a guardrail blocks a post-call response, the synthetic violation response
reported hard-coded zero usage, discarding the token usage the upstream call
had already consumed.
Fix the root cause rather than re-counting tokens:
- Add an optional `original_response` field to ModifyResponseException.
- The unified guardrail's post-call success hook attaches the blocked LLM
response to the exception.
- The /v1/messages and OpenAI-format (/v1/chat/completions, /v1/completions)
block handlers report `original_response.usage` directly. Pre-call blocks
never invoked the LLM, so usage is zero.
Mock-based tests cover the helper (returns original usage / zero), the success
hook attaching original_response, and the endpoint reporting it end-to-end.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guardrails): buffer + cleanly terminate streamed responses on block (#31389)
Streaming moderation improvements for the unified guardrail post-call
streaming iterator hook:
- streaming_buffer_until_moderated: withhold all chunks until end-of-stream
moderation passes, then release the original response (clean) or only the
block message (blocked) -- the original content is never delivered on a
block. Snapshot chunks with a shallow list() copy (end-of-stream builds a
separate assembled response; chunks aren't mutated in place).
- Clean Anthropic SSE on block: synthesize a well-formed termination sequence
instead of a bare data: {"error": ...} blob that truncates the stream.
Provider-specific synthesis lives in AnthropicMessagesHandler via
build_block_sse_chunks (format-agnostic routing stays in the hook).
- Mid-stream blocks continue the in-progress message (close open content
block, append block message, terminate) rather than emitting a second
message_start, which clients reject. Standalone envelope only when no chunks
were sent (buffered path).
- ModifyResponseException imported under TYPE_CHECKING + locally at runtime to
avoid a module-level cyclic import.
Adds regression tests for buffering (content withheld on block) and mid-stream
continuation (single message_start).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: report real usage on streaming blocks, disable buffered mode for content-rewriting guardrails
- _standalone_block_chunks and _block_continuation_chunks now read real
token usage from ModifyResponseException.original_response instead of
hardcoding zero, matching the non-streaming _blocked_response_usage path.
Shared helper moved to guardrail_translation/utils.py.
- streaming_buffer_until_moderated is now forced off when the guardrail has
mask_response_content=True, since buffered replay releases the withheld
original chunks verbatim -- unsafe for a guardrail that rewrites content
(e.g. PII masking).
- Fix inverted streaming-flag precedence comment.
* style: ruff format after greploop fixes
* fix: handle Anthropic streaming guardrail blocks
* fix(responses): check terminal event type for streaming guardrail end-of-stream detection
_check_streaming_has_ended assumed responses_so_far held ModelResponse
objects with .choices, but for the Responses API the accumulated chunks
are raw SSE event dicts, causing an AttributeError on every call
* fix: preserve Anthropic blocked stream usage
---------
Co-authored-by: FERNANDO IZAR <fizar@me.com>
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* perf(lint): skip and cache base gate passes, parallelize make lint, skip redundant prisma generate
make pre-commit paid for a full second basedpyright pass over a merge-base
worktree on every run even when no rule was over its ceiling, re-generated an
unchanged Prisma client, and ran seven independent checks sequentially. The
basedpyright and ruff strict gates now skip the base pass when head is within
every limit (the same early-out type_discipline_gate already had), the
basedpyright base counts are cached under the git common dir keyed by
merge-base commit, pyrightconfig.json, and uv.lock, prisma generate only runs
when the schema or prisma version changed, and make lint fans its checks out
through a parallel sub-make after a single setup phase
* fix(lint): keep the base-cache scratch file out of the prune glob
The tmp+rename scratch in store_counts was named basedpyright-base-<hash>.json.tmp,
which the stale-entry prune glob (basedpyright-base-*) also matches, so a concurrent
lint run from another worktree sharing the same git common dir could unlink it between
write_text and replace and crash the gate with FileNotFoundError. The scratch is now
dot-prefixed so the glob can never see it, pid-suffixed so concurrent writers of the
same entry never share a scratch, and the prune glob is restricted to committed
*.json entries
* fix(bedrock): map guardrailConfig to InvokeModel guardrail headers
The InvokeModel API takes the guardrail identifier, version and trace as
X-Amzn-Bedrock-* request headers, unlike Converse which takes them in the
request body. The invoke transformer never set these headers, so
guardrailConfig was silently dropped (or leaked into the request body)
and Bedrock guardrails never ran on invoke-route models. Pop
guardrailConfig in AmazonInvokeConfig.validate_environment, validate it,
and set the headers before SigV4 signing; explicitly passed headers keep
winning over guardrailConfig so existing workarounds are unaffected
* fix(bedrock): reject guardrailConfig missing guardrailIdentifier
A guardrailConfig without guardrailIdentifier (e.g. an empty dict) would
validate, produce no guardrail headers, and let the request proceed with
guardrails silently not applied; that silent skip is the exact failure
mode this fix exists to remove, so fail fast with a 400 instead
* feat(tencent): add Tencent TokenHub as a provider
Tencent TokenHub is OpenAI- and Anthropic-compatible. This registers it as a
new provider: TencentChatConfig routes /v1/chat/completions and gates the
thinking/reasoning_effort params behind supports_reasoning, and
TencentAnthropicMessagesConfig routes the Anthropic-compatible Messages API.
Adds cost tracking, the deepseek-v4-pro/flash model entries, and provider
endpoint support metadata.
* test(tencent): add unit tests for Tencent TokenHub provider
Covers TencentChatConfig (chat completions) and TencentAnthropicMessagesConfig
(messages API) across transformation, param mapping, URL building, and header
validation, plus get_optional_params routing. Tests mock supports_reasoning to
stay independent of remote model cost data.
* fix(tencent): correct max_output_tokens and reuse parent messages env validation
Raise max_output_tokens/max_tokens for tencent/deepseek-v4-pro and tencent/deepseek-v4-flash from 8192 to 384000, matching Tencent TokenHub's published DeepSeek-V4 output limit; the 8192 value mirrored the native DeepSeek default and would have rejected valid larger requests before they reached Tencent
Delegate validate_anthropic_messages_environment to the parent via super() so the Tencent messages endpoint keeps content-type and anthropic-beta header injection instead of dropping them, keeping only the TENCENT_API_KEY resolution overridden
Add regression tests covering beta-header injection, the cost-calculator delegation, provider-info secret resolution, and validate_environment key handling
* fix(tencent): normalize messages URL when TENCENT_API_BASE has chat completions suffix
* fix(tencent): register tencent in models_by_provider
The provider was added to the LlmProviders enum and cost map but not to the
models_by_provider lookup, so test_models_by_provider (which asserts every
litellm_provider present in the cost map is registered) failed once the tencent
models were loaded. Add the tencent_models set, populate it from the cost map,
and expose it under the tencent key, mirroring deepseek.
* fix(tencent): import generic_cost_per_token from its canonical module
Import generic_cost_per_token from litellm.litellm_core_utils.llm_cost_calc.utils
instead of the top-level litellm.cost_calculator dispatcher, which imports the
tencent cost module at load time. Removing the back-reference avoids the circular
import and matches how deepseek and the other providers source the helper.
---------
Co-authored-by: Felipe Rodrigues Gare Carnielli <felipe.gare@hotmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
A 401 while listing tools (a missing or expired per-user OAuth token, or an
upstream 401 for any auth_type) was swallowed to an empty tool list, so a
single-server client got a 200 with no tools and no WWW-Authenticate challenge
instead of a 401 it could re-authenticate against. Only oauth pass-through and
delegate-to-upstream oauth2 servers surfaced it; every other auth_type, and the
missing-token case for all of them, masked it.
The surface-vs-absorb decision now keys on the route, not the auth_type. An
upstream 401 in _fetch_tools_with_timeout becomes an MCPUpstreamAuthError
regardless of auth_type, and the per-user OAuth challenge raised during client
creation (a bare HTTPException 401 carrying a WWW-Authenticate header) is
converted to the same type in _get_tools_from_server. The challenge is scoped
to 401: a 403 (authenticated but forbidden, e.g. insufficient scope) is not a
re-auth signal and degrades to an empty list like any other non-auth error, and
the stdio-allowlist 403 (no challenge header) stays absorbed. The existing
routing then does the right thing: single-server routes turn the error into a
401 + WWW-Authenticate, while the multi-server aggregator absorbs it to an empty
list so one unauthenticated server does not fail the whole listing.
On the UI tools page, an OBO (per-user authorization_code) server now shows the
Authorize gate when the list call returns 401, not only when no credential row
exists. The backend already refreshes a still-refreshable token on the list
call, so a 401 means there is no valid token and none could be minted (expired
with no usable refresh token), which is exactly when the user must reauthorize.
* fix(bedrock): honor ttl for tool_config cache injection points
Pass cache_control_injection_points control.ttl through to Bedrock
toolConfig cachePoint blocks, matching message/system cache behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex
is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a
manual update for every new Claude release (it already silently missed
Sonnet 5 and Fable 5). Replace it with a lookup against
cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json,
which AWS docs confirm tracks the same 1h-TTL-capable model set.
Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried
that pricing field (their own regional variants didn't have it), which
would have made the JSON-driven check wrongly grant them 1h TTL support.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests
test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and
test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id
(...-20250514-v1:0) that never shipped. This passed under the old regex-based
is_claude_4_5_on_bedrock, which matched on substring alone, but fails now
that it looks up cache_creation_input_token_cost_above_1hr in
litellm.model_cost, since the fake id has no pricing entry.
Also force the bundled local cost map in both tests so ttl eligibility reads
this branch's pricing data instead of the network-fetched main copy, which
lacks the fix until merge.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): restore cache and tool config compatibility
* fix(bedrock): preserve Sonnet 5 parallel tool config
* fix(bedrock): decouple parallel tool support from cache ttl
* refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns
Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and
bedrock_converse_supports_strict_tool_schemas (dead code) with a
supports_parallel_tool_use_config key in model_prices_and_context_window.json,
matching how is_claude_4_5_on_bedrock already reads
cache_creation_input_token_cost_above_1hr from the pricing JSON.
New models pick up parallel tool use support automatically when their
pricing entry ships with the key set, with no code change required
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test
anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in
model_prices_and_context_window.json, so
bedrock_converse_supports_parallel_tool_use_config returned False and the
test died with KeyError on additionalModelRequestFields. Use
jp.anthropic.claude-opus-4-7, a real entry that carries
supports_parallel_tool_use_config without 1h-TTL cache pricing, which is
exactly the decoupling this test exists to cover
* test(utils): allow supports_parallel_tool_use_config in pricing schema
The misc unit test job validates model_prices_and_context_window.json
against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects
unknown keys. Add the supports_parallel_tool_use_config key this PR
introduced so test_aaamodel_prices_and_context_window_json_is_valid
passes again
* fix(bedrock): preserve ttl for regional claude models
* fix(bedrock): fall back to base model entry when regional pricing lacks capability fields
Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit
cache_creation_input_token_cost_above_1hr shadowed the base entry that has it,
so is_claude_4_5_on_bedrock returned False and requested cache ttl values were
dropped for those deployments. Both capability lookups now consult the full
model id and the region-stripped base entry, matching the coverage of the old
name-pattern list. Also restores ToolBlock keyword construction for the
tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every
supported Python version
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Add an optional per-server max_concurrent_requests that caps how many
tool calls LiteLLM sends to one MCP server at once, so batch-processing
backends are not overwhelmed by unbounded parallel dispatch. Excess calls
queue on a per-server asyncio.Semaphore instead of being rejected. Unset
or non-positive means unlimited, preserving existing behavior.
Resolves LIT-2749
The security fix in 34e9be1ba7 removed turn_off_message_logging from
_supported_callback_params to stop callers bypassing global redaction via
the request body. That also killed the documented admin-only per-key or
per-team override because both flows resolve through the same allowlist
in initialize_standard_callback_dynamic_params.
Put turn_off_message_logging back in _supported_callback_params so an
admin-configured metadata.logging[].callback_vars.turn_off_message_logging
survives into StandardCallbackDynamicParams and can override the global
setting for that key or team, as documented at
docs/proxy/team_logging#disableenable-message-redaction.
Consolidate the metadata traversal so the extractor and the proxy strip
walk the same set of client-controllable slots. iter_client_callback_metadata_dicts
in litellm_core_utils/initialize_dynamic_callback_params.py is the single
source of truth for metadata, litellm_metadata, and litellm_params.metadata;
_strip_client_message_redaction_opt_out imports it so a future addition
to one side automatically reaches the other. The extractor iterates the
helper in reversed order so litellm_params.metadata keeps overriding
metadata, matching the pre-refactor merge precedence.
Client bypass stays blocked. Restoring the field re-enrolls it in the
auth layer's _BANNED_REQUEST_BODY_PARAMS (derived from
_supported_callback_params via _build_banned_observability_params), so
client submissions at the top level, inside metadata, or inside a
JSON-string litellm_metadata all 401 at ingress. is_request_body_safe
also now descends into litellm_params.metadata for the same 401 defense
against the nested-body attack vector, matching how the metadata and
litellm_metadata slots are handled. _strip_client_message_redaction_opt_out
runs after the litellm_metadata JSON parse and before the admin callback_vars
unpack, so admin values survive while any leftover client-supplied
opt-out is dropped when global redaction is on and the key or team
lacks allow_client_message_redaction_opt_out.
Flip the two dynamic-param e2e tests added by the security fix to
reflect the restored override behavior, keeping the invariant that
proxy client bypass is stopped by the auth layer 401 above.
Co-authored-by: yucheng <yucheng@yuchengs-MBP.attlocal.net>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
For LITELLM_METADATA_ROUTES (responses, /v1/messages, batches, etc.),
the proxy stores admin metadata under data["litellm_metadata"] while
user-supplied metadata lives in data["metadata"]. Tags placed in
metadata.tags by the caller were never merged into litellm_metadata.tags,
causing SpendLogs.request_tags to be empty on these routes
Closes#31584
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): gate OAuth authorize/token/register/discovery on auth_type=oauth2
A non-oauth2 MCP server (notably auth_type=none, access-group gated) has no
client_id and no authorization URL, yet the gateway OAuth endpoints did not
check auth_type. authorize() raised "client_id is required" before the
auth_type was ever examined, and the .well-known discovery builders always
advertised authorization_servers / authorization_endpoint / token_endpoint /
registration_endpoint, so spec-compliant MCP clients were pointed at an OAuth
flow that can never succeed.
Add an auth_type != oauth2 guard to the authorize, token, register,
protected-resource and authorization-server paths (covering the internal UI
OAuth endpoints too). The discovery guard sits after the OAuth pass-through
branch so genuine pass-through servers keep proxying their upstream metadata.
oauth2 servers are unaffected.
* fix(mcp): accurate non-oauth2 message; 404 unknown discovery names to close enumeration oracle
Address review feedback on the auth_type gate.
The 400 message no longer claims access is governed by access groups, which is
only true for auth_type=none; it now states that the gateway runs the OAuth
client_id/authorize/token/register flow only for oauth2 servers and that the
server is reached using its configured auth_type, which is accurate for every
non-oauth2 type (api_key, oauth2_token_exchange, etc.).
The discovery gate previously 404'd a named non-oauth2 server but still returned
200 metadata for an unknown name, which both serves a broken document for a typo
and lets an unauthenticated caller enumerate non-OAuth server names by comparing
404 vs 200. A named discovery request now returns 200 only when it resolves to an
oauth2 server; unknown (or hidden) and non-oauth2 names return the same 404. Root
discovery and pass-through servers are unaffected.
* Apply suggestions from code review
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Route non-full-admin callers through _sanitize_mcp_server_list_for_non_admin,
matching the pattern the fetch and list handlers adopted. Replace the two
regression tests that pinned the old partial-blank behavior with a
sanitize/full-admin pair mirroring the fetch/list coverage.
Resolves LIT-3929
* fix(mcp): roll up MCP tool spend to user counters and usage UI
Direct REST MCP tool calls now fire success logging so spend_logs and
user/team rollups include configured mcp_server_cost_info charges.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): gate key-info enrichment to requests missing user_id; fix import order
- Only call _enrich_failure_metadata_with_key_info when user_api_key_user_id is
absent, avoiding a cache/DB lookup on every normal LLM request.
- Move LiteLLMProxyRequestSetup import to correct alphabetical position (I001).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): scope MCP spend aggregate by api_key to prevent cross-tenant disclosure
Add api_key = ANY($2) to the MCP session aggregate query so it is
bounded by the same ownership already applied to the main page query.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix spend logs for call and list mcp tools
* Add tags in mcp logging
* Fix ruff
* fix(lint): replace List/Dict with list/dict in new annotations (UP006)
Replace the 8 new UP006 violations introduced by the mcp-tags changes:
- Optional[List[str]] → Optional[list[str]] for request_tags params
- List[str] return type → list[str] in _get_parent_request_tags
- Dict[str, Dict[...]] → dict[str, dict[...]] for mcp_spend_map annotation
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): keep call_tool_rest_api within complexity budget and narrow MCP spend enrichment except to PrismaError
* fix(mcp): keep final streaming chunk when draining inner stream fails
* fix: handle MCP logging edge cases
* fix: propagate MCP logging cancellation
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* fix: p0 issues, added types and shared functions for each test suite
* style: carry clearer status_code comparison into renamed e2e dir
* refactor: migrate to gateway client
* fix: add new tests, split gateway
* test(e2e): add live batches suite across providers and routing scenarios
* test(batches): cover real cost tracking on completed batch retrieve
* test(e2e): assert managed vs raw file and batch id shapes per routing scenario
* test(e2e): assert full response shape of each batches and files endpoint
* test(e2e): only accept transitional statuses for a freshly created batch
* test(prompt-factory): make test_convert_url deterministic with a data URL
picsum.photos is down (HTTP 522), so test_convert_url failed on every
run. Swap the live external image for an inline data: URL and assert the
round-trip through convert_url_to_base64 genuinely.
A data URL is already inline base64 image data, so convert_url_to_base64
now short-circuits it instead of attempting an impossible HTTP fetch;
add a regression for that branch in the mapped image_handling test
* fix: pass through async image data urls
* fix(image-handling): short-circuit data URLs in async path too
Bugbot flagged that convert_url_to_base64 returns data: base64 URLs
unchanged but async_convert_url_to_base64 still tried to fetch them,
so async OCR flows (Bedrock, Azure) would reject inline images the sync
path accepts. Add the same guard to the async function and a regression
test that asserts the async path returns the data URL without touching
the HTTP client
* Fix: openai batches lifecycle
* Fix: add e2e azure openai tests
* Fix e2e for vertex ai
* Add all models for testing
* test(managed-files): assert idempotent upsert in store_unified_file_id
store_unified_file_id switched from create to upsert to avoid
UniqueViolationError when re-storing the same unified_file_id (e.g.
batch output files stored before metadata is available). Update the
unit test to assert the upsert call and its create payload instead of
the removed create call.
* test(batches): reconcile vertex_ai native batch-id comment with fallback guard
* fix(test-config): keep rust-ocr models in model_list by moving files_settings after it
* fix(test-config): move batch models after OCR block to keep merge with internal_staging clean
* fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: ruff format transformation.py and endpoints.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition
* test(vertex-ai/batches): align completion_window assertion to 24h
* fix: update managed file metadata on upsert
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ui): show info message when MCP tool preview returns 403
Internal users submitting MCP servers hit an admin-only preview endpoint; replace the red connection error with a clear review notice while leaving other failures unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): let BYOM submitters see their approved servers
Approved user-submitted MCP servers defaulted to no access groups and allow_all_keys=false, so submitters could not see them after admin approval. Grant creator visibility for active submissions in get_allowed_mcp_servers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Improve dialogue box
* fix(security): restrict MCP semantic filter settings to proxy admins
Add an explicit PROXY_ADMIN check on PATCH /update/mcp_semantic_filter_settings
and hide Semantic Filter and Network Settings tabs from non-admin users in
the MCP Servers UI.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): use list[str] instead of List[str] to satisfy UP006 budget
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(mcp): cache BYOM submitter server lookup with 60s TTL
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: fix ruff format and prettier formatting
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: preserve approved BYOM server visibility
* fix(mcp): keep no-mcp-servers opt-out absolute and gate BYOM union by key scope
The autofix in 94fd2bf made the no-mcp-servers sentinel return the caller's
submitted BYOM servers, which weakened an explicit key-level opt-out into a
soft preference. Restore the absolute opt-out and additionally skip the BYOM
union for keys with an explicit object_permission.mcp_servers list and for
toolset-scoped requests, mirroring how allow_all_keys servers are handled.
Add unit tests for the sentinel, explicit scoping, toolset scope, the cache
invalidation helper, the cache-miss DB path, and the db.py query helper.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(bedrock): drop strict/additionalProperties from toolSpec for Claude Sonnet 4
Claude Sonnet 4 on Bedrock Converse rejects toolSpec.strict and
additionalProperties the same way Opus 4.7/4.8 do. Add
bedrock_converse_supports_strict_tools: false to all Sonnet 4 regional
variants so those fields are suppressed before the request is sent.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(bedrock): assert additionalProperties dropped for strict-unsupported models
Rename the regression test to reflect Opus 4.7/4.8 and Sonnet 4 coverage,
and assert both strict and additionalProperties are stripped from toolSpec.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(fireworks): skip embeddings live test when provider account is suspended
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <noreply@anthropic.com>
* feat(vertex_ai): pass full imageConfig dict for Gemini image generation
Support all ImageConfig fields (aspectRatio, imageSize, personGeneration,
imageOutputOptions) when calling Vertex AI Gemini image generation endpoints.
Previously only aspectRatio and imageSize were extracted; other fields were
silently dropped.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: ruff format vertex_gemini_transformation
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(vertex_ai): warn on non-dict imageConfig instead of silently dropping
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): trigger Nova Sonic generation on response.create so realtime sessions stop hanging (LIT-2239)
* fix(bedrock): reopen audio content at client sample rate after trigger block
* test(bedrock): cover realtime handler disconnect flush and stream-end guard
* fix(bedrock): always close realtime input stream even if close flush fails
* fix(lint): use contextlib.suppress in bedrock realtime cleanup to satisfy BLE001 budget
* fix(bedrock): suppress bedrock close send errors per-message so promptEnd/sessionEnd still flush
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(bedrock/converse): drop toolSpec.strict for Opus 4.7/4.8
Bedrock Converse routes Claude Opus 4.7/4.8 through an Anthropic-compatible
validator that maps toolSpec to the native tool shape and rejects the extra
`strict` key with `tools.N.custom.strict: Extra inputs are not permitted`,
even though Anthropic's native API accepts `strict` as a top-level tool field
for the same models. Sonnet 4.5/4.6 and Opus <=4.6 accept `toolSpec.strict`
unchanged.
The existing gate `get_bedrock_base_model(model).startswith("anthropic")`
(introduced in #29814 to forward `strict` for Claude on Bedrock Converse) is
too broad and regressed Opus 4.7/4.8 callers — see #31582.
Replace the inline check with a small `bedrock_converse_supports_strict_tools`
helper that excludes the Opus 4.7/4.8 family from strict forwarding. All
other Anthropic models on Bedrock keep the existing behavior.
Closes#31582.
* fix(bedrock/converse): move strict-tools regression to a clean test file
The original regression test was added to
test_litellm_core_utils_prompt_templates_factory.py, which has
pre-existing ruff-format violations throughout (multi-line asserts that
fit on one line). The lint workflow runs `ruff format --check` on
changed files only, so touching that file surfaces those pre-existing
violations and fails CI for unrelated reasons.
Move the #31582 regression coverage into a new dedicated test file so
the format check stays green. Also collapses the helper's `not any(...)`
onto a single line to satisfy ruff format.
Covers: #31582
* refactor(bedrock/converse): drive strict-tools gate from model cost map
Replace the hardcoded Opus 4.7/4.8 pattern list with a
bedrock_converse_supports_strict_tools flag on the affected entries in
model_prices_and_context_window.json, resolved via get_model_info with a
local cost map fallback, so future models with the same restriction only
need a JSON update
* chore: revert unrelated credential_migration.py reformat
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
The handler returned decrypted callback environment values and alerting
routing values verbatim to callers who were not full PROXY_ADMIN. Gate
those on full-admin role, matching the posture used on the sibling
config-inspection endpoints. Non-sensitive routing fields (host / base
URL / port style values) stay visible so the UI can still label which
integration is wired up. Full PROXY_ADMIN sees everything unchanged so
the edit form round-trips on save.
Resolves LIT-4115.
* fix(model_prices): apply claude-sonnet-5 introductory pricing through 2026-08-31
Anthropic launched Sonnet 5 with introductory pricing of $2/$10 per million
input/output tokens through August 31, 2026 (sticker price $3/$15 applies
from September 1, 2026). Bedrock, Vertex AI, and Azure Foundry mirror the
introductory rate. LiteLLM was charging the sticker price on all ten
claude-sonnet-5 entries, over-billing by 50% during the introductory period.
Update input, output, cache write (5m and 1h), and cache read costs on the
base entries to the introductory rate, and keep the 10% cross-region premium
on the us/eu/au/jp Bedrock inference profiles on top of it. Also add an
anthropic-sonnet-5 entry to the dev proxy config.
* test: document exact sticker prices to restore on 2026-09-01
* feat(gdc): add Google Distributed Cloud Gemini provider support
Introduce support for the Google Distributed Cloud (GDC) Gemini provider by adding "gdc" to the list of chat providers and enabling the gdc/ model prefix. The implementation defines a new GDCGeminiConfig class which handles authentication via Google Distributed Cloud service account credentials, manages token generation, formats GDC Gemini request URLs, and transforms request structures accordingly
The PreProcessNonDefaultParams class is also updated to exclude vertex parameters from filtering when the custom LLM provider is GDC, allowing vertex parameters to be passed properly during GDC initialization
* fix: resolve issues identified in PR #30702
* fix(gdc): harden credentials, fix vertex param filtering, add tests
The supports_vertex_params branch regressed vertex_ai and vertex_ai_beta: the `if custom_llm_provider in [...]: pass` was a no-op, so those providers fell through to the config lookup, found no supports_vertex_params, and had their vertex_ params stripped. The check is now a single _provider_supports_vertex_params helper that keeps vertex_ params for the vertex family and for any config that opts in, and only swallows the expected ValueError from an unknown provider string instead of a blanket except
GDC project and location now resolve from the deployment's litellm_params and the litellm.vertex_project / litellm.vertex_location globals before falling back to request optional_params, matching how vertex_ai resolves them, so a proxy caller can no longer route a request to a project the deployment did not expose
A request api_key is no longer treated as a filesystem path, so a caller can't make the host open a local service-account file; api_key must be a literal service-account JSON string or a bearer token
The opt-in token cache is hardened: the lock and cache dict are created in __init__ instead of via a racy hasattr lazy-init, the token is read inside the lock, and the audience is stripped of a trailing slash once so the cached and non-cached paths agree
Also declares gdc_api_base, switches the lazy-import entry to the relative path every other entry uses, adds the missing trailing comma in the provider config map, and drops the api_base fallback that only ran when api_key was None
Adds unit tests covering the vertex-param filter, deployment-over-request precedence, the api_key file-path rejection, URL construction branches, environment validation, token caching, and the gdc completion dispatch; transformation.py is fully covered
* fix(gdc): prefer GDC-specific config, honor vertex_ai aliases, harden URL and bool parsing
* fix(gdc): mint the GDCH token audience from the host, not the full base
When api_base embedded /v1/projects/... and the deployment set project/location, get_complete_url rebuilt the request URL from the host while validate_environment still derived the token audience from the full original api_base, so the bearer token could target a different audience than the URL actually called. The audience is now the scheme://host of api_base in every case, matching the host get_complete_url builds against
* fix(gdc): restrict JSON api_key to GDCH service accounts
Only accept a credential whose type is gdch_service_account before
calling google.auth.load_credentials_from_dict, so a caller-supplied
external_account/identity_pool/pluggable credential carrying arbitrary
token or credential_source endpoints is rejected before any token
refresh runs. GDC only ever uses GDCH service accounts, and non-GDCH
credentials could not have completed auth anyway (with_gdch_audience is
GDCH-only), so this narrows the credential-refresh surface without
changing valid GDC behavior.
* fix(gdc): validate project and location as plain identifiers
vertex_project and vertex_location can come from request params and were
interpolated as raw path text into the GDC request URL and the
x-goog-user-project header. A caller-supplied value containing / ? # or
.. could reshape the path and make the proxy send its GDC-authorized
request to a different endpoint under the configured host. Validate both
against a strict identifier pattern before building the URL or header and
raise an auth error otherwise; GCP project ids and locations are plain
identifiers so valid deployments are unaffected.
* fix(gdc): bind x-goog-user-project quota header to the deployment
The quota project header was resolved with request-level vertex_project
taking effect, so with a preformed deployment api_base a caller could set
vertex_project to a different project and have it sent under the proxy's
GDC credential, misattributing quota or billing. Resolve the header
project the same way the URL is resolved: a preformed api_base without a
deployment override binds to the project embedded in the URL, otherwise
deployment and global config win over request params. This keeps the URL
and the quota header consistent.
* fix(gdc): always rebind x-goog-user-project, stripping caller-forwarded values
The quota project header was only set when absent, so with client header
forwarding an authenticated caller could send their own
x-goog-user-project (any casing) and have it ride on the proxy's GDC
credential, bypassing the deployment-derived binding. Strip every casing
of the header and always set it from _effective_project before the
request is signed.
* fix(gdc): make a preformed api_base authoritative for project routing
get_litellm_params copies caller-supplied vertex_project and vertex_location into litellm_params via OPTIONAL_KWARGS_KEYS, so litellm_params cannot be treated as a deployment-only source. The previous _deployment_overrides_path inference let an authenticated caller flip a pinned preformed api_base such as /v1/projects/pinned/... to /v1/projects/attacker/..., driving requests to a caller-chosen project with the proxy's configured GDC credentials and quota header
A preformed /v1/projects/ api_base is now authoritative; get_complete_url returns it unchanged and _effective_project binds the x-goog-user-project quota header to the project embedded in that URL, so a caller can no longer redirect a pinned deployment or move the quota header off it. The two tests that asserted the override behavior are now regression tests that fail if the rewrite is reintroduced
* fix(gdc): make a preformed api_base self-sufficient in get_complete_url
get_complete_url resolved and required a params-derived vertex_project before returning a preformed /v1/projects/ api_base, so a deployment that pins its project in the api_base path was forced to also pass vertex_project or hit 'project is required'. validate_environment already extracts the project from a preformed URL and needs no such param, so the two paths disagreed
The preformed-URL early return now runs before project/location resolution, matching validate_environment: a preformed api_base is returned as-is with no redundant param, and non-preformed bases still require vertex_project and vertex_location as before. Adds a regression test that a preformed base with no project/location params returns the URL unchanged
---------
Co-authored-by: Paige O'Connor <lostpaige@google.com>
Co-authored-by: Tim Laubach <tlaubach@google.com>
* feat(github_copilot): route /v1/messages to Copilot native Anthropic endpoint
Add a GitHub Copilot Anthropic Messages transformation that routes supported Claude models through the native /v1/messages endpoint. This covers request URL construction, default headers, and supported model metadata.
* fix(github_copilot): address PR review feedback
Tighten the Anthropic Messages environment validation and web search interception behavior after review feedback. Avoid treating non-web-search requests as web-search-only paths.
* style(github_copilot): apply black formatting
Apply Black formatting to the GitHub Copilot Anthropic Messages tests.
* test(github_copilot): cover ProviderConfigManager dispatch for Anthropic Messages
Add coverage for ProviderConfigManager dispatch when GitHub Copilot models use the Anthropic Messages API, including non-Anthropic models returning no config.
* fix(github_copilot): apply messages-proxy intent header to /v1/messages
Set the messages-proxy interaction header for GitHub Copilot Anthropic Messages requests so /v1/messages uses the expected Copilot intent.
* fix(github_copilot): use modern generic annotations
Replace legacy typing generics in the GitHub Copilot Anthropic Messages transformation so the strict Ruff budget gate stays within its ceiling.
* refactor(github_copilot): decouple web-search short-circuit and harden /v1/messages URL
Address review feedback on the Copilot native Anthropic messages path.
Replace the hardcoded LlmProviders.GITHUB_COPILOT check in the web-search
interception handler with a handles_web_search_natively() method on
BaseAnthropicMessagesConfig (default True), overridden to False in
GithubCopilotAnthropicMessagesConfig. Provider-specific behavior now lives in
llms/ and the handler stays provider-agnostic, so a future provider in the same
situation needs no carve-out here.
In get_complete_url, reuse the already-resolved api_base returned by
validate_anthropic_messages_environment instead of reading the authenticator a
second time, removing redundant I/O and the mid-request inconsistency window.
The caller-supplied base is still discarded in validate, which is the security
boundary. Normalize a trailing slash on the base in both methods so a
tenant-specific host never yields a double-slash //v1/messages URL.
* fix(github_copilot): forward anthropic-beta headers on /v1/messages
The Copilot config inherited should_filter_anthropic_beta_headers()==True
from BaseAnthropicMessagesConfig, so update_headers_with_filtered_beta
stripped every anthropic-beta value after validate_anthropic_messages_environment
injected them (github_copilot has no mapping in
anthropic_beta_headers_config.json). That silently disabled header-gated
features like context_management and structured outputs on the native
passthrough. Override the hook to False, matching OpenAILikeAnthropicMessagesConfig.
* test(github_copilot): remove dead branch in beta-header regression test
The anthropic-beta filtering test guarded the fix with an if branch on
should_filter_anthropic_beta_headers(), which is always False, so the branch was
unreachable. Replace it with a direct assertion that running the provider-scoped
filter for github_copilot strips every beta value, proving why the override is
load-bearing and catching a regression that flips it back on.
---------
Co-authored-by: ririnto <ririnto@kakao.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The `_check_permissions_caller_permission` helper introduced in
#31469 was only wired into `_common_key_generation_helper`. This
change wires it into `_validate_update_key_data` and `regenerate_key_fn`
so the three write paths share the admin gate, and refactors the
helper to accept the full request model so it can key on
`"permissions" in data.model_fields_set` rather than truthiness. The
presence check keeps the model-level omit default flowing through
unchanged while treating any explicit value (including `{}` / `null`)
as an admin-only write.
In `regenerate_key_fn` the gate is placed before the `premium_user`
license check so the rejection is consistent across premium and
non-premium deployments. That ordering is pinned by
`test_regenerate_key_non_admin_permissions_rejected_before_enterprise_gate`
Tests in tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py:
- test_update_key_non_admin_permissions_non_empty_rejected
- test_update_key_non_admin_permissions_explicit_empty_rejected
- test_update_key_non_admin_permissions_explicit_null_rejected
- test_update_key_non_admin_omits_permissions_succeeds (control)
- test_update_key_admin_can_set_permissions (control)
- test_regenerate_key_non_admin_permissions_rejected
- test_regenerate_key_non_admin_permissions_explicit_empty_rejected
- test_permissions_explicit_empty_rejected_for_non_admin_on_generate
- test_regenerate_key_non_admin_permissions_rejected_before_enterprise_gate
Mutation-killed against gate removal on either wire, against reverting
the helper to a truthiness check, and against reordering the gate past
the enterprise-license check
* fix(proxy): authorize /health/test_connection against loaded deployment's team_id (VERIA-441)
POST /health/test_connection looked up a deployment by request-supplied model_info.id, dumped its
litellm_params (including api_key) into the outbound probe, merged request params over it, and then
authorized the call against the caller-supplied model_info.team_id. A team admin could pass another
team's deployment id together with their own team_id and an attacker-controlled api_base, sending
the victim team's provider key to that URL.
Capture the loaded deployment's model_info alongside its litellm_params in both the id-lookup and
the model_name fallback paths, and pass that captured value to can_user_make_model_call. When no
deployment is loaded (caller is probing fresh, request-supplied credentials), keep using the
request body's model_info; no foreign deployment is in scope and the existing role check still
requires admin or team-admin.
Add two regression tests that wrap (not mock) ModelManagementAuthChecks.can_user_make_model_call,
one per resolution path, asserting HTTP 403 and that the auth check was reached with the loaded
deployment's team_id. Both fail on the pre-fix code.
* test(health): add positive-path regression through real auth (VERIA-441)
The two deny tests already exercise the real (wrapped) ModelManagementAuthChecks.
Add a matching positive-path test so a mutation that swaps the auth team_id for
a deny-all value on the legit path also fails: loaded deployment owned by team-X,
caller admin of team-X -> asserts HTTP 200 and that the auth check ran with the
LOADED deployment's team_id.
* refactor(test): rename health endpoint tests for clarity (VERIA-441)
Rename test functions and variables from attacker/victim/owner framing to
neutral team-a/team-b terminology. Update docstrings to remove exploit-specific
language. Tests remain functionally identical, covering deny paths (cross-team
deployments) and the positive path (same-team deployments).
Register bedrock_mantle/xai.grok-4.3 with /v1/responses in
supported_endpoints so the data-driven gate routes it through
BedrockMantleResponsesAPIConfig (which inherits SigV4 signing via
BedrockMantleAuthMixin). Without this entry the model falls through to
None and forces bearer-token-only auth.
Pricing sourced from AWS Bedrock pricing page.
Closes#31196
Co-authored-by: unknown <>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
On macOS, tempfile.mkdtemp returns a path under /var/folders, a symlink
to /private/var. The base pass in type_check_gate.py resolved each
diagnostic path (yielding /private/var/...) but not the worktree root,
so relative_to raised ValueError for every diagnostic, base counts came
back empty, and the vacuous-run guard failed every local
make lint-basedpyright run. type_discipline_gate.py already resolves
root the same way; ruff_strict_gate.py counts rule codes without
touching worktree paths, so it is unaffected. CI runs Linux where the
temp dir is not a symlink, which is why this only bit local macOS runs
* chore(lint): raise basedpyright per-rule slack to 50% of baseline
The per-rule ceilings in basedpyright-code-budget.json sat at roughly 10% slack over baseline, which several in-flight PRs are already bumping into. Raise the slack on every rule to at least 50% of its baseline so there is ample headroom for a long while, while never lowering any rule that already had more generous slack (e.g. reportReturnType stays at 100).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(lint): collapse type/lint budgets to a single per-rule limit
The three non-frontend budget files (ruff-strict, type-discipline, basedpyright-code) tracked a per-rule baseline and slack whose sum was the ceiling. Nothing consumed the split beyond that sum, so this replaces both keys with a single limit equal to the old baseline + slack; the original baselines live in git history if anyone needs them.
The gate scripts and the ratchet guard now read limit directly. lint-budget-update no longer re-captures raw counts; it ratchets each rule's limit down by the number of violations this branch cleared since its branch point (the merge-base), so the granted headroom shrinks by exactly what was fixed and a limit never rises. The ratchet guard reads either schema so it still compares correctly across the migration boundary.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore(lint): surface staged-vs-working parity for pre-commit and budget-update
make pre-commit selects which checks to run from the staged index but runs the linters over the working tree, so unstaged edits to tracked files and untracked files skew a green/red away from what a commit of only the staged changes would produce. There is no safe in-place way to lint the index, so the script now warns when unstaged or untracked changes are present, and CLAUDE.md documents that you must stage everything first for both make pre-commit and make lint-budget-update to predict CI correctly.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* docs(lint): list type-discipline budget in lint-budget-update instruction
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
MCP servers created through the UI are persisted to the database independent of
store_model_in_db, but the in-memory registry that GET /v1/mcp/server reads was
hydrated from the database only through add_deployment, which runs solely when
store_model_in_db is True. On a DB-backed single-instance proxy with
store_model_in_db unset the registry started empty after a restart, so the MCP
Servers page showed nothing until an add or edit triggered a reload.
Hydrate the registry from the database on startup regardless of store_model_in_db
via a new ProxyConfig.init_mcp_servers_from_db, honoring supported_db_objects.
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
Include top-level scalar fields from standard_logging_metadata in the
combined metadata dict used by custom_prometheus_metadata_labels. Previously
only nested sub-dicts (requester_metadata, user_api_key_auth_metadata,
spend_logs_metadata) were spread into combined_metadata, so fields like
user_api_key_project_alias were inaccessible and always resolved to None.
Co-authored-by: unknown <>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): audit remaining system-wide settings updates
Extends the audit logging framework introduced in the parent PR to the
rest of the LiteLLM_Config writers and the two adjacent settings tables:
/config/update (general, environment_variables, litellm_settings,
router_settings sections), /config/field/update, /config/field/delete,
/config/callback/delete, /update/default_team_settings,
/update/mcp_semantic_filter_settings, /add/allowed_ip,
/delete/allowed_ip, /update/sso_settings, /update/ui_theme_settings,
/update/ui_settings.
Each writer records the actor, action, the affected config section, and
a redacted before/after snapshot. SSO and UI settings rows use their own
table_name (LiteLLM_SSOConfig, LiteLLM_UISettings). The /config/callback
and /update/sso_settings audits fire BEFORE the proxy reload and the env
cleanup step respectively, so a failure in either leaves the audit row
intact.
The audit-actor parameter on _update_litellm_setting is now required
rather than optional; the chokepoint covers default_team and
mcp_semantic_filter for free, and a future caller that forgets the
actor fails loudly instead of silently skipping the audit. The two
direct-calling tests pass a dummy actor.
The environment_variables section redacts every value rather than
relying on key-name matching, because it carries credentials under
non-secret-looking uppercase keys (e.g. DATABASE_URL).
* fix(proxy): capture redacted SSO before-snapshot in audit log
Greptile review of #31754 flagged update_sso_settings as the one endpoint
where before_value is permanently None, so the LiteLLM_SSOConfig audit
trail has no pre-change state. An auditor reviewing a secret-rotation
event could see what the SSO settings were changed to but not what they
were before.
Read the existing SSO row before the upsert, decrypt it via
proxy_config._decrypt_db_variables, and pass it as before_value.
create_config_audit_log's secret-name redaction then masks the
*_client_secret fields, so neither the old nor the new plaintext secret
lands in the audit row.
Add a regression test asserting the before-snapshot reflects the
pre-change values for non-secret fields (google_client_id) and is
redacted for secret fields (google_client_secret). Mutation-checked
against reverting to before_value=None.
The pre-existing SSO tests now also mock litellm_ssoconfig.find_unique
since the endpoint reads it; the read returns None for tests that do not
care about the before-state.
* fix: remove committed zero init migration
* refactor(proxy): audit config writes via asyncio.create_task everywhere
PR A's chokepoint audit call was refactored from a blocking await to
asyncio.create_task so that a post-save audit-log failure could not
surface as a 500 to the caller. The 12 other audit call sites added in
this PR were still using await, reintroducing the exact 500-after-commit
exposure at every sibling endpoint. Wrap them all in asyncio.create_task
to match the model_management_endpoints / key_management_endpoints /
hooks / config_override_endpoints / team_callback_endpoints /
cache_settings_endpoints house pattern, so the codebase tells one story.
The two direct-invocation tests (test_update_config_general_settings and
test_delete_config_general_settings, which call the handler in-process
rather than via TestClient) yield with `await asyncio.sleep(0)` after the
handler returns so the scheduled audit task runs before the assertion.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: add cache control injection support for v1/messages endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: normalize string content to list for Anthropic-native cache_control injection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: simplify cache control injection, fix system=[] bug, fix handler system type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: extract cache control logic into static helper on AnthropicCacheControlHook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails/headroom): add CCR (compress-cache-retrieve) support via agentic loop
When Headroom's /v1/compress returns messages containing hash markers
(hash=[a-f0-9]{24}), inject a headroom_retrieve tool into the request.
When the LLM calls that tool, intercept via async_should_run_agentic_loop
and async_build_agentic_loop_plan, call GET /v1/retrieve/{hash} on the
Headroom sidecar, and replay the LLM with the original content as a tool
result -- all transparent to the caller.
* style: run ruff format on headroom guardrail and tests
* fix(guardrails/headroom): detect headroom_retrieve calls in both OpenAI and Anthropic response formats
* test(guardrails/headroom): add test for Anthropic content block format detection in CCR loop
* ci: trigger CI checks
* fix(guardrails/headroom): replace List/Dict with list/dict to fix UP006 ruff violations
* fix(guardrails/headroom): replace except Exception with except ValueError to fix BLE001
* fix(guardrails/headroom): add Responses API output format detection for CCR tool calls
* refactor(guardrails/headroom): extract format-specific helpers to fix C901 complexity
* fix(guardrails/headroom): scope CCR retrieval to hashes produced by current request
Previously any LLM-supplied hash in a headroom_retrieve tool call was
forwarded to the Headroom retrieve API, letting a crafted tool call
fetch arbitrary cached content. Validate the hash against the set
produced by compressing the current request's messages before calling
retrieve.
* fix(guardrails/headroom): track issued hashes server-side, fix Responses API replay shape
Hash validation now also checks an in-memory cache of hashes actually
returned by /v1/compress, not just whether the hash text appears
somewhere in the request's messages. The message-text check alone is
forgeable: an attacker can plant a hash-shaped string in their own
prompt and have it treated as valid.
Responses API follow-up now emits function_call/function_call_output
items keyed by call_id instead of chat-style assistant/tool messages,
since the Responses API does not accept the latter as input. Also
fixes call_id/id field priority when extracting tool calls from
Responses API output, since call_id (not id) is what must match
between the function_call and its output.
* fix(guardrails/headroom): drop redundant quoted type annotations
UP037 flags quotes on annotations that are already lazily evaluated
via `from __future__ import annotations`.
* test(guardrails/headroom): add missing pytest.mark.asyncio decorators
Functional under asyncio_mode=auto, but every other async test in the
file has the decorator for consistency.
* fix(guardrails/headroom): scope CCR hashes per call_id, fix Anthropic replay shape
Two real gaps found in review:
1. The instance-wide issued-hash cache combined with a message-text
check did not actually scope retrieval to the request that produced
the hash. A hash issued for request A stays in the shared cache
until TTL expiry, and the message-text check is satisfied by any
request whose own messages happen to echo that hash string. Request
B could plant A's hash in its own prompt and retrieve A's content.
Fixed by keying the issued-hash cache by litellm_call_id, matching
the pattern already used in compression_interception: a hash is
only honored when it was issued under the exact call_id resolving
for the current request.
2. The Anthropic Messages replay path fell through to the chat-style
assistant/tool-message builder, which Anthropic does not accept.
Anthropic requires the tool_use block echoed in an assistant message
paired with a tool_result block in a user message, keyed by
tool_use_id. Added a dedicated branch for this shape.
* docs: note proactive API-fragmentation helper convention
Add a bullet to the coding-conventions list: look for or add a shared
helper when logic branches on API surface (chat completions vs
Anthropic Messages vs Responses API), instead of duplicating
format-detection per module.
* fix(guardrails/headroom): fix Anthropic tool-shape detection, extract shared cross-API tool util
Live e2e testing against the real Anthropic API surfaced two bugs the
mocked unit tests couldn't catch because they used MagicMock responses
instead of realistic response shapes:
1. has_headroom_retrieve_tool only recognized OpenAI-shaped function
tools. By the time an Anthropic Messages response reaches the
agentic-loop gate, the tool this guardrail injected has already been
transformed into Anthropic's native shape (type: "custom", top-level
"name"), so the gate never fired for real Anthropic requests.
2. AnthropicMessagesResponse is a TypedDict, so real responses are
plain dicts at runtime, not objects with attribute access. The
extractors and format detectors used bare getattr(), which silently
returns nothing for dict responses instead of reading the actual
key.
Extracted the cross-API-surface tool-call extraction and tool-presence
check into litellm/litellm_core_utils/prompt_templates/factory.py
(get_tool_calls_from_response, has_tool_with_name) so this format
fragmentation is handled in one place instead of being duplicated
per-guardrail, and reused the existing repair-aware
parse_tool_call_arguments from common_utils instead of a naive
json.loads. headroom.py now delegates to these shared helpers.
Confirmed live against the real Anthropic API: the retrieve loop now
fires and successfully retrieves the correct hash's content through
the full compress -> tool-call -> retrieve -> replay round-trip.
* fix(guardrails/headroom): fix ruff-strict UP006/I001 budget violations
Use lowercase list/dict generics in the new factory.py tool-call
helpers instead of typing.List/Dict, drop the now-unused Tuple import
in headroom.py, and reorder the new factory import ahead of the
llms.custom_httpx import to satisfy import sorting.
* fix(guardrails/headroom): match Anthropic tools without a type field
Anthropic's documented client tool format is just name + input_schema;
type: "custom" is only one possible value, not a requirement. Match
any non-OpenAI-shaped tool on its top-level name instead of requiring
type == "custom".
