* feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools
GET /v1/mcp/tools returns the MCP tools available to the calling key, the same
data already exposed through /mcp/tools/list and /mcp-rest/tools/list, both of
which are in llm_api_routes. The /v1/mcp/tools path was in no route group, so
virtual keys created from the UI (which default to allowed_routes=["llm_api_routes"])
got a 403 listing tools one way but not the other.
Add it to mcp_inference_routes. Unlike /v1/mcp/server, this path has no
management write counterpart, so it does not need the method-aware carve-out
used for server discovery.
* test(proxy): parametrize MCP inference route check over the full endpoint set
Delete the in-product survey and Claude Code feedback prompts end to end.
Frontend: remove the src/components/survey/ module, the index page's nudge
state/effects/handlers, the getInProductNudgesCall helper, and the orphaned
"Disable UI nudges" toggle in the admin UI Settings page; prune the stale
eslint-suppressions entries.
Backend: remove the now-dead /in_product_nudges route, the InProductNudgeResponse
type, and the disable_ui_nudges UI setting (Field + allowlist). Nothing read it
for logic and the UISettings model is extra="allow", so existing stored configs
are unaffected (the value is just no longer surfaced). schema.d.ts is regenerated
and the two tests covering the removed route/setting are dropped.
* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes (#30089)
* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes
Add the realtime WebRTC HTTP sub-routes (/realtime/client_secrets,
/realtime/calls and their /v1 + /openai/v1 variants) to
LiteLLMRoutes.openai_routes so is_llm_api_route() classifies them as
LLM API routes. Without this, non-admin virtual keys received
401 'Only proxy admin can be used to generate, delete, update info
for new keys/users/teams' when calling these endpoints.
Fixes#29923
* fix(proxy): validate session.model for realtime routes in model-access check
The GA Realtime WebRTC HTTP routes resolve the effective model from the
nested session.model (falling back to the top-level model), but the auth
layer's get_model_from_request() only extracted the top-level model. A
model-restricted virtual key could therefore place a disallowed model in
session.model, leave the top-level model unset, and skip can_key_call_model()
entirely - obtaining an ephemeral token for a model it is not allowed to use.
Extract session.model for the realtime client_secrets/calls routes so the
model-access check runs against the model the request will actually use.
Legitimate callers are unaffected; their permitted model still validates.
Relates to https://github.com/BerriAI/litellm/issues/29923
* fix(proxy): classify realtime transcription_sessions routes as LLM API routes
Add the GA Realtime WebRTC transcription_sessions HTTP routes to
openai_routes so is_llm_api_route() returns True for them, matching the
client_secrets and calls routes already fixed. These endpoints are
registered with user_api_key_auth in realtime_endpoints/endpoints.py, so
without this a non-admin virtual key calling
POST /v1/realtime/transcription_sessions would hit the admin-only 401
branch. Extends the regression test parametrization accordingly.
---------
Co-authored-by: habonlaci <4699494+habonlaci@users.noreply.github.com>
* feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models (#30272)
* feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models
* fix(proxy): degrade /v1/models gracefully when model-group lookup fails
---------
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix: sort tiered token-cost thresholds numerically (#30375)
* fix: sort tiered token-cost thresholds numerically
_get_token_base_cost iterated input_cost_per_token_above_<N>_tokens keys with a
lexicographic sort, so for tiers whose thresholds have different digit lengths
(e.g. 90k vs 128k) a request crossing both was billed at the lower tier that
sorted first. Sort by the parsed numeric threshold instead, so the highest tier
the request actually crosses is applied.
* refactor: reuse _parse_above_token_threshold for inline threshold parse
---------
Co-authored-by: Eric (GabiDevFamily) <271972409+santino18727-debug@users.noreply.github.com>
* fix(openai): preserve cache_control for openai-compatible custom endpoints (#30387)
* fix(openai): preserve cache_control for openai-compatible custom endpoints
* fix(openai): use parsed hostname to detect real OpenAI for cache_control preservation
* fix(proxy): drain all daily-spend batches per flush cycle (#30281) (#30505)
* fix(types): prevent internal parallel_request_limiter fields from leaking to upstream providers (#30545)
* fix(types): add internal parallel_request_limiter fields to all_litellm_params to prevent forwarding to upstream providers
* test(types): add regression test for internal rate-limit fields in all_litellm_params
* fix(init): add bool type annotation to suppress_debug_info (#30531)
Module-level `suppress_debug_info = False` had no annotation, so strict
type checkers (e.g. ty) infer it as `Literal[False]`. Reassigning it to
`True` (as done in proxy_server.py and router.py) then fails with an
invalid-assignment error. Annotate it as `bool` to match every other
flag in this module.
* fix: coalesce null aggregates in update_metrics for no-spend keys (#29945)
* feat(team_endpoints): add query parameter `key_limit` to `/team/info` endpoint (#30006)
* feat(team_endpoints): Add query parameter key_limit to /team/info
* feat(team_endpoints): update schema.d.ts to include the new query parameter
* feat(team_endpoints): add tests for limitting key count in /team/info response
* feat(team_endpoints): Apply suggestions from greptile
* Set greater-than constraint on key-limit
* Fix type
* fix(router): release aiohttp connection when stream iteration ends abnormally (#30271)
* fix(router): release aiohttp connection when stream iteration ends abnormally
A streaming response that terminates with a mid-stream read timeout, a task
cancellation (client disconnect), or GeneratorExit never closed the underlying
aiohttp ClientResponse. aiohttp only auto-releases the connector slot at body
EOF, so each abnormally terminated stream permanently leaked one slot from the
shared TCPConnector pool. During a backend traffic spike the pool drains; once
exhausted every subsequent request to that host waits for a slot, times out
and surfaces as a 408, indefinitely, even after the backend recovers. Only a
proxy restart cleared the in-memory sessions, which matched the reported
symptom of a router stuck returning 408 for a healthy vLLM backend.
Close the response in a finally clause when iteration ends. On a fully read
response the connection was already released at EOF and close() is a no-op,
so keep-alive reuse for normal requests is unchanged.
Fixes#30192
* test(aiohttp): cover GeneratorExit path with a mock instead of a live socket
The previous slot-release test started a real aiohttp TCP server, which can
flake in offline CI and does not exercise this fix's code path directly.
Replace it with a dependency-injected mock that closes the stream generator
(GeneratorExit) and asserts the response is closed, covering the third
abnormal-exit path the finally block handles
* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (#30273)
* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery
* refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils
* fix(proxy): make model_list request param optional for direct callers
* feat(dashscope): add Responses API support (#30286)
* feat(dashscope): add Responses API support
DashScope's OpenAI-compatible endpoint serves /responses, so register a
DashScopeResponsesAPIConfig that routes dashscope/* responses calls to
{api_base}/responses without rewriting the upstream model id, instead of
falling back to the chat-completions -> responses emulation pipeline.
Closes#29780
* feat(dashscope): mark responses API as not supporting native websocket
Matches the hosted_vllm/perplexity/openrouter responses configs, which all
override supports_native_websocket() to False since the OpenAI-compatible
endpoint has no native wss:// responses transport.
---------
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix(spend-logs): preserve error_message on ProxyException failures (#30381)
* fix(spend-logs): preserve error_message on ProxyException failures
`StandardLoggingPayloadSetup.get_error_information` used
`str(original_exception)` to populate the human-readable error message
stored in `spend_logs.metadata.error_information.error_message`.
`ProxyException` (litellm/proxy/_types.py:3453) sets `self.message` in
its constructor but does NOT call `super().__init__(message)` and does
NOT define `__str__`. As a result, `str(ProxyException(...))` returns
the empty string, and every auth/budget/quota rejection was landing
in spend_logs with `error_message=""` despite a fully populated
traceback.
Operator impact: dashboard "LLM Failure" rows became untriageable —
the only way to tell a 401 from a 429 was to manually unpack the
traceback JSON via psql. Burst failure patterns (e.g. a UI session
polling with a stale token) produced 20-30 indistinguishable
`error_code=401` rows per second.
Fix: prefer the `.message` attribute (set by ProxyException and every
litellm.exceptions.* class) over `str(exc)`. The `str(exc)` fallback
is retained for non-litellm exception types, preserving prior behavior.
Test plan:
- 2 new unit tests in tests/test_litellm/litellm_core_utils/
test_litellm_logging.py:
* test_get_error_information_prefers_message_attribute_over_str
* test_get_error_information_falls_back_to_str_when_no_message_attr
- Existing test_get_error_information_error_code_priority still passes
- End-to-end verified: bad-key 401 now stores full
"Authentication Error, Invalid proxy server token passed..."
message in spend_logs.metadata.error_information.error_message
* fix(spend-logs): preserve explicit empty .message + drop dead reference
Greptile P2 on #30381. The truthiness check `if message_attr:`
silently skipped an explicit empty-string `.message` and fell
through to `str(original_exception)`. For ProxyException-shaped
objects both produce empty, so the bug was latent; for other
exception types it would inject a different string into
error_information.error_message and corrupt the signal.
Use `is not None` so an empty string survives verbatim.
Also drop the stale `See e2e/cases/11.` comment reference — that
path does not exist anywhere in the repo and confuses future
readers.
Regression test added: an exception with `.message=""` and a
non-empty `super().__init__()` arg must yield error_message == "".
* ci: retrigger workflows after base branch change to litellm_internal_staging
* fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response (#30382)
* fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response
The non-streaming /v1/messages response carries a LiteLLM-injected
usage.total_tokens = input_tokens + output_tokens that is not part of
the Anthropic API spec. This caused three problems:
1. Shape divergence with streaming on the same endpoint.
message_delta.usage in the SSE path never carries total_tokens.
Clients parsing both paths get two different schemas from one endpoint.
2. Shape divergence with upstream. Direct calls to
https://api.anthropic.com/v1/messages return no total_tokens field,
so clients using the official Anthropic SDK couldn't rely on it,
and clients that did rely on the LiteLLM-injected one broke when
bypassing the proxy.
3. Numerical misuse. total = input + output undercounts when
cache_read_input_tokens and cache_creation_input_tokens are
non-zero, because cache tokens are reported in their own fields.
A 100k-token cached prompt with 1 non-cache input token + 200
output tokens reports total_tokens = 201, off by ~99.8% from any
reasonable definition of "total."
Fix: add _strip_total_tokens_from_anthropic_response in
litellm/proxy/anthropic_endpoints/endpoints.py and invoke it in the
success path of anthropic_response right before returning. Only mutates
dict-shaped responses; streaming (which already lacks the field) is
left untouched.
spend_logs / Prometheus continue to compute total_tokens internally
for billing — this fix only strips the field from the wire response.
Scope: only the Anthropic passthrough endpoint /v1/messages. The
OpenAI-shape /v1/chat/completions is unaffected.
* fix(anthropic): gate total_tokens strip behind flag + handle Pydantic .usage
Two P1 greptile threads on #30382:
P1 — **Backwards-incompatible removal without a feature flag**
Stripping `usage.total_tokens` unconditionally breaks any client
currently reading the LiteLLM-shaped non-streaming /v1/messages
response. Per the codebase's policy (mirrors #30418), gate behind
a new flag.
- `litellm.strip_anthropic_total_tokens: bool = False` (default —
backward-compat: clients keep seeing total_tokens).
- Env override: `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS=true`.
- Docstring: planned to flip to True in a future major release;
opt in early.
P1 — **Silent no-op if `result` is a Pydantic model**
`base_process_llm_request` may return a Pydantic-style object
whose `.usage` is a plain dict (the most common shape — e.g.
objects wrapping raw upstream JSON). The original
`isinstance(response, dict)` guard skipped strip on those, so
`total_tokens` would still hit the wire. Helper now also reads
`getattr(response, "usage", None)` and strips when that's a dict.
Strongly-typed Pydantic `Usage` sub-models with required
`total_tokens` fields are still skipped — those impose type
constraints the helper doesn't try to subvert.
Tests:
- `test_strips_total_tokens_on_pydantic_model_with_dict_usage`
- `test_flag_defaults_off`
8/8 pass locally.
* fix(anthropic): drop env var for strip flag (docs CI)
Mirrors #30418's pattern (`expose_router_debug_in_errors: bool = True`,
no `os.getenv`). The `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS` env var
introduced in the prior commit was flagged by
`tests/documentation_tests/test_env_keys.py` because the documentation
file `docs/my-website/docs/proxy/config_settings.md` lives in
`BerriAI/litellm-docs` (separate repo) and registering a new env key
requires a parallel docs PR — a friction we avoid here by exposing
the flag only as a Python attribute + `litellm_settings` config key,
both of which load through the existing proxy config plumbing without
needing the env-var registry to be updated.
No semantic change: default still False, behavior identical when set
via `litellm.strip_anthropic_total_tokens = True` or
`litellm_settings.strip_anthropic_total_tokens: true` in config.yaml.
Verified locally: env scan no longer surfaces the key; 8/8 tests pass.
* ci: retrigger workflows after base branch change to litellm_internal_staging
* fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 (#30413)
* fix(pricing): correct swapped input/output token costs for command-r7b-12-2024
* test: resolve model prices JSON relative to test file for pip installs
* fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError (#30417)
* fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError
Some Gemini-compatible gateways (e.g. new-api) wrap a 429 rate-limit
signal from upstream inside an HTTP 500/503 envelope, with the real
code only surfaced in the JSON body:
{"error":{"message":"...high demand...","type":"upstream_error",
"param":"","code":429}}
Previously LiteLLM only looked at the HTTP status and mapped this to
InternalServerError, which Router treats as non-retryable for many
configs — so users got hard 500s instead of fallback/retry.
Now the Gemini/Vertex exception mapper parses error.code from the body
and routes code 429 to RateLimitError before falling through to the
HTTP-status branches. Other body codes fall through unchanged.
Tests cover:
- new-api gateway's `code:429` payload now maps to RateLimitError
- Genuine 500-body responses stay InternalServerError
- Non-JSON body strings fall through to status-code mapping unchanged
* fix(exception-mapping): scope body-code 429 promotion to 5xx envelopes
Addresses greptile P1/P2 + @Sameerlite's review on #30417. The new
elif branch was firing for any HTTP status, so a gateway response of
HTTP 400 with body {"error":{"code":429,...}} would be incorrectly
promoted to RateLimitError (retryable) instead of falling through
to BadRequestError. Same trap for 401 -> AuthenticationError.
Scoped the body-code 429 check to `500 <= status_code < 600` —
covers 500/502/503/504 (gateways wrapping upstream 429 in any 5xx
envelope) without inviting the 4xx misclassification.
Tests: parametrized table now covers 5xx (500/502/503), 4xx (400/401),
and the existing fall-through cases, asserting each maps to the
exception type that matches the HTTP status code. 50/50 pass locally.
* ci: retrigger workflows after base branch change to litellm_internal_staging
* feat(router): add expose_router_debug_in_errors flag (default True) to redact internal model_group/fallback names (#30418)
* feat(router)!: redact internal model_group/fallback names from exception messages
The Router was unconditionally appending internal config names onto
exception.message:
- "Received Model Group=..."
- "Available Model Group Fallbacks=..."
- "No fallback model group found... Fallbacks={...}"
- "context_window_fallbacks={...}"
- Deployment-timeout messages including model_group
- Fallback failure detail listing fallback chain
ProxyException forwards .message verbatim to clients, so gateways were
leaking their model_name / fallback wiring in every failed call.
Fix: gate all five mutation sites on a new
`litellm.expose_router_debug_in_errors` flag (default False). Set to
True to restore upstream debug behavior for local debugging.
Why: matches the redaction posture this codebase already has for
upstream model identifiers (cf. _litellm_returned_model_name) and
removes the last common error-path leak of internal model_group names.
Breaking change marker (!): if anything parses "Received Model Group="
out of client error messages, flip the flag on or migrate to the
x-litellm-* response headers instead.
Tests: 7 cases covering each of the 5 redaction sites + the flag-on
inverse path, plus a "default off" sanity check.
* test(router): cover sites 1 + 3 of expose_router_debug_in_errors gate
Addresses Greptile / codecov feedback on #30418: patch coverage was
55.6% with 4 lines uncovered in litellm/router.py. The existing tests
exercised sites 2 (ContextWindowExceededError), 4 (no-fallback-found),
and 5 (Received Model Group) — both default and flag-on. Sites 1 and 3
were declared in the PR description as covered by "site 5 also fires"
but the gate body lines for each (the `e.message +=` inside the
`if litellm.expose_router_debug_in_errors:` branch) only execute when
the flag is on AND the specific exception path is taken, which neither
existing test triggered.
Added 4 new tests (default + flag-on × 2 sites):
- test_default_does_not_leak_deployment_timeout_debug
- test_flag_on_leaks_deployment_timeout_debug
- test_default_does_not_leak_content_policy_fallback_hint
- test_flag_on_leaks_content_policy_fallback_hint
Trigger details:
- Site 1 (litellm.Timeout in _acompletion) is reached via the
Router-supported `mock_timeout=True` + `timeout=0.001` kwargs on
`acompletion(...)`. Cannot embed a Timeout instance in model_list
because Router.__init__ deep-copies it and Timeout.__reduce__ does
not preserve the required positional args.
- Site 3 (ContentPolicyViolationError without content_policy_fallbacks
set, in async_function_with_fallbacks_common_utils) is reached by
passing a `mock_response=litellm.ContentPolicyViolationError(...)`
instance via the call-site kwarg — same deepcopy-avoidance reason.
11/11 tests pass locally. Patch coverage on litellm/router.py for this
PR's diff should now be 100%.
* chore(router): flip expose_router_debug_in_errors default to True
Addresses @Sameerlite's review on #30418 — maintain backward
compat on the wire. Redact becomes opt-in via setting the flag
to False; the historical behavior (leak internal model_group /
fallback wiring through exception messages) is preserved as the
default.
- litellm/__init__.py: default flipped to True, docstring rewritten
with deprecation note pointing at a future flip to False (redact
by default) in a major release.
- tests/test_litellm/test_router_exception_redaction.py: fixture
resets to True (was False); the "off" tests now explicitly set
False; the "default_leaks_*" tests rely on the fixture default.
test_flag_defaults_off -> test_flag_defaults_on.
- No router.py change needed; the gate keys off the same flag,
only the default changes.
- PR title no longer needs the breaking-change `!` marker — no
client sees a behavior change at default settings.
11/11 pass locally.
* ci: retrigger workflows after base branch change to litellm_internal_staging
* feat(guardrails): integrate Repelloai Argus guardrail (#30465)
* feat(guardrails): add RepelloAI Argus guardrail integration (#1)
* feat(guardrails): add RepelloAI Argus guardrail integration
Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.
* fix(guardrails): harden RepelloAI Argus guardrail
- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
block on unknown/malformed verdicts so an API change can't silently
disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
expose unreachable_fallback in the config schema, and stop leaking the
raw provider response / exception strings to the client
* fix(guardrails): address RepelloAI Argus review feedback
- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description
* docs(guardrails): add RepelloAI Argus docs page and dashboard listing
- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution
* fix(guardrails): keep RepelloAI asset_id optional in config model
A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.
* Add comment for last user turn scanning
* feat(guardrails): harden repelloai scanning
* feat(guardrails): expand repelloai scanning to include tool definitions
Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.
* refactor(guardrails): fix and harden repelloai schema text extraction
- Fix duplicate text in _iter_schema_text: previously all dict values were
re-queued onto the stack even after scalar/list keys were already extracted
explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param
* fix(guardrails): resolve pyright type errors in repelloai guardrail
- Narrow async_handler.post return from Response|None to Response with
explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
type in pyright
* fix(guardrails/repello): include Responses API instructions field in prompt scan
The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.
* feat: add api_key to config model and read prompt from data dict
* fix(guardrails/repello): plug input_text and tool-call response bypass gaps
Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.
Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.
Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.
* fix(guardrails/repello): scan Responses API function_call output arguments
Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.
* fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (#30486)
* fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients
When an Anthropic server-side tool (web_search, id `srvtoolu_...`) is used, its
result is carried in `provider_specific_fields.web_search_results` — PRs #17746
/ #17798 restore it for callers that round-trip provider_specific_fields. A
generic OpenAI client that does NOT preserve provider_specific_fields (e.g. Open
WebUI talking to a Vertex/Anthropic model over /chat/completions) drops it on
replay and instead sends back an assistant `tool_call` + a `tool` message both
keyed to the `srvtoolu_` id. The transform then produced a bare `server_tool_use`
(with no following *_tool_result) plus a user `tool_result` for the same id —
both invalid, so the next turn 400s:
messages.N.content.0: unexpected `tool_use_id` found in `tool_result` blocks:
srvtoolu_... Each `tool_result` block must have a corresponding `tool_use`
block in the previous message.
This is the commonly-reported vertex_ai symptom where Gemini works but Claude
400s on the 2nd turn of a web-search chat.
Fix (litellm/litellm_core_utils/prompt_templates/factory.py):
- convert_to_anthropic_tool_invoke: only emit a server_tool_use when its matching
*_tool_result is available to pair with it; otherwise skip it (a bare
server_tool_use is itself rejected).
- anthropic_messages_pt: drop a replayed `tool`/`function` message whose
tool_call_id starts with `srvtoolu_` (a server-executed tool produces no client
result; a user tool_result for it is invalid).
The existing reconstruction path (provider_specific_fields present, e.g. the
litellm SDK) is unchanged, as is regular client tool_use/tool_result.
Tests (tests/llm_translation/test_prompt_factory.py):
- update test_convert_to_anthropic_tool_invoke_server_tool ->
test_convert_to_anthropic_tool_invoke_server_tool_without_result_is_dropped
- add test_anthropic_messages_pt_generic_client_drops_orphan_server_tool
Follow-up to #17746 / #17798; addresses the generic-client (no
provider_specific_fields) case of #17737.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(anthropic): cover the srvtoolu_ round-trip fix in the test_litellm unit suite
The regression tests added in tests/llm_translation/test_prompt_factory.py aren't
run by the coverage CI job (it runs tests/test_litellm), so the new factory.py
branches showed as uncovered (codecov patch coverage). Add equivalent focused
tests in the unit suite so both new branches are exercised there:
- convert_to_anthropic_tool_invoke drops a srvtoolu_ server_tool_use when no
matching *_tool_result is available.
- anthropic_messages_pt drops the orphaned srvtoolu_ tool message a generic
OpenAI client replays.
Refs #17737
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(anthropic): cover the server_tool_use + result valid-pair path in unit suite
Covers the remaining patch-coverage lines codecov flagged: convert_to_anthropic_tool_invoke
emitting server_tool_use followed by its web_search_tool_result when the matching
result is present (the litellm-SDK round-trip path). Refs #17737
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(anthropic): flatten srvtoolu_ tool-message guard to a negated if
Addresses the Greptile style nit: replace the if-pass/else with a single negated
`if not (...)` guard around the tool_result append. Behavior unchanged. Refs #17737
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(proxy): require premium only when enabling premium metadata fields (#30285) (#30506)
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix(perplexity): stop double-billing reasoning tokens in manual cost fallback (#30488)
* fix(perplexity): stop double-billing reasoning tokens in manual cost fallback
When perplexity_cost_per_token cannot use the API-provided usage.cost.total_cost short-circuit and falls back to manual calculation, it multiplies the full usage.completion_tokens by output_cost_per_token and then adds reasoning_tokens * output_cost_per_reasoning_token on top. Per the OpenAI/Perplexity usage convention codified for the central path in PR #18607, completion_tokens already INCLUDES reasoning_tokens, so the manual fallback double-bills reasoning at both the output and reasoning rate.
Concrete impact on perplexity/sonar-deep-research (input 2e-6, output 8e-6, reasoning 3e-6): for the exact usage shape exercised by the live response fixture in tests/llm_translation/test_perplexity_reasoning.py (prompt_tokens=9, completion_tokens=20, reasoning_tokens=15) the current code charges 0.000223 vs the convention-correct 0.000103, a 2.165x overcharge. The bug is reachable whenever Perplexity omits the cost object (streaming chunks, fixture-driven paths, older API versions).
Subtracts reasoning_tokens (clamped at zero) from completion_tokens before applying the output rate, mirroring how dashscope/cost_calculator.py and the central generic_cost_per_token already handle it. Preserves the existing fallback behaviour when output_cost_per_reasoning_token is unset (all completion_tokens stay at the output rate).
Existing tests in tests/test_litellm/llms/perplexity/test_perplexity_cost_calculator.py asserted the buggy math and are updated to the convention-correct math. Adds a focused regression test using the exact usage shape from the live response fixture so this class of bug cannot be silently reintroduced.
* style(perplexity): drop redundant type annotation on else branch to satisfy mypy
mypy [no-redef] flagged 'completion_cost' as declared in both if and else arms; keeping the annotation only on the first declaration matches existing patterns in this file.
* fix(perplexity): update integration test expected costs for non-double-billed math
Three tests in test_perplexity_integration.py asserted the old buggy expectation
that reasoning_tokens are billed in addition to the full completion_tokens
count. After the fix in cost_per_token, reasoning_tokens are billed at the
reasoning rate and the remaining (completion_tokens - reasoning_tokens) at the
standard output rate, matching OpenAI/Perplexity convention (PR #18607).
Updates: test_end_to_end_cost_calculation_with_transformation,
test_main_cost_calculator_integration, test_high_volume_cost_calculation.
The high-volume sanity threshold drops to 0.25 to reflect the corrected total.
* fix(ui): use dynamic proxy base URL in MCP usage examples (#30487)
Replace hardcoded http://localhost:4000 with getProxyBaseUrl() in the
MCP server usage example and copy-to-clipboard snippet so the generated
configuration works for non-local deployments.
Fixes#30466
* feat: add missing UK PII entity types to Presidio guardrail (#30537)
* feat: add missing UK PII entity types to Presidio guardrail
Add UK_PASSPORT, UK_POSTCODE, and UK_VEHICLE_REGISTRATION to PiiEntityType enum and PII_ENTITY_CATEGORIES_MAP. These entity types are supported by Microsoft Presidio but were missing from litellm's type definitions, preventing users from configuring UK-specific PII detection.
* test: remove fragile hardcoded entity count test
Remove test_uk_category_entity_count which hardcodes len() == 5. The test_uk_entities_match_presidio_recognizers test already verifies exact set equality, making the count test redundant and fragile to future Presidio additions.
* style: apply Black formatting to match CI requirements
* fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (#30357)
Volcengine (Doubao) models define `tiered_pricing` but no flat per-token cost, so cost_per_token fell through to generic_cost_per_token (which only reads flat costs) and tracked them at $0
Route custom_llm_provider == "volcengine" to the shared tiered-pricing handler in litellm/llms/dashscope/cost_calculator.py, which already computes graduated tier costs. Make that handler provider-agnostic by adding a custom_llm_provider argument (default "dashscope" preserves existing behavior) so get_model_info resolves the correct model map entry
Fixes#30346
* feat(mcp): make MCP gateway name and description configurable via env vars (#30473)
* feat(mcp): make MCP gateway name and description configurable via env vars
* Rename function _restore_env to _apply_env
* docs(mcp): document import-time capture of env-backed identity constants
Address Greptile review feedback: clarify that LITELLM_MCP_SERVER_NAME and
LITELLM_MCP_SERVER_DESCRIPTION are read once at import and require a module
reload to observe env changes after import.
Generated with AI assistance
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Yevhen Luhovtsov <yevhen.luhovtsov@intapp.com>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(mcp): preserve native tools in semantic filter hook (#26650)
* fix(mcp): preserve native tools in semantic filter hook
The SemanticToolFilterHook.async_pre_call_hook passed ALL tools (MCP +
native) to filter_tools(), which only knows MCP-registered tool names.
Native tools silently failed the name match in _get_tools_by_names()
and were dropped from the request.
Fix: partition tools into native and MCP-registered before filtering.
Run the semantic filter only on MCP tools, then merge native tools
back unconditionally.
Changes:
- Robust _is_mcp_tool() using shape-based detection for OpenAI-format
dicts, safe regardless of future _extract_tool_info changes
- Single-pass partition loop (no double _is_mcp_tool calls)
- Preserve native tools in MCP expansion path (mixed requests)
- Track MCP expansion to prevent expanded tools bypassing filtering
- filter_stats reports MCP-only counts for accurate metrics
- Extracted _emit_filter_metadata() helper
- Skip spurious filter headers for all-native tool requests
Closes#26212
* remove stale docstring note referencing tools_expanded_from_mcp
* fix: handle Responses API name collision and preserve tool ordering
- Classify Responses API tools ({type: 'function', name: '...'}) as
native to prevent name collisions with MCP canonical names
- Preserve original request tool ordering using id()-based merge
instead of naive native+mcp concatenation
- Add 2 regression tests: name collision and ordering preservation
* style: apply black formatting
* fix(mcp): harden semantic filter — preserve all native tool formats, safe metadata access, graceful expansion failure, name-based merge
* lint: suppress PLR0915 on async_pre_call_hook (matches codebase convention)
* ci: retrigger checks after rebase onto litellm_internal_staging
* feat(fireworks): sync Fireworks AI model registry with current platform catalog (#30616)
Adds 12 new Fireworks serverless models and updates 3 existing entries in
model_prices_and_context_window.json and its bundled backup to match the
current Fireworks platform model list. New direct models: glm-5p2,
qwen3p7-plus, minimax-m3, minimax-m2p7, kimi-k2p7-code, kimi-k2p6,
deepseek-v4-pro, deepseek-v4-flash. New router endpoints: glm-5p1-fast,
kimi-k2p6-fast, kimi-k2p7-code-fast. Updated: glm-5p1, gpt-oss-120b, and
gpt-oss-20b now carry correct output token caps, cache-read pricing, and
explicit capability flags
max_tokens is set equal to max_output_tokens (not the full context window)
for models whose generation cap is below their context window. This avoids
the shared input+output budget path in get_modified_max_tokens, which would
otherwise let callers request output sizes the model cannot produce. The
same fix corrects the pre-existing glm-5p1, gpt-oss-120b, and gpt-oss-20b
entries that had max_tokens equal to the full context window
Short-form aliases (fireworks_ai/<model>) are added for every direct
accounts/fireworks/models/ entry so cost attribution works for callers
using bare model names. Router endpoints get short-form aliases too, and
transform_request now routes bare names ending in -fast to the
accounts/fireworks/routers/ path instead of defaulting every bare name to
models/. This keeps the kimi-k2p6-fast router from being misrouted to the
nonexistent models/kimi-k2p6-fast endpoint
kimi-k2p6-turbo is intentionally excluded; kimi-k2p6-fast is its
replacement. Context windows for deepseek-v4 and kimi models use the
power-of-two values (1048576 and 262144) published on the Fireworks model
pages, matching the convention already used by existing entries
Two regression tests in test_utils.py assert the exact per-token costs,
token limits, capability flags, and short-form-to-long-form equality for
all 15 models against both the main and backup cost maps. Two routing
tests in test_fireworks_ai_chat_transformation.py verify bare -fast names
route to routers/ and bare direct-model names route to models/
* fix(bedrock): handle role:"system" inside the messages array on /v1/messages (#29698) (#30443)
* feat(anthropic): hoist leading in-array system to top-level (helper)
* test(anthropic): cover _system_content_to_blocks edge cases; deepcopy cache_control
* test(anthropic): mid-conversation system normalization cases
* feat: add supports_mid_conversation_system flag to Claude Opus 4.8
Add supports_mid_conversation_system: true to all 9 claude-opus-4-8 cost-map
entries (Anthropic-native, Bedrock, Vertex, Azure AI) in both the root cost
map and the bundled package backup, since the runtime helper and tests read
the backup in local/offline mode.
Pin the mid-system passthrough regression test to the local cost map via the
existing local_model_cost_map fixture so it reads the branch-local flag rather
than the network-fetched main copy.
* fix(bedrock): normalize in-array system in /v1/messages handler (#29698)
Wire normalize_system_messages_for_anthropic into anthropic_messages_handler
so all Bedrock /v1/messages paths (Invoke / Mantle / ClaudePlatform /
Converse-bridge) hoist leading in-array system entries (and demote
mid-conversation ones on models lacking supports_mid_conversation_system) into
the top-level system field. The normalized messages/system are written back
into the local_vars snapshot the base_llm branch reads from, otherwise the
Invoke/Mantle fix would silently no-op.
Also fix the helper to resolve supports_mid_conversation_system through the
prefix-aware AnthropicModelInfo._supports_model_capability resolver. The raw
_supports_factory could not see the flag once get_llm_provider left the
invoke/ prefix on the model id, which would have wrongly demoted
mid-conversation system on a Bedrock invoke opus-4-8 path.
* fix(bedrock): resolve mid-conversation-system flag through mantle/invoke/converse route prefixes; drop unused param
* fix(types): widen system param to Union[str, List] for hoisted system blocks
* refactor(bedrock): drop dead local_vars messages writeback
* fix(bedrock/converse): translate in-array system in anthropic->openai adapter (#29698)
* fix(bedrock/converse): preserve cache_control on in-array system; test drop-empty
* fix(bedrock/converse): rename colliding local to satisfy mypy; test handler system-merge branches
* fix(types): register supports_mid_conversation_system in model-info schema
The cost-map JSON-schema validation test (test_aaamodel_prices_and_context_window_json_is_valid)
rejects unknown properties, so adding supports_mid_conversation_system to the opus-4-8
cost-map entries failed CI with 'Additional properties are not allowed'. Register the flag
in the INTENDED_SCHEMA allow-list and in the ProviderSpecificModelInfo TypedDict so it is a
typed, first-class capability flag alongside its peers (supports_output_config, etc.).
---------
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload (#28885)
* fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload
By default the agentcore provider flattens the last message to a text-only
{"prompt": "..."} payload via convert_content_list_to_str, silently dropping
OpenAI multimodal blocks (image_url, file, input_audio, ...).
This adds an opt-in `forward_multimodal_content` litellm param. When truthy and
the last message's content is a list containing a non-text block, the original
OpenAI content list is forwarded verbatim under a new "content" field so an
attachment-aware AgentCore agent can read it. Default off keeps the payload
byte-identical to the legacy {"prompt": "..."} shape — existing agents are
unaffected.
The flag is read from optional_params (where other AgentCore params land) with a
litellm_params fallback, and accepts a bool or a config/env string ('true', '1', ...).
AgentCore Runtime is schemaless on the agent side — the agent's @app.entrypoint
parses arbitrary JSON up to 100 MB (per
https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html),
so this is a purely upstream change; no AgentCore-side schema is asserted.
* fix(bedrock/agentcore): shallow-copy forwarded multimodal content list
Address review feedback (Sameerlite): payload["content"] = last_content
aliased the caller's mutable messages[-1]["content"] list. Harmless today
because the payload is JSON-serialized immediately, but a latent footgun if
a future caller mutates the returned payload before serialization. Forward
list(last_content) so the payload owns its own list. Block dicts stay shared
on purpose — a deep copy would clone potentially large base64 media on the
request hot path, and the flagged risk was the shared list, not the blocks.
Update the passthrough tests to assert equality + distinct identity, and add
a regression test that mutating the payload list can't leak back into the
original message content.
* Revert "fix(mcp): preserve native tools in semantic filter hook (#26650)"
This reverts commit 438c825bd4.
* Revert "feat(guardrails): integrate Repelloai Argus guardrail (#30465)"
This reverts commit 54da7857f2.
* Revert "feat(dashscope): add Responses API support (#30286)"
This reverts commit 67662565e8.
* Revert "fix(bedrock): handle role:"system" inside the messages array on /v1/messages (#29698) (#30443)"
This reverts commit b8a8083308.
* Revert "fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (#30486)"
This reverts commit 6e9c0b0dd2.
* Revert "fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (#30357)"
This reverts commit 172e302dab.
* Revert "feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (#30273)"
This reverts commit 4e3188525e.
* fix: pass key_limit=None in team_member_update and patch model_cost in pricing test
team_member_update called team_info without key_limit, so the fastapi.Query
default object (not None) was passed through to get_data, which failed when
serializing it. Pass key_limit=None explicitly to avoid this.
test_get_model_info_costs patched litellm.model_cost from the local backup so
the assertion holds before the PR is merged and the remote main URL is updated.
* fix(security): validate resolved model in /realtime/client_secrets for non-transcription sessions (#30710)
Omitting both model and session.model caused the endpoint to default to
gpt-4o-realtime-preview without running can_key_call_resolved_model, so
any key could access that model regardless of its allowed-model list.
The transcription path already called can_key_call_resolved_model; this
adds the same call for the realtime path before returning.
* fix(lint): fix F821 undefined model_info and F841 unused metadata in create_model_info_response
* fix: black formatting and stub get_model_group_info in third team translation test
* fix: reformat utils.py with black 26.3.1 to match CI
* fix: replace Optional[X] with X | None to satisfy UP045 ruff strict gate
---------
Co-authored-by: Habon Laszlo <habonlaci@users.noreply.github.com>
Co-authored-by: habonlaci <4699494+habonlaci@users.noreply.github.com>
Co-authored-by: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com>
Co-authored-by: santino18727-debug <santino18727@gmail.com>
Co-authored-by: Eric (GabiDevFamily) <271972409+santino18727-debug@users.noreply.github.com>
Co-authored-by: Nitish Agarwal <1592163+nitishagar@users.noreply.github.com>
Co-authored-by: jho1-godaddy <171078705+jho1-godaddy@users.noreply.github.com>
Co-authored-by: 安妮的心动录 <74543653+anneheartrecord@users.noreply.github.com>
Co-authored-by: Harshith Gujjeti <153299927+Harshxth@users.noreply.github.com>
Co-authored-by: Tomoya Tabuchi <t@tomoyat1.com>
Co-authored-by: Vedant Agarwal <43557509+Vedant-Agarwal@users.noreply.github.com>
Co-authored-by: Prathamesh Jadhav <55660103+lollinng@users.noreply.github.com>
Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com>
Co-authored-by: Kropiunig <48442031+Kropiunig@users.noreply.github.com>
Co-authored-by: Lavish Bansal <lavish.bansal619@gmail.com>
Co-authored-by: Shane Emmons <27679+semmons99@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Anuj ojha <ojhaanuj224@gmail.com>
Co-authored-by: Nahrin <nahrin@nahrinoda.com>
Co-authored-by: Nbouyaa <67773915+FadelT@users.noreply.github.com>
Co-authored-by: Vineeth Sai <vineethsai4444@gmail.com>
Co-authored-by: Eugene Lugovtsov <34510252+EugeneLugovtsov@users.noreply.github.com>
Co-authored-by: Yevhen Luhovtsov <yevhen.luhovtsov@intapp.com>
Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com>
Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com>
Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com>
Co-authored-by: Jón Levy <levy@apro.is>
Add bare /v1/vector_stores/{vector_store_id} to openai_routes so retrieve, update, and delete classify as LLM API routes for internal user and internal viewer roles.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): match passthrough registry routes bare-to-bare with SERVER_ROOT_PATH
After #28547, get_request_route strips the deployment prefix while registry
lookup still re-inflated stored paths via SERVER_ROOT_PATH, causing 404s
under paths like /llmproxy/ml. Compare normalized bare routes in both
is_registered_pass_through_route and get_registered_pass_through_route.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(proxy): patch utils.get_server_root_path in passthrough auth tests
After removing get_server_root_path from pass_through_endpoints, route
and JWT tests must mock litellm.proxy.utils where normalization reads it.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(auth): let internal users view search tools
Internal users could not see search tools in the UI even when an admin
created them, while vector stores were visible. The Search Tools page
rendered but its list calls 403'd because the read routes were not in
internal_user_routes. Grant internal users read-only access to the
listing and provider routes; create/update/delete stay admin-only.
Resolves LIT-3150
* fix(auth): scope /search_tools/list to caller-allowed tools
Adding the read routes to internal_user_routes let any internal user
call /search_tools/list, which returned every configured tool's id,
provider, api_base, and metadata regardless of the caller's
object_permission.search_tools allowlist. The api_key was masked, so
this was metadata disclosure rather than a credential leak, but it
ignored the key/team scoping that /search already enforces.
Filter the listing through the same can_key_call_search_tool /
can_team_call_search_tool checks (exposed as a boolean
can_user_view_search_tool), so non-admin callers only see tools they
may invoke; admins still see all. Mirrors how /vector_store/list
scopes results.
* fix(types): resolve mypy errors in list_search_tools
The config and DB build loops reused one loop variable, so mypy pinned
it to the config element type (SearchToolTypedDict); its .get() calls
returned object and the DB element (SearchTool) failed the reuse
assignment. Give each loop its own variable so each gets its real type,
and coerce the config tool's SearchToolInfoTypedDict to a plain dict to
match SearchToolInfoResponse.search_tool_info.
* fix(proxy): enforce allowed_passthrough_routes for auth=true pass-through
Pass-through endpoints with auth=true were injected into openai_routes,
so teams with openai_routes access bypassed per-team allowed_passthrough_routes.
Gate auth-enforced pass-through at JWT, virtual-key, and non-admin route checks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): clarify JWT passthrough denial
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): make pass-through auth checks method-aware
Prevent allowlist bypass when the same path is registered with different auth settings per HTTP method.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix passthrough route auth checks
* fix(proxy): reject unregistered pass-through HTTP methods
Enforce method-aware JWT checks and return 405 when stale FastAPI routes accept requests outside the current pass-through registry.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): remove duplicate request_method in JWT team lookup
Fixes SyntaxError on proxy startup caused by passing request_method twice to find_team_with_model_access.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix passthrough route auth enforcement
* fix(proxy): raise passthrough-specific 403 directly in virtual-key path
* fix(proxy): load team for RBAC role-claim JWT passthrough gating
* Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326)
This reverts the Bedrock CI account migration (#28728). The original account
(888602223428) was put under an AWS security restriction after a leaked key
and has since been reactivated, while the replacement account (941277531214)
lacks access to several models the suites exercise (legacy Bedrock Claude 3
models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship
Opus). Pointing CI back at the reactivated account restores that coverage.
This is the exact inverse of #28728: all hardcoded 941277531214 references go
back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs
and their suffixes, batch execution role ARN, and the example proxy config),
the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs
revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge
Base revert to their original ids, and the live-call tests go back to the
legacy model strings. The grid_spec fail_reason workaround for the unentitled
Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field
added after the migration.
The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at
941277531214 and must be set to the reactivated account's fresh credentials
separately via the CircleCI API; AWS_REGION_NAME stays us-west-2.
(cherry picked from commit f11c12d157)
* fix(proxy): scope pass-through 405 to registry routes; grant rerank passthrough in rpm tests
The auth=true pass-through 405 guard fired for mapped provider routes
(e.g. /assemblyai/*) that are not in the in-memory registry, since
get_registered_pass_through_route returns None for them while
is_registered_pass_through_route matches via mapped_pass_through_routes.
Only raise 405 when the path is registered but the request method is not
allowed, so mapped provider pass-throughs fall through to the default
target params as before.
The rpm-limit pass-through tests register /v1/rerank with auth=true but
gave their keys no allowed_passthrough_routes, so the new default-deny
returned 403 before the rate limiter ran (non-deterministically,
depending on registry insertion order). Grant the keys explicit
passthrough access so the tests exercise rate limiting under the new
auth model.
* fix(proxy): guard request method lookup against scopes without a method
Starlette's Request.method property reads scope["method"] and raises
KeyError when the scope omits it (e.g. minimally-constructed test
requests). getattr only swallows AttributeError, so the new
_get_request_method helper propagated the KeyError up through
user_api_key_auth and surfaced as a ProxyException. Catch KeyError
(and AttributeError) and fall back to None.
* test(passthrough): pin SERVER_ROOT_PATH in unregistered-method test
test_custom_proxy.py sets os.environ['SERVER_ROOT_PATH'] = '/my-custom-path'
at module import with no cleanup. When that module is collected into the same
xdist worker as this test, the leaked root path is prepended to registered
pass-through paths, so is_registered_pass_through_route misses '/test/path'
and the handler returns 404 instead of the expected 405 (order-dependent).
Pin SERVER_ROOT_PATH to '' so the test is deterministic.
* test(passthrough): restore regression coverage for non-auth-enforced pass-through via llm_api_routes
* fix(proxy): record auth flag in pass-through registry for allowlist enforcement
Auth-enforced pass-through detection inferred enforcement from the FastAPI
dependency stored at registration time. The management create and update
endpoints register routes with dependencies=None even though auth defaults to
true, so is_auth_enforced_pass_through_route treated those DB-created routes as
unenforced. A key allowed for llm_api_routes could then call a management-created
auth-enabled pass-through route without matching allowed_passthrough_routes.
Store the auth setting on each registry entry and read it directly when deciding
whether the allowlist applies, instead of deriving it from dependency metadata.
* fix(proxy): include bool in pass-through registry value type for auth flag
The auth flag stored in _registered_pass_through_routes is a bool, which
was not part of the registry value Union, so mypy rejected the dict literal.
Add bool to the Union and narrow route_methods to a list before the
membership check so the in-operator stays valid.
* fix(proxy): preserve stored auth flag on pass-through endpoint update
model_dump(exclude_none=True) re-included the auth=True default whenever
a partial update omitted auth, silently flipping an existing auth=false
pass-through to auth-enforced and 403ing every team/key without
allowed_passthrough_routes. Merge only explicitly set fields via
exclude_unset so omitted fields keep their stored value.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): allow llm_api_routes virtual keys to list MCP servers
Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET
/v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that
virtual keys configured with `allowed_routes=["llm_api_routes"]` can
discover the MCP servers they have access to. Previously these calls
failed with 'Virtual key is not allowed to call this route. Only allowed
to call routes: [llm_api_routes]'.
The GET handlers already sanitize the response for restricted virtual
keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping
credential-bearing fields (url, headers, env). Write methods
(POST/PUT/DELETE) on the same paths remain gated by the existing
handler-level admin role checks.
The new discovery list is intentionally kept OUT of
`mcp_inference_routes`, so `is_llm_api_route()` still returns False
for these paths — this preserves the existing contract that
DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP
servers.
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
* refactor(proxy): make MCP discovery carve-out method-aware
Replace the `mcp_discovery_routes` group in `llm_api_routes` with a
method-aware special case inside `is_virtual_key_allowed_to_call_route`.
Virtual keys with allowed_routes=["llm_api_routes"] are now permitted
to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} —
non-GET methods and multi-segment admin sub-paths fall through to the
existing 403. This keeps the general llm_api_routes list free of
management paths and avoids accidentally exposing POST/PUT/DELETE
writes through the route-check layer.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
* fix: patch Host-header auth bypass in get_request_route
Starlette reconstructs request.url from the Host header. A malformed
Host like `localhost/?x=1` causes Starlette to build the full URL as
`http://localhost/?x=1/health`, which url-parses to path="/". Since "/"
is in LiteLLMRoutes.public_routes, all protected routes became reachable
without authentication.
Fix: read scope["path"] (set by uvicorn from the HTTP request line,
not derivable from headers) instead of request.url.path. Sub-path
deployments are handled via scope["app_root_path"] / scope["root_path"],
mirroring Starlette's own base_url construction logic.
Affected variants confirmed fixed:
Host: localhost/?x=1
Host: localhost:4000/?x=1
Host: localhost/#test
Host: localhost:4000/#test
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* style: reduce comments in route fix
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block credential fields in RAG ingest vector_store options
Credential fields (vertex_credentials, aws_access_key_id, api_key, etc.)
in ingest_options.vector_store are now rejected at the API boundary with
a 400 error. Credentials must be configured server-side.
Previously any authenticated user could supply a vertex_credentials dict
with type=external_account pointing credential_source.file at an
arbitrary path (e.g. /proc/1/environ) and token_url at an
attacker-controlled server. google-auth's identity_pool.Credentials
refresh() would read the file and POST its contents to the attacker.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block /key/update self-escalation by assigned users
Non-admin users who were assigned a key (created_by != caller) could
update any non-budget field — models, rpm_limit, guardrails, etc. —
without admin authorization, allowing privilege self-escalation.
Gate: only the key creator (created_by == caller) may edit their own
key without admin check; budget changes always require admin regardless
of creator status. All other callers must pass _check_key_admin_access.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block user-controlled api_base in RAG ingest vector_store options
A user-supplied api_base in ingest_options.vector_store caused the server
to forward its configured provider credentials (Gemini, OpenAI) to an
attacker-controlled endpoint via SSRF.
Add api_base to the blocked credential params set alongside api_key and
the existing credential fields.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: restrict /utils/transform_request to PROXY_ADMIN and apply body safety check
Any authenticated internal_user could POST arbitrary provider config
(aws_sts_endpoint, api_base, etc.) to /utils/transform_request and have
the server forward its credentials to an attacker-controlled endpoint.
- Gate the endpoint on PROXY_ADMIN role (403 for all other roles)
- Call is_request_body_safe() to reject banned params even for admins
- Convert ValueError from safety check to HTTP 400
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: apply banned-param check to /utils/transform_request
Without is_request_body_safe(), any authenticated user could pass
aws_sts_endpoint, api_base, or aws_web_identity_token to
/utils/transform_request and have the server forward its configured
provider credentials to an attacker-controlled endpoint during SDK
credential resolution.
Applies the same banned-param blocklist already used by LLM endpoints.
Endpoint remains accessible to all authenticated users.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: block SSRF via api_base in /prompts/test dotprompt YAML frontmatter
Any frontmatter key not in ["model","input","output"] flowed into
optional_params and was merged into the LLM call data dict, bypassing
is_request_body_safe. An attacker with any bearer key could set
api_base in YAML to redirect the outbound LLM request — including the
provider API key — to an attacker-controlled host.
Fix: call is_request_body_safe on the constructed data dict after
optional_params are merged, before invoking ProxyBaseLLMRequestProcessing.
ValueError from the banned-param check is surfaced as HTTP 400.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* Update litellm/proxy/rag_endpoints/endpoints.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix: coerce nested config strings before banned-param check
_NESTED_CONFIG_KEYS descent used isinstance(nested, dict) which silently
skipped litellm_embedding_config when delivered as a JSON string via
multipart/form-data. Banned params (api_base, aws_sts_endpoint, etc.)
nested inside the stringified value were invisible to is_request_body_safe.
_NESTED_METADATA_KEYS already used _coerce_metadata_to_dict which parses
JSON strings before checking. Apply the same coercion to _NESTED_CONFIG_KEYS.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: replace substring match with prefix match in is_llm_api_route
mapped_pass_through_routes used `_llm_passthrough_route in route` (substring)
so any admin-only path whose URL contained a provider name (openai, anthropic,
azure, bedrock, etc.) was misclassified as an LLM API route and bypassed the
admin gate in non_proxy_admin_allowed_routes_check.
Confirmed live: non-admin key could GET /credentials/by_name/openai (read
masked provider API key) and DELETE /credentials/openai (delete credential).
Fix: use exact match or startswith(prefix + "/") — the same pattern used
everywhere else in RouteChecks — so only routes that actually start with a
passthrough prefix are allowed through.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: stabilize PR #27878 test failures
- key_management_endpoints: extend can_skip_admin_check to team keys so
team members with /key/update permission can update non-budget fields.
can_team_member_execute_key_management_endpoint already validates team
membership + permission and raises if unauthorized; reaching the admin
check on a team key means the caller was authorized.
- test: set created_by on mock key in
test_update_key_non_budget_fields_allowed_for_internal_user so
caller_is_creator resolves correctly (MagicMock default ≠ user_id).
- auth_utils.get_request_route: guard against non-dict request.scope
(e.g. MagicMock in unit tests) to prevent a MagicMock leaking into
UserAPIKeyAuth.request_route and failing Pydantic validation.
- ci: assign test_multipart_bypass_repro.py to the proxy-runtime shard
in test-unit-proxy-db.yml to satisfy the shard-coverage check.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(lint): add explicit str() cast in get_request_route for MyPy
scope.get() returns Any|None which MyPy cannot coerce to str implicitly.
Wrap both scope.get() calls in str() to satisfy the type checker.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: guard bare-/ root_path strip + make total_spend migration idempotent
auth_utils.get_request_route: when Starlette sets scope["app_root_path"]
to "/" (e.g. behind some middleware), the old stripping logic would
remove the leading slash from every path ("/team/new" → "team/new"),
breaking route matching and causing auth to misclassify protected routes.
Skip stripping when root_path is bare "/".
migration: add IF NOT EXISTS to total_spend ALTER TABLE so the migration
is safe to replay when a prior partial run already created the column.
Without this guard, prisma migrate deploy fails on CI DBs that were
partially migrated, causing all subsequent DB operations (including
/team/new) to 500.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: require creator still owns key for personal-key bypass in /key/update
caller_is_creator now requires both created_by == caller AND user_id ==
caller. Previously checking only created_by let a demoted admin who
originally created a key for another user continue editing non-budget
fields on it after reassignment, bypassing _check_key_admin_access.
Adds regression test: creator whose key was reassigned is blocked (403).
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: extract auth checks to fix PLR0915 + broaden max_budget assertion
internal_user_endpoints._update_single_user_helper exceeded 50 statements
(PLR0915). Extract authorization checks into _check_user_update_authz helper
to bring statement count under the limit.
test_validate_max_budget: assert "negative" (substring of both the local
"cannot be negative" and the CI "non-negative finite number" messages) so
the test is stable regardless of which exact wording the function uses.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
Greptile P1: the unsafe-method branch of `_check_proxy_admin_viewer_access`
ended with a blanket `if route in management_routes: return`. That set is a
mix of reads (info/list — handled via the safe-method GET branch above) and
writes. The fallback let Admin Viewer POST to write endpoints not enumerated
in `_ADMIN_VIEWER_BLOCKED_WRITE_ROUTES`, including:
- /team/block, /team/unblock, /team/permissions_update
- /jwt/key/mapping/{new,update,delete}
- /key/bulk_update
- /key/{key_id}/reset_spend
Remove the fallback. The two remaining allow sets (admin_viewer_routes and
global_spend_tracking_routes) are both read-only, so removal does not affect
the legitimate POST-as-read cases (e.g. /spend/calculate, which is in
spend_tracking_routes ⊂ admin_viewer_routes).
Tests:
- 8 new parametrized cases pinning each previously-leaking management write
endpoint to 403 on POST for PROXY_ADMIN_VIEW_ONLY.
`_check_proxy_admin_viewer_access` enumerates write routes a
PROXY_ADMIN_VIEW_ONLY caller may not invoke, then falls through to
"allow" for any management route not listed. Several write endpoints
were never added to the blocklist, so a viewer could:
- block or unblock any team via `/team/block` / `/team/unblock`
- mutate team permissions via `/team/permissions_update` and
`/team/permissions_bulk_update`
- create, update, or delete JWT key mappings via
`/jwt/key/mapping/{new,update,delete}`
- bulk-edit keys via `/key/bulk_update`
- reset key spend via the path-parameterized `/key/{id}/reset_spend`
Hoist the blocklist into a module-level frozenset and a tuple of
suffix patterns so it's clear what to extend when a new write route
is added, and pull the existing key write routes from the
`KeyManagementRoutes` enum so the two stay in sync. Adds parametrized
tests over the newly-blocked routes plus baseline coverage for routes
that should remain allowed (info / list / daily-activity reads).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: admin_viewer_routes was an explicit allowlist, so every newly-added
GET endpoint anywhere in the codebase silently 403'd for admin viewer until
someone remembered to add it. We had whacked /spend/logs/ui, /customer/list,
/guardrails/list, /policies/attachments/list, /invitation/info, and several
others in serial — but the next round still surfaced /in_product_nudges,
/health/latest, /credentials, /v1/mcp/network/client-ip, /claude-code/plugins,
/policy/templates. This pattern keeps repeating because the model is wrong.
Structural fix in `_check_proxy_admin_viewer_access`:
- Default-allow safe HTTP methods (GET / HEAD / OPTIONS) on any
non-inference route. Admin Viewer's principle is read parity with
Proxy Admin; HTTP semantics already mark GET as side-effect-free, so
using the method as the allow signal is the correct primitive.
- Unsafe methods (POST/PUT/PATCH/DELETE) still go through the existing
explicit allowlists + the hard-blocked write set
(/user/new, /team/new, /key/generate, …).
- LLM/inference routes still 403 (cost-incurring).
The existing admin_viewer_routes list is retained as a backstop for the
small set of routes implemented as POST but semantically read (e.g.
/spend/calculate). Adding new GET endpoints no longer requires touching
this list.
Models page tab/panel off-by-one (UI bug for Admin Viewer):
Tremor's TabList filters falsy children but TabPanels does not, so
conditionally hiding "Add Model" with `{!shouldHideAddModelTab && ...}`
left a phantom panel slot — clicking "LLM Credentials" showed nothing,
and clicking "Pass-Through Endpoints" showed the credentials panel.
Refactor to a single source-of-truth `visibleTabs` array; tab and
panel indices now can never desync.
Tests:
- 12 parametrized tests covering the 6 user-reported endpoints + 4
hypothetical-future endpoints + 2 already-fixed ones, all asserting
Admin Viewer GET succeeds via the default-allow path (no allowlist
entry needed).
- 5 parametrized tests for POST writes still 403'ing
(random-future-write, /user/new, /team/new, /key/generate, /model/new).
- All 207 existing route_checks tests still pass — backward-compatible.
User reported six more 403s and "still restricts access to keys + models" after
the first round. Root causes:
1. Six read endpoints were missing from admin_viewer_routes:
- /guardrails/list, /v2/guardrails/list (Guardrails page)
- /guardrails/submissions, /guardrails/submissions/{guardrail_id}
- /guardrails/usage/overview (Guardrails Monitor page)
- /policies/attachments/list (Policies page)
- /get/mcp_semantic_filter_settings (Settings page)
2. /guardrails/submissions handler treated admin viewer as non-admin, filtering
them to only their team submissions. Switch to _user_has_admin_view() so
admin viewer sees all submissions (read parity with Proxy Admin).
3. UI Keys page (user_dashboard.tsx) and Models page (ModelsAndEndpointsView.tsx)
each had a hard "Access Denied" block specifically for "Admin Viewer" — a
leftover from the pre-parity era. Remove the blocks; gate the "Create Key"
button on the Keys page so admin viewer can read keys but not mint them.
Also drop the post-login redirect that forced admin viewers to /usage on
sign-in (page.tsx).
Tests:
- Extend ADMIN_VIEWER_SETTINGS_ROUTES parametrize list to cover all 7 new
routes (route-checks layer is now the layer production traffic actually
hits, vs. the dependency-override-bypass that was masking the gap).
Greptile review caught that the /invitation/info handler relaxation was
dead code: the route_checks layer rejects admin viewers before the handler
runs because /invitation/info was never added to admin_viewer_routes.
Add /invitation/info to admin_viewer_routes and extend the route-level
parametrized test to cover it.
The handler-level integration test passed previously because
`app.dependency_overrides[user_api_key_auth]` bypasses route_checks; this
new route-level test exercises the layer that production traffic hits.
Admin Viewer (proxy_admin_viewer) was being blocked from endpoints it should
be able to read. Most visibly the UI Logs page rendered empty because every
filter and detail call (/spend/logs/ui, /spend/logs/ui/{id},
/spend/logs/session/ui, /customer/list) was rejected at the route_checks
layer even though the underlying handlers permit admin-viewer.
Backend:
- Extend admin_viewer_routes to include spend_tracking_routes,
/customer/{list,info}, /spend/logs/* detail routes, callback / config /
budget / alerting reads, and model cost map status/source.
- Replace bare `user_role != PROXY_ADMIN` checks in read-only handlers
(/budget/list, /budget/settings, /alerting/settings, /invitation/info,
/config/field/info, /config/list, /schedule/model_cost_map_reload/status,
/model/cost_map/source) with `_user_has_admin_view()`.
UI:
- Add `rolesAllowedToViewWriteScopedPages` (rolesWithWriteAccess + Admin
Viewer) and use it for the "Models + Endpoints" and "Agents" sidebar
items so admin viewers see them read-only. Playground stays gated by
rolesWithWriteAccess (cost-incurring).
- Hide Add / Edit / Delete buttons in the LLM Credentials panel for
non-proxy-admin viewers.
Tests:
- 31 parametrized route_checks cases for the Logs + settings endpoints,
with internal-user negative coverage to ensure the gate isn't widened.
- 9 handler-level integration tests (FastAPI TestClient) verifying
admin viewer is no longer blocked at the handler layer.
- New leftnav cases asserting Playground hidden / Models + Agents / Logs
visible to Admin Viewer.
- New roles + credentials test cases for the UI write-gate.
The routes in `global_spend_tracking_routes` (e.g. /global/spend/report,
/global/spend/teams, /global/spend/keys) return spend aggregated across
every team, customer, and api_key in the proxy. They were included in
`internal_user_routes` and `internal_user_view_only_routes`, so non-admin
roles could read proxy-wide spend.
Drop them from both non-admin route lists. PROXY_ADMIN and
PROXY_ADMIN_VIEW_ONLY access is preserved through their existing branches
in route_checks.py, and the `get_spend_routes` permission opt-in
continues to grant access for keys that need it.
Updates two pre-existing test parametrizations whose expected results
flip from True to False, and adds parametrized coverage over every
route in `global_spend_tracking_routes` for: PROXY_ADMIN_VIEW_ONLY
allowed, INTERNAL_USER blocked, INTERNAL_USER_VIEW_ONLY blocked,
INTERNAL_USER + get_spend_routes permission allowed.
MCP server CRUD endpoints (/v1/mcp/server*) were bundled with MCP
tool-call / passthrough endpoints under llm_api_routes, so setting
DISABLE_LLM_API_ENDPOINTS=true on admin-only nodes also blocked the
Admin UI from listing, adding, or attaching MCP servers.
Separate mcp_inference_routes (data-plane, gated by
DISABLE_LLM_API_ENDPOINTS) from mcp_management_routes (control-plane,
gated by DISABLE_ADMIN_ENDPOINTS). Keep mcp_routes as a union for
backward compat with allowed_routes=["mcp_routes"] virtual key configs.
Upgrade is_management_route to pattern-aware matching so
/v1/mcp/server/{path:path} resolves for concrete IDs.
* feat: add brave/search to model_prices_and_context_window.json (#25042)
Brave Search is supported by litellm as a search provider (documented at
docs.litellm.ai/docs/search/brave and listed in provider_endpoints_support.json)
but was missing from model_prices_and_context_window.json, making it invisible
to any code that discovers search providers from litellm.model_cost.
Cost: $0.005/query ($5 per 1,000 requests) per https://brave.com/search/api/
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock (#24588)
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock
Add model definition for nvidia.nemotron-3-super-120b-a12b-v1 via
Bedrock Converse API with pricing, context window (256k/32k), and
capability flags (function calling, tool choice, system messages).
* fix model ID to nvidia.nemotron-super-3-120b + add tests
Correct the Bedrock model ID from nvidia.nemotron-3-super-120b-a12b-v1
(NVIDIA's internal name) to nvidia.nemotron-super-3-120b (the actual
AWS Bedrock programmatic model ID). Add unit tests verifying model
resolution, pricing, and context window.
* fix(proxy): allow JWT auth for /v1/mcp/server sub-paths (#24698)
mcp_routes only contained "/v1/mcp/server" (exact match). Starlette's
compile_path produces an end-anchored regex, so sub-paths like
/register, /health, /submissions, /oauth/* all failed the JWT
allowed_routes_check. Add a {path:path} wildcard entry so all
sub-paths are covered.
---------
Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
- Only remove wildcard path from openai_routes when the route entry has
type="subpath", avoiding accidental removal when two endpoints share
the same base path but differ in include_subpath
- Clean up _registered_pass_through_routes in the test finally block to
prevent stale entries from polluting subsequent tests on failure
- Add dedup guard for base path registration (prevents unbounded list
growth on config reload)
- Clean up base path and wildcard path from openai_routes when an
endpoint is removed via remove_endpoint_routes
- Rewrite test to exercise initialize_pass_through_endpoints directly,
covering registration, dedup on reload, and cleanup on removal
When a pass-through endpoint has both auth=true and include_subpath=true,
non-admin users got 401 errors on subpath requests because only the base
path was registered in openai_routes. Now the wildcard path is also
registered so the auth check recognizes subpath requests as LLM API routes.
Also fixes pre-existing pyright error where logging_obj was possibly
unbound in the except block.
- Thread org objects {organization_id, organization_alias} instead of bare IDs from
users/page.tsx → view_users.tsx → CreateUserButton so the selector can show aliases
- Replace single-select org dropdown with multi-select; always shown when organizationIds
is non-null; disabled/pre-selected for single-org admins; displays "Alias (id)"
- handleCreate: maps organization_ids → organizations before POST, removes redundant
organizationMemberAddCall (backend _add_user_to_organizations handles it)
- _user_is_org_admin: also checks organizations list field in addition to singular
organization_id so /user/new succeeds for org admins
- Add 5 backend unit tests for _user_is_org_admin and 2 frontend tests for new form behavior
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Org admins and team admins opening the invite-user modal could not see
the 4 global proxy roles because GET /user/available_roles has no
request body, so the org-admin route check (which requires
organization_id in the payload) always returned False and blocked them.
Add /user/available_roles to self_managed_routes so the route-access
check passes for any authenticated user. The endpoint's existing
Depends(user_api_key_auth) still requires a valid API key.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The route-level auth check was blocking internal_user role (team admins)
from reaching /key/{key}/reset_spend because KEY_RESET_SPEND was missing
from key_management_routes. Added it so team admins pass the route check
and the endpoint's existing _check_proxy_or_team_admin_for_key enforces
actual authorization.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy): support slashes in google route params
* fix(proxy): extract google model ids with slashes
* test(proxy): cover google model ids with slashes
Fixes#16470
Video generation endpoints (/v1/videos, /videos/{video_id}, etc.) were
incorrectly restricted to proxy_admin role only. These routes are now
added to openai_routes list, making them accessible to internal_user
role as they should be - video generation is a legitimate user feature,
not a management/admin operation.
Changes:
- Added 8 video route patterns to LiteLLMRoutes.openai_routes in _types.py
- Added comprehensive tests verifying internal_user and virtual key access
- All existing route permission tests continue to pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>
* feat(key_management_endpoints.py): Support new 'key_type' field
allow user to specify if key should be 'management' or 'llm api' key
Security fix
* test(test_route_checks.py): add unit tests
* fix(create_key_button.tsx): add ui component to select key type
allows specifying if key can call llm api vs. management routes
* feat(create_key_button.tsx): add specifying key type to ui
* fix(route_checks.py): add sensitive data masker for user id on not allowed error message
prevent leaking sensitive information
* feat(route_checks.py): allow admin to disable proxy management endpoints on instance
useful for preventing multiple instances from doing admin actions
* docs(scaling_multiple_instances.md): add architecture doc on scaling multiple litellm instances
provide guidance on scaling proxy
* docs(scaling_multiple_instances.md): add doc on scaling across multiple regions for litellm
* fix(route_checks.py): allow disabling llm api endpoints on an instance
allows pure admin instance to exist
* refactor(enterprise/route_checks.py): refactor env var checks
* refactor: finish refactoring
* docs(control_plane_and_data_plane.md): refactor docs
* test: update tests
