* fix(bedrock_guardrails): select latest user message by original role in apply_guardrail (#23476)
* test(bedrock_guardrails): cover masking write-back through unified handler (#23476)
* fix(bedrock_guardrails): guard masked write-back on unresolved slice, not length
* chore(bedrock_guardrails): use builtin generics and extract write-back helper to satisfy strict ruff gate
#29256 made auth=true pass-through routes deny-by-default unless the key/team
has allowed_passthrough_routes configured, but this integration test was not
updated. The test key had no allowlist, so the auth=true parametrizations
(rpm_limit=0 -> expect 429, rpm_limit=2 -> expect 207) now hit the 403 gate in
auth before reaching the rpm/forwarding logic they mean to exercise.
Grant the test key allowed_passthrough_routes for /api/public/ingestion so it
clears the gate. Also removes a latent order-dependency: the case only passed
locally when an earlier (auth=false) parametrization registered the route first;
under worker isolation (CI xdist) it failed with 403.
The monolith images shipped whatever UI bundle was committed to
litellm/proxy/_experimental/out, so refreshing the UI for a release meant
running build_ui.sh out of band and committing the regenerated bundle. Add a
ui-builder stage to all three monolith Dockerfiles (root, database, non_root)
that compiles the Next.js static export from this exact source and replaces the
committed bundle before the final uv sync.
The stage is pinned with FROM --platform=$BUILDPLATFORM so the
architecture-independent static export compiles once on the native builder even
in a multi-arch (linux/amd64,linux/arm64) build, rather than once per target
arch under QEMU emulation. The destination is cleared before the COPY because
COPY merges directories and would otherwise leave the committed bundle's
content-hashed chunks behind alongside the fresh ones. build_admin_ui.sh still
runs afterward so the enterprise custom-color override is preserved.
The UI base image is pinned by digest to match LITELLM_BUILD_IMAGE,
LITELLM_RUNTIME_IMAGE and UV_IMAGE, and .dockerignore now excludes the local
.next/out so a developer's build artifacts never enter the context.
When a streaming request hits a mid-stream 429 the streaming handler wraps it
in the internal MidStreamFallbackError so the router can attempt fallbacks. With
no fallbacks configured, async_function_with_fallbacks_common_utils falls through
to re-raising that wrapper, which the streaming iterators caught and re-raised
verbatim, so the client received MidStreamFallbackError (an internal type) rather
than a clean RateLimitError (429).
When the fallback path produces a MidStreamFallbackError that carries an
original_exception (i.e. no fallback handled it), the iterators now raise that
underlying provider exception instead of the wrapper, chained with from. Users
with fallbacks are unaffected since their path never reaches this branch. Applied
consistently to the chat async, chat sync, and responses streaming iterators.
Resolves LIT-3503
Fixes#26015
handle_accumulated_json_chunk re-ran json.loads on the entire accumulated
buffer after every fragment. For a streaming response fragmented across many
chunks that is O(n^2) total work in a single GIL-holding C call, so a large
enough Gemini response freezes the asyncio event loop for seconds, liveness
probes time out, and the proxy pod gets killed and restarted.
A complete Gemini stream value is a JSON object or array, so the buffer can
only become parseable once its last non-whitespace byte can close one. Gate
the json.loads attempt on that, which makes the common fragmented-response
case parse roughly once instead of once per fragment. An 8MB payload drops
from a 6.9s event-loop freeze to ~0.3s with identical parsed output.
Resolves LIT-3503
Fixes#26181
* fix(proxy): skip OpenAI model override for search responses
Search responses omit a model field by spec but still set model on the
request for routing, which caused noisy errors and dict injection.
* fix(proxy): drop redundant search-specific model override skip
The silent return for responses without a model field already covers
SearchResponse objects; remove the extra search type check.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): skip model override for dict responses without model key
Dict-shaped responses (e.g. search) must not get a spurious model field
injected when they never had one; only override when model is present.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(proxy): cover swallowed setattr failure in model override
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): prevent stale Vertex bearer token causing /v1/messages 401 after token expiry
Router shallow-copies litellm_params so extra_headers is a shared reference.
The chat/completions path was calling headers.update() on that shared dict,
persisting the Vertex OAuth bearer. After ~1 h the token expired and /v1/messages
kept reusing it (skipping refresh due to Authorization-already-present guard).
- Build a new headers dict in the Claude partner-models completion path instead
of mutating the shared extra_headers object.
- Always call _ensure_access_token() in validate_anthropic_messages_environment
regardless of an existing Authorization header; the token cache makes this cheap.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(vertex_ai): copy headers in validate_anthropic_messages_environment to prevent shared-dict mutation
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes#30926)
* style: format common_utils.py with black
* fix(anthropic): extract api_base from litellm_params in batches/files validate_environment
* fix(anthropic): scope Bearer key check to custom api_base endpoints
* fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives
The Anthropic streaming protocol emits `message_start.usage.output_tokens=1`
as a placeholder cursor; the real cumulative output count only arrives in
the final `message_delta` event. When a stream is cancelled before
`message_delta` lands (common for thinking models on long-tail prompts),
ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left
completion_tokens stuck at 1. Because 1 is truthy, the
`completion_tokens or token_counter(text=...)` fallback in
calculate_usage() never fired, and requests were billed for 1 output
token even when several thousand tokens of text had actually streamed.
Fix: track whether any chunk's completion_tokens exceeded 1
(saw_non_cursor_completion). If the only update we saw was the cursor,
reset completion_tokens to 0 so the text-based fallback estimates from
the real completion content.
Legitimate 1-token completions (model returns "Yes." etc.) are unaffected
in practice — token_counter on a 1-token completion_output also yields
~1, so billing stays approximately correct.
Tests:
- TestAnthropicCursorBug (6 cases) — pins the post-fix behavior
- TestNonAnthropicStreamingIntact (2 cases) — guards against regression on
providers without the cursor pattern
All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests
still pass.
* fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival
Addresses both Greptile P2 threads on PR #30420:
CLASS A — Anthropic-specific heuristic was applied globally
============================================================
The `completion_tokens == 1 and not saw_non_cursor_completion` reset
lived in provider-neutral `streaming_chunk_builder_utils.py`. Any
non-Anthropic provider that legitimately reports completion_tokens=1
in a single usage chunk (perfectly normal for short OpenAI / Bedrock /
Vertex single-token replies with stream_options.include_usage=true)
would have its value silently rewritten to 0 and re-billed via
token_counter — producing a different number than what the provider
actually charged.
Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved
from the first chunk's `_hidden_params` (the same field set by
streaming_handler.py:722 on the live path). Unknown / missing provider
is treated as non-Anthropic and skips the reset, so newer providers and
custom plugins are also safe by default.
CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies
============================================================
Previous condition was `usage_chunk_dict["completion_tokens"] > 1`,
which never fires for an Anthropic stream where the model legitimately
emits exactly one output token (e.g., "Yes."). Anthropic still sends
message_start (output_tokens=1, the cursor) AND message_delta
(output_tokens=1, the real value) — same value, but two distinct usage
events. The old check couldn't tell that apart from a cancelled stream
where only message_start landed.
Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion`
when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR
(2) we've seen >=2 completion-bearing usage events (positive evidence
that message_delta arrived). Cancelled cursor-only streams still have
exactly one event and still hit the reset; cache chunks with
completion_tokens=0 don't count toward the threshold.
Tests
============================================================
- _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default
"anthropic") so the gate is exercised by every existing test —
none of them needed assertion changes besides the legitimate-single-
token case, which now expects exactly 1 (was a fuzzy 0..3 range).
- New: test_anthropic_cache_only_chunks_after_message_start_still_resets
- New: test_non_anthropic_provider_completion_tokens_one_not_reset
- New: test_unknown_provider_completion_tokens_one_not_reset
11/11 tests pass.
* chore: add Co-authored-by trailer for attribution
Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
* fix(anthropic): preserve messages cache usage
* style(anthropic): format messages cache usage helper
* fix(anthropic): accept integral float cache token counts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(anthropic): accept integral float cache token counts
* test(anthropic): cover cache usage edge cases
* fix(gemini): preserve thoughtSignature for server-side tool responses
When Gemini API returns toolCall and toolResponse parts, they might have
different thoughtSignatures. Previously, LiteLLM merged them into a single
dict, overwriting the response's thoughtSignature with the call's.
This fix extracts them separately and re-injects them correctly.
TAG=agy
CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6
* fix(gemini): address PR comments on thoughtSignature handling
- Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature
- Add missing assertions in existing tests
- Add new unit tests for orphan-response signature handling
TAG=agy
CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6
* feat(mcp): include server alias and server_id in mcp_info response
- Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint
- Update rest_endpoints.py to surface alias from server config
- Add test coverage in test_mcp_server.py and test_rest_endpoints.py
Fixes#31015
* fix(proxy): reject non-finite spend via validate_finite_spend
A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a
shared finite-value guard, defined above the litellm.proxy.* imports to
avoid the module-level cyclic-import warning.
* fix(proxy): require admin for any /key/update spend, reject non-finite
Gate the admin check on the presence of `spend` (not a value diff): the
DB spend lags the live cross-pod counter, so an "unchanged" spend on the
non-admin path let a key owner / team member overwrite the live counter
below real usage. Also reject NaN/+-inf spend before the DB write.
* fix(proxy): invalidate spend counter on /user/update spend change
A direct spend change on /user/update wrote the DB row but left the warm
cross-pod counter at the stale value, so enforcement kept reading the old
spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB),
and reject non-finite spend before the write.
* fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244)
The semantic cache's embedding model is a proxy Router alias whose AWS
credentials (aws_role_name, aws_session_name) live only in the Router
deployment's litellm_params. The sync embedding paths called litellm.embedding()
directly, bypassing the Router, so they could neither resolve the alias nor
assume the configured role; cross-account Bedrock semantic caching failed with
"bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup
because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding
during cache construction, while llm_router is still None.
Fix A: make the sync paths mirror the already-correct async paths. A shared,
dependency-injected helper (litellm/caching/_embedding_router.py) decides whether
to route through llm_router.embedding(...) when the model is a Router deployment,
else fall back to direct litellm.embedding(...). Redis and qdrant sync
set_cache/get_cache now precompute the embedding and pass vector= to the backend,
exactly as the async astore/acheck already do. Both async _get_async_embedding
methods are unified onto the same helper and now forward the caller's full
metadata instead of a hand-picked subset.
Fix B (Redis only): defer redisvl index construction from __init__ into a lazy,
memoized llmcache property, so the dimension-probe embedding fires on first cache
use, after llm_router is wired. A failed build is not memoized, so a transient
outage recovers on the next request.
Known limitation: resolve_embedding_router gates on an exact model-name match
(same as the shipped async path); wildcard/alias/team-public routes still fall
back to direct embedding. Tracked as a follow-up.
* fix(cache): harden embedding-router and shrink Any surface (review)
Address review feedback on the semantic-cache aws-role fix (#28244):
- resolve_embedding_router now skips deployment entries missing model_name
instead of raising KeyError on a malformed model_list (Greptile P2);
add a regression test that fails on the old direct-key access.
- Replace the `**kwargs: Any` passthrough on the four cache _get_embedding /
_get_async_embedding helpers with an explicit, typed
`metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only
ever consumed kwargs["metadata"], so this is behavior-preserving, makes the
forwarded field obvious at the call site, and removes three bare-Any
annotations (keeps the strict-rule ANN401 budget within ceiling).
- Note in _build_llmcache that redisvl's dimension-probe embedding adds one
extra billable embedding on the first cache request (Greptile P2).
* fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models
Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist"
Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved
A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop
The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities
acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash
* test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression
Adds the regression coverage codecov flagged on the two responses to completion
bridge guard lines and the bedrock route-prefix helper. The handler tests drive
both the sync and async fallback paths with litellm.completion and
litellm.acompletion mocked, and assert the forwarded kwargs carry
_skip_responses_api_bridge=True, so dropping either flag line fails the suite.
The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer
resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids
still do, exercising both branches of _model_has_route_prefix.
Also aligns update_messages_with_model_file_ids model_id to Optional[str],
matching its Responses API sibling, so the defensive model_info fallback no
longer introduces a new reportArgumentType in completion(); the file-id lookup
narrows model_id before the dict get
* chore(ui): sync generated OpenAPI types for optional test_connection mode
The test_model_connection mode body param default changed from chat to None so
the mode is auto-detected from model capabilities, which makes the field
optional in the proxy OpenAPI spec. Regenerate the committed schema so the
dashboard types match: mode becomes optional and the description and default
JSDoc follow the spec, keeping the Check UI API Types Sync gate green
* refactor(bedrock): match all explicit route prefixes at path-segment boundary
Migrates the remaining substring route checks to the existing
_model_has_route_prefix helper so every explicit route token matches only as a
leading path segment, consistent with get_bedrock_route and the mantle route.
Covers _explicit_converse_route, _explicit_claude_platform_route,
_explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route,
_explicit_converse_like_route, _explicit_async_invoke_route and
_explicit_openai_route. This also stops invoke/ from substring-matching
async_invoke/. Route precedence and order are unchanged, and a note on the
segment invariant is added to the helper docstring
* test(bedrock): cover explicit route prefix segment matching
Exercises all eight migrated _explicit_*_route helpers (converse, converse_like,
invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each
matches its token as a leading path segment and rejects the token glued to a
preceding segment, so reverting any method to the old substring check fails the
suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete
improvement of the segment-boundary migration
* test(proxy): assert negative spend is allowed (one-time grant use-case)
Negative spend is intentionally permitted so admins can grant extra
allowance for the current budget period only, without raising the
recurring budget ceiling. Cover it explicitly in validate_finite_spend
and via the /user/update invalidation test.
* fix(google_genai): forward native generateContent top-level fields
Google's native generateContent REST body carries safetySettings, toolConfig,
cachedContent and labels at the top level as siblings of generationConfig. The
proxy's :generateContent endpoint spread them into agenerate_content as loose
kwargs and then dropped them, so callers had to wrap them in extra_body for them
to take effect; safetySettings, for instance, was silently ignored
The provider config now exposes the native top-level field names and
setup_generate_content_call collects whichever are present, merging them into the
outgoing request body through the existing extra_body merge so they reach Google
verbatim. An explicit extra_body still wins on conflict. The sync
generate_content_stream path now also forwards systemInstruction, matching the
other three entry points
Fixes#12671
Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK
* fix(proxy): resolve env refs for DB-stored models
* fix(proxy): restrict DB env ref resolution
* fix(proxy): block team DB env ref resolution
* fix(lint): resolve ANN401/UP045/C901 strict-gate violations
- Replace Optional[X] with X | None (UP045) in 8 files
- Replace Any return/param types with concrete types or object (ANN401)
- Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix
Users who pass a key already prefixed with "Bearer " get Authorization: Bearer.
All other keys continue to use x-api-key, preserving backward compatibility with
custom api_base endpoints that expect x-api-key rather than Authorization.
Also consolidates get_auth_header to reuse _make_api_key_auth_header helper,
eliminating the duplicated custom-endpoint routing logic.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base
The backwards-compat change broke existing tests that verify the intentional
Bearer-for-custom-base behavior (Fixes#30926). Restore original logic while
keeping the _make_api_key_auth_header helper for code deduplication.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag
Previously the auth-header switch from x-api-key to Authorization: Bearer
applied unconditionally for non-sk-ant- keys on a custom api_base, silently
breaking existing deployments that proxied to gateways expecting x-api-key.
Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header,
get_anthropic_headers, and get_auth_header. validate_environment reads it from
litellm_params so callers can opt in per-model without any API surface change.
Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981)
DEL was the only Redis cache operation that skipped check_and_fix_namespace,
so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the
namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM
error on deployments with an ACL restricting DEL to the litellm:* pattern,
and a silent no-op on all other deployments since the un-prefixed key was
never stored.
* style(anthropic): reformat common_utils.py with Black (--target-version py312)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: preserve cache metadata and spend counters
* style: apply ruff format to streaming_iterator.py
* refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate
Extract Anthropic message_start cursor reset into
_reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter
invalidation into _invalidate_user_spend_counter_if_changed, keeping both
_calculate_usage_per_chunk and _update_single_user_helper under the
max-complexity ceiling. Use builtin generics in the new signatures so no
new UP006 violations are introduced. Behavior unchanged.
---------
Co-authored-by: rupak-eng <rupakji99@gmail.com>
Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com>
Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com>
Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com>
Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com>
Co-authored-by: Andrii Butko <booandrew23@gmail.com>
Co-authored-by: Kent <kingdooo@gmail.com>
Co-authored-by: kunal2002 <k.nayyar2002@gmail.com>
Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com>
Co-authored-by: jesco-absolut <team@srswti.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Matt Hill <mhill@dataminr.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* test(logging): cover streaming /v1/messages OpenAI Responses spend logs
The #28595 fix added unit tests that call _handle_anthropic_messages_response_logging
directly, but nothing exercises the streaming wiring that actually regressed:
a streaming /v1/messages call cross-routed to the OpenAI Responses backend whose
success handler took the no-op async_log_stream_event path and dropped the SpendLogs
row. Add an end-to-end test that drives litellm.anthropic_messages(stream=True) with a
mocked upstream Responses SSE and asserts async_log_success_event fires with non-zero
cost and call_type anthropic_messages, plus a key-gated live counterpart.
* test(logging): exercise stream deltas and assert single success log
Address review on the streaming bridge regression test: emit output_item.added
plus text deltas before response.completed so it covers mid-stream delta handling
rather than only end-of-stream success logging, assert at least one
content_block_delta surfaces, restore litellm.callbacks via monkeypatch instead of
leaking global state, and assert async_log_success_event fires exactly once.
* test(logging): drop live network test from mock-only suite
Greptile flagged that tests/test_litellm only permits mock tests; network calls
belong in tests/e2e. Remove the key-gated live counterpart and keep the
deterministic mocked test as the regression guard. The live verification stays
in the PR description as the proof of fix.
* fix(vertex): preserve Gemini Embedding 2 usageMetadata for cost tracking
* style(vertex): apply ruff format to batch_embed_content_transformation
* fix(vertex): bill files/ image refs in Gemini embedContent at per-image rate
Resolved files/... references whose mime type is an image were not detected
by _is_image_element, so image_count stayed 0 and generic_cost_per_token fell
back to the text token rate instead of input_cost_per_image. Thread the
resolved_files mapping into the usage builder so resolved image references are
counted and billed per image. Also modernize the _flatten_input return
annotation to satisfy the ruff UP006 strict gate.
* fix(vertex): bill Gemini embedding audio per-second and stop video+audio double-billing
Audio-only embedContent responses set audio_tokens, but generic_cost_per_token only
charges audio via input_cost_per_audio_token. gemini-embedding-2 prices audio via
input_cost_per_audio_per_second, so spend stayed at $0. Plumb a new
audio_length_seconds field through PromptTokensDetailsWrapper, parse it in
_parse_prompt_tokens_details, and bill it from _calculate_input_cost. The vertex
embedding transformation derives audio_length_seconds from audio_tokens using
the documented 32 tokens/sec Gemini rate.
The 1-token text floor that protects video billing only fired when no other
modality was billable, but audio presence flipped that flag, leaving text_tokens
at zero for video+audio responses. generic_cost_per_token then rewrote
text_tokens to prompt_tokens minus audio_tokens (the video token count),
charging video tokens as text on top of the per-second video cost. The rewrite
trigger is text_tokens == 0 and image_count == 0; align the floor with that
trigger and ignore audio_tokens.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Build form_data_dict in one pass with groupby instead of rescanning form_items per field name, and assert on the files list directly in the boundary regression test so repeated field names are not collapsed by dict().
Co-authored-by: Cursor <cursoragent@cursor.com>
Records main as an ancestor of internal_staging so the staging->main
promotion (#31384) merges cleanly. Resolves in staging's favor; changes 0
files (staging already supersedes every main-side hotfix). MUST be merged
as a real merge commit (not squash/rebase) or the link to main is lost.
* docs(readme): add Deploy on AWS/GCP with Terraform section
Adds a quickstart for the two published Terraform modules on the public
registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste
main.tf for each cloud, the one-time GCP Artifact Registry remote-repo
command, and pointers to the registry pages for the full input surface.
Sits inside the Get Started section, between the gateway/SDK table and
Run in Developer Mode -- where someone scanning the README for "how do I
deploy this" will land.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): add 1-click deploy buttons for AWS + GCP
GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo
and walks through `terraform apply` via the existing DeployStack
tutorial (already shipped at terraform/litellm/gcp/examples/default/
TUTORIAL.md). User just picks a project.
AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an
in-browser, already-authenticated shell. User runs four commands
(clone + cd + cp tfvars + terraform apply) once inside. There's no
native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN
"Launch Stack" + CodeBuild would be needed for that, and that's a
separate piece of work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): move AWS + GCP deploy buttons next to Render button
* docs(readme): unify deploy button sizes and badge styles
* docs(readme): bump deploy button height to 48 to match Render/Railway
* docs(readme): bump AWS/GCP badge height to compensate for SVG padding
* docs(readme): bump AWS/GCP badge height to 72
* docs(readme): bump AWS/GCP badge height to 84
* fix(readme): make deploy buttons same height (48px)
https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc
* docs(readme): flag GCP project ID substitution in image_registry
* docs(readme): equalize deploy button heights and fix Cloud Shell button font
GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px.
Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.
* docs(readme): collapse Railway deploy anchor to a single line
The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.
* Add Claude Fable 5 cost map entries as a data-only hotfix
Backports only the model map changes from #30064 so deployments on
released litellm versions pick up Fable 5 pricing, context window, and
the adaptive thinking flag through the hosted cost map fetch without
upgrading. Includes the supports_sampling_params flag on the 28
Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read
by the gating that ships with the next release) and the matching
one-line schema declaration so the map validation test passes.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* feat: make rust OCR async-first
* docs: clarify rust provider call flow
* docs: clarify OCR provider transform contract
* docs: note Tokio route contract
* fix: address OCR bridge review comments
* docs: bound rust OCR HTTP exception
* feat: generate rust providers from registry
* chore: move rust provider registry into core
* chore: source rust providers from endpoint registry
* fix: satisfy OCR lint budget
* fix: reduce OCR basedpyright argument errors
* fix: address OCR greptile feedback
* fix: align rust OCR request preparation
* fix: resolve OCR CodeQL alerts
* fix: avoid duplicate Rust OCR authorization header
* ci: rerun CircleCI
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: Ishaan Jaff <ishaan@berri.ai>
Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com>
Passthrough multipart uploads used form.items() and a files dict, so only the last file under a repeated field name reached the upstream. Read multi_items() and send httpx a list of file tuples instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(mcp): add require_key_mcp_access_defined to stop keys inheriting team MCP servers
By default a virtual key that grants no MCP servers of its own inherits its
team's full MCP server list. The new general_settings flag
require_key_mcp_access_defined (default false) flips this so the team list
acts purely as a ceiling: a key reaches only the servers it grants explicitly
(or via an access group), and inherits none. This mirrors the existing
require_end_user_mcp_access_defined setting.
The default is unchanged, so existing deployments keep today's behavior until
they opt in. The no-mcp-servers sentinel and key access-group grants are
unaffected.
* docs(mcp): note require_key_mcp_access_defined effect in resolver docstring
* fix(cost-map): retarget mistral-medium-latest to Medium 3.5 and add date-pinned aliases
Mistral repointed the rolling mistral-medium-latest alias from Medium 3.1
to Medium 3.5, but the static cost map still carried Medium 3.1 specs,
showing wrong pricing/context in the model hub and undercharging spend by
about 3.75x (LIT-3883).
Update mistral/mistral-medium-latest to Medium 3.5 ($1.50/$7.50 per 1M,
256K context, reasoning + vision), add the bare date-pinned aliases
mistral/mistral-medium-2604 (Medium 3.5) and mistral/mistral-medium-2508
(Medium 3.1) that match Mistral's real API model ids, and add
supports_reasoning to mistral/mistral-medium-3-5.
Apply every change to both model_prices_and_context_window.json and the
bundled litellm/model_prices_and_context_window_backup.json so the two
stay in sync, and extend the regression tests to lock the resolved
get_model_info values and the main/backup parity for all touched models.
* test(cost-map): force local cost map in mistral-medium-latest resolution test
get_model_info reads litellm.model_cost, which is fetched from the remote
main branch at import time when LITELLM_LOCAL_MODEL_COST_MAP is unset. Until
this PR lands on main, that remote map still carries the pre-merge Medium 3.1
pricing, so the assertion was only passing when the remote fetch happened to
fail and fell back to the bundled backup. Force the local cost map (the same
fixture pattern the other get_model_info tests use) so the alias resolution is
verified deterministically against the in-repo file.
* feat(spend): store litellm_call_id on spend logs for DB-to-trace correlation
Successful spend logs keyed request_id to the provider response id while
tracing uses x-litellm-call-id, so a DB row could not be correlated with its
trace; this only worked for failures, where request_id already fell back to
the call id. Add a nullable litellm_call_id column to LiteLLM_SpendLogs,
populate it in get_logging_payload, and surface it in the spend logs read
endpoints so correlation works both directions for successful calls
Fixes LIT-3868
* chore: sync schema.prisma copies from root
* test(spend): cover cache-hit and missing-response-id paths for litellm_call_id
Lock the intended behavior surfaced in review: on a cache hit request_id gets
the uniqueness suffix while litellm_call_id stays the raw call id, and when the
provider returns no id request_id falls back to the call id so both columns
match. Both assertions fail when the populate line is reverted
* test(spend): ignore litellm_call_id in spend logs payload comparisons
get_logging_payload now always writes litellm_call_id, so the full-payload
comparisons in test_spend_management_endpoints.py saw an unexpected key and
failed. litellm_call_id is a per-request runtime uuid like request_id, which
is already ignored, so add it to ignored_keys
* test(logging): ignore litellm_call_id in gcs pubsub spend logs comparison
The gcs pubsub spend logs payload comparison flags any key present in the
actual payload but absent from the golden snapshot. get_logging_payload now
always emits litellm_call_id, a per-request runtime uuid like request_id which
is already ignored, so add it to ignored_keys
* refactor(spend): store litellm_call_id in spend log metadata, drop column
Switch DB-to-trace correlation off a dedicated column and onto the existing
metadata JSON, avoiding a schema migration entirely. litellm_call_id is now
written into spend log metadata (already selected and re-hydrated on the read
paths) instead of a new LiteLLM_SpendLogs column, so the three schema.prisma
copies and the migration are reverted and the read SELECTs go back to their
original form. Correlation is queryable via metadata->>'litellm_call_id'
Trade-off: an unindexed JSON lookup rather than an indexed column; acceptable
for this use case and removes all migration risk
* refactor(spend): thread litellm_call_id into _get_spend_logs_metadata
Set litellm_call_id beside the other computed metadata values inside
_get_spend_logs_metadata rather than mutating clean_metadata back in the
caller, matching how applied_guardrails, cost_breakdown and the rest are
threaded. No behavior change; the value still comes from kwargs with a
litellm_params fallback
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(proxy/client): redact api key from key/info client error messages
The keys management client builds GET /key/info?key=<key> and lets the
requests HTTPError propagate. str(HTTPError) renders the failing request URL
verbatim ("... for url: .../key/info?key=sk-..."), so any caller that logs the
exception leaks the full key; the 401 branch leaked the same way through
UnauthorizedError(str(orig_exception))
Redact both branches with the existing redact_secrets helper so the
secret-bearing query param is scrubbed to ?REDACTED while the status code,
reason, and response object are preserved. Server-side responses already mask
the key, so this closes the remaining client-side surface
* fix: preserve key info unauthorized response
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(mistral): support Mistral OCR 4 (mistral-ocr-4-0)
Add the mistral/mistral-ocr-4-0 model to the cost map and reprice
mistral/mistral-ocr-latest, which now resolves to OCR 4 server-side,
at $4 / 1000 pages. Add the include_blocks param so callers can request
OCR 4's paragraph-level bounding boxes and typed content blocks.
OCR 4's new per-page response fields (blocks, confidence_scores, tables,
hyperlinks, header, footer) already pass through transform_ocr_response
via the extra="allow" config on OCRPage; add a regression test pinning
that behavior alongside cost and param coverage.
* fix(mistral): revert unverified OCR 4 annotation_cost_per_page bump
Mistral's published OCR 4 pricing lists $4/1000 pages for the API and no
separate annotation rate; the $5/1000 figure is the distinct Document AI
(Studio) tier. The earlier 0.003 -> 0.005 bump on annotation_cost_per_page
had no cited source, and ocr_cost() never reads that field (it bills off
ocr_cost_per_page), so the value is documentation-only.
Revert annotation_cost_per_page to the existing 0.003 convention for both
mistral-ocr-latest and mistral-ocr-4-0, keeping only the verified, tested
ocr_cost_per_page: 0.004 change.
* fix(mistral): set OCR 4 annotation_cost_per_page to verified $5/1000 rate
Verified against Mistral's authoritative sources: the pricing page, the
OCR 4 announcement, and the ocr-4-0 model card all list OCR 4 at $4/1000
pages for basic OCR and $5/1000 for annotated pages (Document AI). The
$5/1000 figure is the annotated-pages rate, which is exactly what
annotation_cost_per_page encodes, mirroring the original OCR entry's
0.001 basic / 0.003 annotated split.
Restore annotation_cost_per_page to 0.005 for mistral-ocr-latest and
mistral-ocr-4-0; the earlier revert to 0.003 was based on an incomplete
reading that treated Document AI as a separate product. ocr_cost_per_page
stays 0.004, which is the value billed by ocr_cost().
* fix(mistral-rust): include_blocks in Rust OCR supported params
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(aiml): add openai/gpt-image-2 image model
Adds aiml/openai/gpt-image-2 to the cost map and teaches AimlImageGenerationConfig
to route OpenAI-style image models through the upstream OpenAI request schema
instead of the AI/ML flux schema. Without this, size, n, and response_format would
be remapped to image_size/num_images/output_format, which the gpt-image-2 endpoint
on api.aimlapi.com does not accept.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore(aiml): note gpt-image-2 flat-rate pricing basis; apply ruff format
Documents in the cost-map notes that output_cost_per_image is AI/ML's
published medium-quality rate, billed as a flat per-image price like the
other aiml image entries. Reformats the touched files under the repo's
ruff formatter (migrated from black in #31317).
* fix(aiml): drop /v1/images/edits from gpt-image-2 supported_endpoints
LiteLLM only implements an image generation transformer for AIML, so
listing /v1/images/edits overclaimed support. Align with every other
aiml image entry, which lists only /v1/images/generations.
* style(aiml): format transformation.py at line-length 88
The repo formats litellm/ with ruff at line-length 88 (Makefile/CI call
sites), while ruff.toml's global 120 only governs E501/import sorting.
Reformat the transformer to 88 so make format-check / CI lint pass, and
restore the test files to their original layout since tests/ is not part
of the auto-formatted tree.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* ci(image-scan): add Grype image scan for OS + library CVEs
Builds each of the 6 Dockerfiles via a matrix and scans the resulting image
with Grype (pinned v0.114.0, sha256 verified), failing on fixable HIGH or
CRITICAL across both OS/apk and language packages. This catches the layer
osv-scan is structurally blind to (Wolfi/apk OS packages and vendored deps
like prisma's node engine), which is the structural reason the openssl CVE
slipped past CI and a customer's image scanner flagged it.
Skipped on fork PRs so an outside contributor cannot run arbitrary code on
our hosted runner via a malicious Dockerfile RUN line. The same pattern is
used by guard-fork-dependencies.yml.
Grype runs as a pinned binary with a verified checksum, so there is no
mutable-tag GitHub Action in the dependency chain and no vendor credentials
in the scan job. The job uses read-only contents permissions and an empty
top-level permissions block.
* ci(image-scan): scan only Dockerfile.non_root (rootless target)
All Dockerfile variants share the same wolfi base and apk set today, so a single scan of Dockerfile.non_root gives the same OS-layer coverage at one-sixth the build cost. Dockerfile.non_root is the rootless variant we ship (USER 65534), so the scan tracks the image customers actually run. Matrix-scan if the variants ever diverge.
* ci: retrigger checks (proxy_pass_through_endpoint_tests flaked on prior run)
Fixes#29794. Adds bare, gemini/, and vertex_ai/ entries copied from preview models so proxy cost tracking works for GA model names.
Co-authored-by: Cursor <cursoragent@cursor.com>
The namespace configured under cache_params was only applied to get/set/
increment paths. Operations that take keys through other code paths (the Lua
scripts registered via async_register_script, delete, scan_iter, rpush, lpop,
get_ttl, and the sync increment_cache) hit raw keys. With a namespace set, the
rate limiter ({key}:tokens/requests/window), pod-lock release, and budget
limiters wrote keys outside the configured prefix, breaking multi-tenant key
isolation and leaving those operations reading keys the namespaced writes never
created.
check_and_fix_namespace is now applied uniformly across every key-taking
RedisCache operation. It is a no-op when no namespace is configured, so
deployments without a namespace are unaffected. The prefix is prepended ahead of
any {hash-tag}, so Redis Cluster slotting is preserved.
Resolves LIT-3374
* chore(lint): widen ruff budget slack to 10% of baseline for high-volume ANN rules and PLR0913
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore(lint): drop PLR0913 from strict gate to roll out rules gradually
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(lint): ratchet-guard rising baselines even when slack is cut to mask them
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Ignore the compiled, platform-specific Rust extension output (litellm/rust_bridge/_native*.so/.pyd) and the litellm-rust/target/ build dir so local maturin/cargo builds don't show up as untracked files.
Also drop the two stale self-referential .gitignore entries; .gitignore is tracked, so ignoring it did nothing except add confusion.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(otel): hashable scope for _emit_once when guardrail_mode is list
`_emit_once` keys `spans_logged` by `(class, id, *scope)`. When a
guardrail entry's `guardrail_mode` arrives as a `List[GuardrailEventHooks]`
(the shape Presidio expands to with `output_parse_pii: true`, and the
shape `event_hook` carries for any `mode: [...]` in config), the tuple
contains a list and `spans_logged.get(dedupe_key)` raises
`TypeError: unhashable type: 'list'`. On the post-call path this fires
inside the logging callback and is swallowed; the request returns 200 but
the OTEL `guardrail` span is silently dropped. On the blocking path the
same error surfaces as HTTP 500.
Adds `_freeze_for_dedupe`, a small recursive normalizer that turns lists
and tuples into tuples, sets into frozensets, dicts into frozensets of
`(key, value)` pairs, and falls back to `repr` for arbitrary
unhashables. Applied inside `_emit_once` before the dict lookup, so all
three callsites are protected without touching the guardrail-specific
callsite. Helper assumes acyclic input; `guardrail_mode` values are
built fresh from config (str enums, lists of str enums, TypedDict of
str/list-of-str), so no cycle can arise in practice.
Regression tests in `TestOpenTelemetrySpanDedupe` cover the list crash,
distinct-list-scope collision, dict and set scope parts, and an
end-to-end `_create_guardrail_span` exercise that confirms exactly one
`guardrail` span is emitted across repeated lifecycle entrypoints. Each
new test fails on a reverted helper (4/4 mutation kill)
* fix(otel): cap _freeze_for_dedupe recursion depth and ignore in recursive detector
CI's recursive_detector blocks new recursive functions in litellm/ unless they
are in the allowlist with a documented bound. Cap the helper at 16 levels and
return repr(value) past the cap; this is well past the realistic depth of
guardrail_mode (1-3 levels) and means a future caller passing a cyclic
container can no longer push the proxy logging path into a RecursionError.
Add a regression test that exercises the cycle path.
* refactor(otel): annotate _freeze_for_dedupe return as a HashableScope union
Per review feedback from @mateo-berri: replace the loose `-> object` annotation
with a recursive `HashableScope` union (str | int | float | bool | bytes | None
| Tuple[HashableScope, ...] | FrozenSet[HashableScope]) so the helper's contract
is visible at the signature. Replace the `try/except hash(value); return value`
passthrough with an explicit isinstance check over the hashable-scalar types so
the type checker can narrow without requiring `cast(Hashable, value)` on the
return. Symmetric: dict keys also flow through the freezer (a TypedDict key is
already a string in practice, so behaviorally identical). All 16 regression
tests still pass; mutation kill behavior preserved
* fix: avoid explicit casting
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* feat(mcp): add mcp_xff_num_trusted_hops to harden XFF client IP resolution
MCP per-server IP access control reads the client IP from X-Forwarded-For
and trusts the leftmost entry. Behind an append-style proxy or load
balancer (AWS ALB, nginx with $proxy_add_x_forwarded_for, HAProxy, Envoy,
Cloudflare), a client can prepend an arbitrary value to the header, so the
leftmost entry is attacker-controllable even when the direct peer is a
trusted proxy. An attacker can therefore spoof an internal IP and reach
servers marked available_on_public_internet=false.
This adds an optional mcp_xff_num_trusted_hops general setting modelled on
Envoy's xff_num_trusted_hops. When set to N, the client IP is read N entries
from the right of the chain (where N is the number of trusted appending
proxies in front of the gateway) instead of the leftmost value, so any
entries a client prepends are ignored. It composes with mcp_trusted_proxy_ranges,
which still validates the direct peer, and only takes effect once that check
passes; without a validated direct peer the gateway keeps failing closed, so
hop counting cannot be abused by a direct-to-pod attacker. The chain must
contain at least N valid entries or resolution fails closed.
Default is unset, preserving existing behaviour.
* chore(ui): regenerate dashboard schema for mcp_xff_num_trusted_hops
* fix(mcp): warn when mcp_xff_num_trusted_hops is below the minimum
A 0 or negative value is silently treated as disabled, which could leave
an operator believing they enabled append-style X-Forwarded-For hardening
while client IP resolution stays on the spoofable leftmost value. Emit a
warning, consistent with how the module already surfaces invalid CIDR
config, so the misconfiguration is visible in logs.
* fix(mcp): reject mcp_xff_num_trusted_hops < 1 at config-parse time
Add a ge=1 bound to the ConfigGeneralSettings field so the
update_config_general_settings path rejects 0 and negative values with a
clear validation error instead of accepting them, and self-documents the
valid range. The runtime warning stays as defense-in-depth for raw-dict
config that bypasses model validation.
* style(mcp): black-format ip_address_utils.py
* fix(mcp): fail closed when mcp_xff_num_trusted_hops is set but invalid
A present-but-invalid mcp_xff_num_trusted_hops (non-integer, or below 1)
previously made _resolve_num_trusted_hops return None, which the caller
treated identically to "unset" and silently fell back to the legacy
leftmost X-Forwarded-For value. An operator who set the value to harden
client IP resolution but typo'd it would get weaker security than before,
with no fail-closed signal.
Model the setting as a tagged union (_HopCountUnset, _HopCountInvalid,
_HopCount) so the three states are distinct: unset keeps the legacy path,
a valid count drives hop-counting, and an invalid value fails closed
(returns "") instead of reverting to the spoofable leftmost address. The
caller matches on the union exhaustively.
Add a parametrized regression test asserting get_mcp_client_ip returns ""
for 0, -1, "abc", and 1.5 even with a spoofed internal leftmost entry,
and update the resolver unit tests for the new return type.
* fix(streaming): word-sliced cache replay for stream=true cache hits
* fix(streaming): align mypy and replay happy-path test with word-sliced cache replay
* fix(streaming): short-circuit whitespace-only content in cache replay splitter
* fix(streaming): emit tool_calls/function_call only on first replay slice
* refactor(streaming): drop dead delattr guard in cache replay
A non-None usage on the replay base object always lives in
__pydantic_extra__ (it is attached via setattr earlier in the same
function), so delattr can never raise here; the try/except AttributeError
that silently swallowed a failure was dead defensive code that could only
ever hide a real regression, so it is removed in both the async and sync
generators.
Also switches the new replay annotations from typing.List to the builtin
list to satisfy the strict ruff UP006 gate and drops the unused
PLR0915 noqa directives (the rule is not enabled in this repo's ruff
config, so RUF100 flagged them).
* fix(streaming): drop carried-over metadata from later cache replay slices
The word-sliced cache replay deep-copies the full ModelResponseStream per
slice, so reasoning_content, thinking_blocks, logprobs, enhancements,
annotations and the rest of the per-message metadata rode on every slice, not
just the first. Downstream handlers that accumulate streamed deltas would
collect each one once per slice, e.g. duplicating a cached reasoning trace N
times on a stream=true cache hit.
Later slices are now rebuilt as a content-only delta with choice-level logprobs
and enhancements stripped, so the whole metadata class stays on the first slice.
Adds async (logprobs) and sync (reasoning_content/thinking_blocks/logprobs/
enhancements, plus annotations) regression tests
---------
Co-authored-by: Mateo <277851410+mateo-berri@users.noreply.github.com>
* fix(ci): point OSS contributor workflows to litellm_oss_staging
Workflow triggers and guard error messages incorrectly referenced litellm_oss_branch; update them to the branch we actually use for external contributions.
* fix(ci): include test-rust.yml in litellm_oss_staging rename
Missed test-rust.yml when updating OSS contributor target branch references.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the
user to sign in. On an unauthenticated initialize the gateway answered locally
(200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop
either connected empty or hit "OAuth probe timeout after 10000ms".
#30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers
to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth
servers, expecting the upstream to emit its own challenge. On initialize the
gateway never probes upstream, so no challenge ever reached the client.
Replace the `continue` with a preemptive 401 carrying the proxied
resource_metadata (RFC 9728) challenge, the same form passthrough servers and
MCPUpstreamAuthError already use. This keeps #29770 fixed (still no
authorization_uri) while restoring the upstream PKCE sign-in prompt.
* fix(mcp): resolve toolset tools by the server's known prefix
Toolsets store {server_id, bare tool_name} and reconcile that against the
live prefixed tool name at list time. The reconciliation chopped the live
name at the first MCP_TOOL_PREFIX_SEPARATOR with no server context, so a
server whose prefix contains the separator (a hyphenated alias, or the
UUID server_id used as the prefix when a server has no alias) had its
tools silently dropped from /toolset/<name>/mcp while listing fine
everywhere else. Strip the exact known prefix for the tool's server_id
instead of guessing the boundary, on both the resolve and filter sides
Also render toolset tools as {server-prefix}-{tool} in the dashboard
picker result and chips; this is display only, the persisted record
stays {server_id, bare tool_name}
Resolves LIT-3419
* test(mcp): add focused unit tests for strip_known_server_prefix
Cover the LIT-3419 cases directly on the helper with real MCPServer
objects: clean prefix round-trip, hyphenated alias, UUID server_id
fallback, unprefixed passthrough, and the server=None legacy fallback
* fix(mcp): warn loudly when X-Forwarded-For is present but use_x_forwarded_for is off
When a request carries an X-Forwarded-For header but use_x_forwarded_for is
unset, get_mcp_client_ip silently falls back to the direct peer's IP (the load
balancer / reverse proxy). That peer almost always sits inside
mcp_internal_ip_ranges, so the 'Internal network only'
(available_on_public_internet: false) restriction trusts every external caller
as internal and effectively exposes those servers.
Emit a one-shot loud error pointing the operator at use_x_forwarded_for instead
of hard-failing: on a deployment with no load balancer, a crafted
X-Forwarded-For header must not be able to take the service down, and a one-shot
log keeps a flood of crafted headers from spamming the logs.
* fix(mcp): re-arm XFF-disabled warning on config change and harden test assertion
Address PR review: tie the one-shot warning flag to the observed
use_x_forwarded_for value so it re-arms whenever the setting is seen enabled,
restoring the diagnostic on a later rollback to disabled. Also assert against
str(call_args) so the test survives a positional-to-keyword logger refactor.
* fix(mcp): correct misleading no-trusted-proxy warning for XFF access control
* test(mcp): assert the no-trusted-ranges warning was logged instead of relying on StopIteration
* fix(proxy): stop double-decrypting email/slack alerting env vars in get_config
proxy_config.get_config() already returns environment_variables decrypted
(the DB overlay decrypts them in _update_config_fields, and YAML values are
plaintext), so the /get/config/callbacks slack and email blocks were running
decrypt_value_helper() a second time on plaintext. That second decrypt always
failed and the helper swallowed the error and returned None, so every SMTP_*
field came back blank when the Admin UI reloaded the email settings, and the
proxy logged a misleading "Did your master_key/salt key change recently?"
error even when nothing changed.
Consume the already-decrypted values directly, matching process_callback's
handling of the same dict for langfuse/datadog/etc. Sensitive-value masking
is preserved.
Fixes#19221
* fix(proxy): preserve a cleared slack webhook instead of falling back to OS env
Use an explicit is-not-None guard rather than truthiness when deciding whether
to fall back to os.getenv for SLACK_WEBHOOK_URL. With `or`, a webhook the admin
cleared (stored as "") is falsy and would surface a stale SLACK_WEBHOOK_URL from
the OS environment; only a truly absent key should trigger the OS lookup. No
decryption is reintroduced.
Both tests were xfail(strict=True) for known proxy bugs: /team/new writing
budget_limits as a raw list (Prisma 500) and custom per-token pricing leaking into
the shared cost map for sibling deployments. Both are fixed, so the tests pass and
strict mode reports the unexpected pass as a failure. Remove the markers (as their
reasons instructed) so they run as plain regression guards; docstrings updated to
describe the regression each now pins.
The App Router migration moved pages to deeper path segments and the proxy
can be mounted under a sub-path (e.g. /litellm behind a reverse proxy). Local
logo asset paths were emitted without the server root prefix, so they resolved
off the origin root and 404'd. Route every local logo src through a single
resolver that prefixes the live server root path and leaves external URLs
untouched, fixing provider, guardrail, vector store, callback, MCP and
audit-log logos at any route depth and root path.
