* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
Both datadog test files hand-roll what monkeypatch.setenv already does: read the
old value, write the test value, put the old one back on the way out. The cost
management fixture checks the old value for truthiness rather than for None, so
an operator running the suite with DD_API_KEY set to the empty string gets it
deleted rather than restored. Starting from DD_API_KEY="" and running test_init
leaves it None on the current file, and "" after this.
13 raw os.environ writes become monkeypatch.setenv, the two fixtures stop being
yield fixtures because there is nothing left to do on the way out, and the now
unused os import goes with them.
27 tests pass across the two files, 88 across tests/test_litellm/integrations/datadog.
Team-scoped DD credentials (dd_api_key, dd_site) set via POST /team/{id}/callback were silently dropped because _request_blocked_callback_params blocks them from standard_callback_dynamic_params. The security block is correct for request-level injection, but team callback_vars are admin-configured and trusted.
Store the raw init kwargs on the Logging instance and read dd_* params from there in _process_dynamic_callback_list instead of from standard_callback_dynamic_params.
Adds an integration test that exercises the full Logging.__init__ flow with team callback_vars to prevent regression.
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
A second mutation batch scored the previously unmapped mirror files on
current staging. These four generate mutants for the module they are
named after, yet no test in the file executes any of them; their
test-context coverage lands on generic shared machinery or, for the
guardrail translation handler remainder, on no litellm line at all.
Eight sibling findings that do exercise a different real module are
kept for retargeting instead of removal.
* feat(bedrock): add bedrock mantle gemma 4 models (#30264)
* feat(bedrock): add bedrock mantle gemma 4 models
* test(bedrock): harden mantle local cost fixture
* feat(responses): enable the responses API for the Tensormesh provider (#30209)
* feat(responses): enable the responses API for the Tensormesh provider
* Update litellm/llms/openai_like/providers.json
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(langfuse_otel): mark LLM spans as generations (#30250)
* fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (#30240)
stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP
response stream. The invoke transformations splat optional_params into the
provider request body without dropping it, and Bedrock rejects unknown
fields, so any bedrock/invoke request that sets the parameter fails with
ValidationException: stream_chunk_size: Extra inputs are not permitted.
Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta,
ai21) and in the Claude messages-format request builder (the route used
for bedrock/invoke Anthropic models)
* fix(bedrock): stop buffering streamed tool-call argument deltas (#30231)
* fix(bedrock): stop buffering streamed tool-call argument deltas
Two issues made Bedrock tool-use streaming arrive as a single end-of-stream
burst through LiteLLM while plain text streamed fine.
First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14
to null for bedrock and bedrock_converse, so the header was silently stripped.
Without that beta, Anthropic models on Bedrock buffer tool input server-side and
emit all toolUse.input deltas at once (verified against converse-stream and
invoke-with-response-stream directly). Bedrock accepts the beta via
additionalModelRequestFields.anthropic_beta, so it is now forwarded.
Second, the streaming reads re-chunked the AWS event stream with
iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte
blocks, so the small early events (messageStart, contentBlockStart, first
deltas) sat in the buffer until enough bytes accumulated, pushing
time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The
default is now no re-chunking; an explicit stream_chunk_size is still honored.
* test(bedrock): cover explicit stream_chunk_size on sync invoke path
* test(bedrock): cover stream_chunk_size plumbing through converse completion
* test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming
* test(bedrock): merge converse handler tests into existing mapped test file
pytest imports test modules by basename in non-package test dirs, so the new
tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with
the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and
broke collection in CI. Move the new tests into the existing file
* feat(otel): emit v2 cost breakdown + stamp tracer scope version (#30156)
Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on
LLMCallSpanData and emit each component under litellm.cost.* (absent
components omitted, so spans stay sparse). Stamp litellm.__version__ as
the instrumentation scope version so every v2 span carries a
deterministic scope.version.
Tests under tests/test_litellm/integrations/otel/.
* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (#30223)
* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in)
On the non-streaming path, base_process_llm_request awaited the LLM call
with no disconnect monitoring; when the HTTP client went away the
upstream request kept running until completion or request_timeout (6000s
default), holding a backend slot (e.g. a vLLM GPU slot) for output
nobody would read
Add an opt-in general_settings.cancel_on_disconnect flag, default off,
so the default code path is unchanged. When enabled, a receive-based
watcher task observes http.disconnect and cancels the asyncio.gather
driving the upstream call. The resulting CancelledError is converted to
HTTPException 499 only when the disconnect event is set, so
server-initiated cancellations still propagate as-is. The 499 then flows
through _handle_llm_api_exception like any other failure, meaning
post_call_failure_hook still releases max_parallel_requests slots and
fires spend and alerting callbacks; it is logged at info level instead
of a full traceback
Also removes the dead check_request_disconnection helper in
proxy_server.py (zero call sites) along with its behavior-pin tests
Builds on the receive-based design from #25776
Addresses #13774. Re-fixes #22805 (regressed after the #14295 revert)
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
* fix(proxy): scope 499 quiet logging to disconnects and harden watcher
Address the two P2 findings from the Greptile review on #30223. The
info-level logging in _log_llm_api_exception now applies only to the
disconnect-specific HTTPException (status 499 plus the shared
_CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or
guardrails keeps its full traceback. The disconnect watcher now catches
exceptions from request.receive() (e.g. a transport reset) and logs a
warning instead of dying silently, making the degradation to no-op
visible; a test pins that the LLM call is not cancelled in that case
---------
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
* fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (#30200) (#30205)
The inline STS session policy passed to assume_role_with_web_identity
acts as an IAM PERMISSION CEILING — effective permissions are the
intersection of the role's identity policies and this policy. Any
action not listed is silently denied even when the IAM role grants it.
#27678 added the bedrock/claude_platform/<model> route but its
service-side action namespace is aws-external-anthropic:*, not
bedrock:*. Without a matching statement here, every claude_platform
request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s
with 'no session policy allows the aws-external-anthropic:CreateInference
action' — even with a fully permissive identity policy.
Add a second ClaudePlatformLiteLLM statement covering CreateInference,
CreateBatchInference, CancelBatchInference, DeleteBatchInference,
CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the
bedrock statement.
Static creds + IRSA flow through different code paths and are not
affected.
Fixes#30200
* fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (#30098)
* Set Retry-After header on RouterRateLimitError responses
When all deployments for a model are in cooldown, the proxy returns a
429 whose cooldown timing is only available by parsing the error
message string. RouterRateLimitError already carries cooldown_time, so
expose it as a standard retry-after header in
_handle_llm_api_exception. The value is rounded up so clients never
retry before the cooldown window ends.
Fixes#27823.
* Set Retry-After after response-headers hook so cooldown wins
The cooldown-derived retry-after was assigned before the
post_call_response_headers_hook merge, so a callback returning a
retry-after key (including a stale or empty value) silently clobbered
it. Move the RouterRateLimitError block after the callback merge so the
cooldown value is authoritative for this error type.
* fix(router): route aspeech through async_function_with_fallbacks (#30104)
* fix(router): route aspeech through async_function_with_fallbacks
Router.aspeech selected a deployment and awaited litellm.aspeech
directly, so TTS requests got no retry on failure and no failover to
backup deployments; the except block only fired an exception alert and
re-raised. Every other router endpoint (acompletion, aembedding,
atranscription, arerank) already delegates to
async_function_with_fallbacks
Mirror the atranscription pattern: move deployment selection and the
litellm.aspeech call into a private _aspeech method, then have the
public aspeech set kwargs["original_function"] = self._aspeech and
await self.async_function_with_fallbacks(**kwargs). _aspeech also picks
up the shared _get_async_openai_model_client helper and the same
total/success/fail call accounting the sibling endpoints use
Fixes#27778.
* fix(router): apply deployment kwargs and rpm semaphore in _aspeech
Bring _aspeech fully in line with _atranscription: call
_update_kwargs_with_deployment so deployment metadata, model_info,
timeout, and default litellm params flow into the request, and wrap
the litellm.aspeech call with the max_parallel_requests semaphore plus
async_routing_strategy_pre_call_checks so TTS respects rpm limits the
same way the other router endpoints do
Also add a unit test that exercises _aspeech directly and asserts the
deployment metadata reaches the underlying call
* fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (#30106)
* fix(slack_alerting): skip hanging request alerts below the threshold
The hanging request check alerted on any cached request whose
completion status was not yet recorded, with no minimum age check.
Since the background loop runs every alerting_threshold / 2 seconds,
any request that happened to be in flight at a check fired a
"hanging - Ns+ request time" alert even if it was only seconds old,
producing a steady stream of false positives.
Add a created_at timestamp to HangingRequestData, stamped when the
request enters the hanging request cache, and skip requests younger
than alerting_threshold without evicting them, so a later check can
still alert if they never complete. Extend the cache TTL from
threshold + 60s to 1.5x threshold + 60s; with the age check, entries
only become alertable after threshold seconds, and the check period
is threshold / 2, so the old TTL could evict a genuinely hanging
request before any check saw it cross the threshold.
Fixes#27855.
* fix(slack_alerting): alert once per hanging request
The min-age gate stops false positives for young in-flight requests, but
a genuinely hanging request still re-alerted on every checker tick within
the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra
Slack notifications per stuck request at the default 600s threshold.
Flag a HangingRequestData entry as alerted once its alert fires and skip
flagged entries on later ticks, so each hang produces exactly one alert.
The cache reference is mutated in place, so the TTL is untouched and still
handles cleanup. Adds a regression test asserting one alert across multiple
ticks.
Fixes#27855.
* fix(health): treat all-proxy-models keys as unrestricted in /health (#30087)
* fix(health): treat all-proxy-models keys as unrestricted in /health
A key granted all model permissions stores the literal
"all-proxy-models" marker in its models list. The /health access
filter compared that marker against real model_names, so the model
list filtered down to nothing and the WebUI health check returned
healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter
(both the live path and the background-cache model_id scoping) when
the marker is present, matching how auth_checks treats
SpecialModelNames.all_proxy_models.
Fixes#29744.
* fix(health): resolve all-team-models sentinel to the team allowlist
Same failure shape as the all-proxy-models case: a key carrying the
literal "all-team-models" entry matches no real model_name, so the
/health access filter would zero out the model list. Resolve the
sentinel to the key's team models when team_id is set, matching
get_key_models in model_checks.py. Without a team_id the sentinel
stays unresolved and matches nothing, denying rather than widening
access, mirroring _resolve_key_models_for_auth_check.
* feat(proxy): auto-enable drop_params for Claude Code requests (#30218)
* feat(proxy): auto-enable drop_params for Claude Code requests
Claude Code identifies itself with a claude-cli/<version> user agent and
sends Anthropic-specific params (top_k, thinking, etc.) on every request.
When the proxy routes those requests to a non-Anthropic provider, the
unsupported params fail the call unless drop_params is configured. Detect
the Claude Code user agent in add_litellm_data_to_request and default
drop_params to true for those requests, without overriding an explicit
drop_params value sent by the caller.
* feat(proxy): respect operator litellm_settings drop_params over Claude Code default
An explicit drop_params in the operator's litellm_settings (true or false)
now suppresses the Claude Code user agent default, so an operator who
deliberately configured drop_params: false keeps strict param validation
for Claude Code clients too. The auto-default only fills the gap when
neither the request body nor the config sets a value.
* fix(snowflake): migrate to native endpoints with auto-routing for Claude models (#29964)
* fix(snowflake): migrate to native Cortex REST API endpoints
Replaces the legacy /api/v2/cortex/inference:complete endpoint with the
native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint,
fixing error 390142 (Incoming request does not contain a valid payload)
when using model: snowflake/<model> in LiteLLM proxy.
Changes:
- litellm/llms/snowflake/chat/transformation.py: route to native
/cortex/v1/chat/completions, remove Snowflake-specific tool_spec
payload transformation, remove content_list response handling,
add stream to supported params
- litellm/llms/snowflake/anthropic/transformation.py (new):
SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages
with anthropic-version header and Anthropic->OpenAI response transform
- tests: 29 unit tests covering URL routing, auth headers, payload
format, and response parsing
* fix(snowflake): map max_tokens to max_completion_tokens for native endpoint
* fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion
- _extract_system_and_messages now preserves tool_calls from assistant messages
and converts them to Anthropic tool_use content blocks
- tool role messages are converted to user role with tool_result content blocks
(as required by Anthropic Messages API)
- Added _transform_tools_to_anthropic() to convert OpenAI tool format
(type/function/parameters) to Anthropic format (name/input_schema)
- Added comprehensive tests for multi-turn tool conversations
Addresses review feedback on PR #29964
* test: add coverage for malformed JSON and non-string tool arguments
* fix(tests): update chat transformation tests for native OpenAI-compatible endpoint
* style: apply black formatting
* fix: resolve mypy type errors in anthropic transformation
* fix: correct mypy type: ignore error codes (attr-defined)
* fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility
* refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing
- Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory
- SnowflakeConfig now auto-routes based on model name:
- Claude models → /messages endpoint (Anthropic format)
- All others → /chat/completions endpoint (OpenAI format)
- No new provider needed (stays as SNOWFLAKE = 'snowflake')
- Tool message transformation for Claude: tool_calls → tool_use blocks,
tool role → user with tool_result
- OpenAI → Anthropic tool format conversion (parameters → input_schema)
- Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig
* fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint)
* fix(tests): update assertions for Claude auto-routing to /messages endpoint
* fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path
* fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path
* fix(snowflake): collect multiple system messages to prevent guardrail override
* chore: remove committed .pyc files and add __pycache__ to .gitignore
* fix: remove unused Union import
* fix: restore original .gitignore (accidentally replaced in earlier commit)
* feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats
* fix: remove unused AsyncIterator and Iterator imports
* fix: add missing total_tokens to ChatCompletionUsageBlock
* fix(snowflake): coalesce consecutive tool results into single user message for Anthropic
* fix(snowflake): handle message_start event for streaming input_tokens tracking
* fix: evict last deleted model in multi-instance deployments (#28608)
* fix: evict last deleted model in multi-instance deployments
_delete_deployment had an early return when db_models was empty,
preventing eviction of the last deleted model during reconciliation.
- Remove len(db_models)==0 early return from _delete_deployment
- Return None (not []) from _get_models_from_db on DB failure so
callers can distinguish a transient failure from a genuinely empty DB
- Guard _update_llm_router against None to skip updates on DB failure
Fixes#28443
* test: remove dead MagicMock assignment in type_mismatch test
* fix: update test to pass [] not None to _update_llm_router
test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing
None as new_models to get through to the proxy_logging_obj check, but
the None guard we added now returns early before reaching that path.
Pass [] instead so the test exercises the intended AttributeError case.
Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
* chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec
Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
---------
Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
* fix: invalidate Redis spend counter on /key/reset_spend (#29694)
* fix: set Redis spend counter to reset_to value on /key/reset_spend
Previously, the Redis spend counter was always set to 0.0 after a reset,
even when reset_to was a non-zero value (partial reset). This caused
the budget to be under-enforced for up to 60 seconds until the counter
expired and fell through to the DB.
Now the counter is set to the actual reset_to value, so partial resets
are reflected correctly and budget enforcement is consistent.
* test: update reset_key_spend test to match direct cache set
The implementation now sets spend_counter_cache directly instead of
calling _invalidate_spend_counter. Update the test to verify the
in_memory_cache.set_cache call with the correct key, value, and ttl.
---------
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
* fix: add scaleway models pricing (#27659)
* fix: Add embeddings support for Scaleway provider
* fix: resolve merge conflicts
* fix(main): clarify backend route handling for Swagger static assets (#30196)
* fix(main): clarify backend route handling for Swagger static assets
* fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets
* fix(voyage): route multimodal embeddings to correct endpoint (#30193)
* fix(voyage): route multimodal embeddings to correct endpoint
* test(voyage): cover multimodal embedding edge cases
* test(voyage): cover api key fallback
* fix(voyage): raise early on missing api key and malformed image url
* test(voyage): cover utils routing and helper
* fix(voyage): route supported openai params for multimodal models
* style: apply black formatting
* fix(ui): infer Azure API version from API base (#30204)
* fix(ui): infer Azure API version from API base
* fix(ui): address Azure API version feedback
* Update litellm/llms/snowflake/chat/transformation.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(datadog): add team-scoped Datadog callback support (#29947)
Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.
* Merge pull request #29528 from aanchal22/litellm_byok-alias-merge
fix(proxy): atomic merge for team model aliases and team.models on BYOK create
* feat: add EmpirioLabs as an OpenAI-compatible provider (#30278)
Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
* fix: resolve failing tests and lint in snowflake/team endpoints
- Black-format snowflake/chat/transformation.py to fix lint failure
- Update Anthropic config test to expect default max_tokens of 4096 (matches implementation)
- Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test
- Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(test): update test_db_error_new_model_check for new _delete_deployment logic
_delete_deployment no longer short-circuits on empty db_models — it now
treats [] as a valid empty-DB state and proceeds to check config models.
Mock get_config to return the two router deployments so they appear in
combined_id_list and are protected, which matches the real-world scenario
where a DB error occurs but the models are config-backed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (#30295)
* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list
Follow-up to #30223 per maintainer review: documents the flag in
ConfigGeneralSettings with a short description and adds it to
allowed_args in get_config_list so the UI and /config/list expose it.
A test pins that /config/list returns the field with type Boolean,
which requires both registrations to be present
* chore(ui): regenerate schema.d.ts for cancel_on_disconnect
---------
Co-authored-by: kursad <kursad.lacin@brado.net>
* fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations
Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or
dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent
as the DD-API-KEY header to that destination. Gate the env-var fallback behind an
allow_env_credentials flag, set to False when the destination is caller-supplied,
mirroring the existing langfuse/langsmith pattern.
---------
Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: daitran-tensormesh <dai@tensormesh.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Muspi Merol <me@promplate.dev>
Co-authored-by: fangkang <fangkangm@gmail.com>
Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com>
Co-authored-by: kursadlacin <kursadlacin@gmail.com>
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com>
Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com>
Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl>
Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com>
Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com>
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com>
Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(mcp): handle OAuth IdP error responses in /callback (LIT-2750)
Per RFC 6749 section 4.1.2.1, when the IdP rejects an OAuth authorization
request it redirects back to the client with ?error=...&error_description=...
and no code. The MCP /callback handler declared code and state as required
query params, so FastAPI rejected such error responses with a 422 before
the handler ran -- stranding the MCP client waiting on the loopback.
This change:
- Makes code and state optional and accepts the RFC-defined error,
error_description, and error_uri params.
- When state decodes to a trusted client redirect_uri, propagates the
error params back to that URI with the client's original (un-wrapped)
state preserved, so the client's OAuth library can surface the failure.
- When state is missing/undecryptable or the encoded redirect_uri is no
longer trusted, renders a 400 HTML page with the (HTML-escaped) error
details instead of leaking to an attacker-controlled redirect.
- Preserves the existing success path (code + state -> 302 to validated
client redirect_uri with original state).
Fixes LIT-2750.
* test(mcp): regression tests for /callback handling IdP error responses (LIT-2750)
Adds a new test module covering the LIT-2750 fix: the MCP OAuth /callback
endpoint must accept IdP error responses (e.g. ?error=access_denied) per
RFC 6749 section 4.1.2.1 instead of returning a 422 because ``code`` is missing.
Coverage:
- IdP error with no state -> 400 HTML page surfacing the error.
- HTML escaping of user-controlled error / error_description fields.
- IdP error with a trusted (loopback) state -> 302 propagating
error / error_description / original client state to the client.
- IdP error with an untrusted redirect_uri encoded in state -> 400 inline
(no open-redirect to attacker-controlled origin).
- IdP error with an undecryptable state -> 400 HTML fallback.
- Bare GET /callback with no params -> 400 HTML (not Pydantic 422).
- Success path (code + state) still 302 to validated client redirect_uri
with the original (un-wrapped) state preserved.
* refactor(mcp): drop unused _OAUTH_ERROR_PARAMS constant (Greptile P2)
The tuple was leftover scaffolding from an earlier draft of the LIT-2750
fix; nothing references it. The explanatory RFC 6749 §4.1.2.1 comment block
above the callback handler covers the same intent.
* fix(mcp/oauth): preserve empty original_state and clarify missing-param error in /callback
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* fix: apply black formatting to base_llm chat transformation
Fix CI black --check failure on is_thinking_enabled return formatting.
Co-authored-by: Cursor <cursoragent@cursor.com>
* merge main (#28836)
* fix(proxy): Bedrock Knowledge Base pass-through: preserve SigV4 headers and signed request body (#27526)
* Fix Bedrock KB pass-through SigV4 headers and signed body
Coerce botocore HeadersDict to a dict for pass-through routes. When
forward_headers is true, drop request headers that collide case-insensitively
with signed headers so client Bearer auth does not shadow AWS SigV4.
Send prepped.body as raw content so the outbound payload matches the
signature after logging hooks mutate the parsed dict.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Simplify pass-through raw body handling
Read the SigV4-signed bytes directly from request.state inside
pass_through_request instead of threading a custom_raw_body argument
through three functions. Helper methods are restored to their original
signatures, and the new branch lives in one place at each httpx call site.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Harden pass-through raw body read from request.state
Guard missing request.state (test fixtures) and ignore non-bytes/str
values so MagicMock does not trigger the SigV4 raw-body path.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Test pass_through_request state_raw_body uses httpx content=
Cover non-streaming (async_client.request) and streaming (build_request)
paths so SigV4 bytes on request.state are not replaced by json= of a
hook-mutated dict.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)
* chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214
The original account (888602223428) was put under a security restriction by
AWS after a root access key leaked in a PR comment. While that account works
its way through the AWS Support unlock process, Bedrock-touching CI tests have
been migrated to a fresh account (941277531214).
Changes:
- Replace 26 hardcoded references to 888602223428 with 941277531214 across
8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime
ARNs, batch execution role ARN, and example proxy config).
- The provisioned-model and imported-model ARNs are referenced only from
mocked unit tests — no AWS resources to recreate.
- The batch execution IAM role has been recreated in the new account with
the same name and equivalent permissions.
- The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC,
hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account
under the same names — see tools/agentcore-deploy/ in a follow-up.
CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME
were updated separately via the CircleCI API to point at the new account.
Smoke-tested locally against the new account:
aws bedrock-runtime converse --region us-west-2 \
--model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \
--messages '[{"role":"user","content":[{"text":"ping"}]}]'
→ 200, model returned 'pong'
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes
The first migration commit replaced just the account ID, but AgentCore
auto-assigns a random 10-char suffix to every runtime on creation — we
can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the
new account. Updated the AgentCore-runtime ARNs in the three files that
reference real runtime IDs (not the mock-based unit-test ARNs).
Deployed runtimes:
arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp
arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy
Both runtimes are status=READY and pass a smoke invoke:
$ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}'
→ 200, {"result": "echo: ping"}
The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the
deploy artifacts). Tests that only verify the SDK wiring will pass; if any
test asserts on agent output content, swap the echo for the real agent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore(tests): point Bedrock batch tests at new-account S3 bucket
The account migration (888602223428 -> 941277531214) was a flat
account-ID swap, which only rewrites ARNs that embed the account
number. S3 bucket names carry no account ID, so the live Bedrock
batch tests still uploaded to `litellm-proxy` — a bucket that lives
in the old account. S3 names are globally unique, and the old account
still holds that name, so it can't be recreated in the new account.
Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees
global uniqueness). The bucket must be created in 941277531214 and the
batch execution role granted s3:GetObject/PutObject/ListBucket on it
before this job is run in CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(tests): point live S3 logging test at new-account bucket
Same account-ID-free blind spot as the batch bucket: `load-testing-oct`
lives in the old account and its name can't be reused globally. The
`logging_testing` CI job is wired into the workflow and runs
test_basic_s3_logging, which uploads to this bucket with the CI env
creds, then lists and deletes objects — a live dependency.
Rename to `load-testing-oct-941277531214`. The bucket must exist in the
new account with the CI IAM principal granted
s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(tests): repoint Bedrock guardrail IDs to new-account guardrails
The migration left guardrail IDs untouched (no account ID in them), so
all live guardrail tests failed with "guardrail identifier or version
does not exist" against 941277531214. Recreated both guardrails in the
new account and updated the hardcoded IDs:
- wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD,
with explicit inputAction=ANONYMIZE so masking applies to INPUT,
which is the source litellm's moderation hook sends)
- ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set
to the exact string the tests assert on)
Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the
guardrailConfig in test_bedrock_completion.py. Verified locally: the 5
previously-failing guardrail tests now pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(bedrock): migrate legacy models to current inference profiles
The new CI account (941277531214) cannot invoke legacy Bedrock models
(AWS gates them: "marked by provider as Legacy... not actively using in
the last 30 days"). Migrated the live-call tests:
- anthropic.claude-3-sonnet-20240229 -> us.anthropic.claude-sonnet-4-5-20250929-v1:0
- anthropic.claude-3-haiku-20240307 -> us.anthropic.claude-haiku-4-5-20251001-v1:0
Current Claude models on Bedrock require the us. inference-profile prefix
(bare on-demand ids are rejected).
cohere.command-r-plus has no working replacement (all Cohere is legacy-
gated in the new account): swapped to claude-haiku-4-5 in provider-
agnostic param lists. amazon.titan-image-generator skipped (no working
replacement). Mocked/transformation/cost tests that reference the legacy
strings are intentionally left unchanged. Verified live against the new
account.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(bedrock): repoint SageMaker + Knowledge Base to new-account resources
These referenced account-scoped resources by hardcoded id that only
existed in the old account, so the migration's account-ID swap missed
them. Recreated in 941277531214 and repointed:
- SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614
-> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge)
- Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless
vector store + titan-embed-text-v2, seeded with a LiteLLM doc)
Verified live: test_sagemaker.py (12 passed) and
test_bedrock_knowledgebase_hook.py (12 passed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214)
claude-opus-4-7 is listed in the new Bedrock CI account's foundation
models but invoke is denied (AccessDeniedException: "not available for
this account"). Bedrock access to the flagship Opus requires an AWS
Sales request, not the self-serve model-access toggle, so it can't be
enabled inline with the rest of the account migration.
Add an optional `skip_reason` to ModelEntry and set it on the
bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip.
Cell count (231) and route coverage are unchanged, so the structural
asserts still pass. Restore coverage by deleting the one skip_reason
line once access is granted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(bedrock): swap/skip legacy-gated models unavailable on new CI account
The migrated AWS account (941277531214) cannot access several models that
the old account could, so the remaining red CI jobs were hitting real
Bedrock "Access denied / Legacy" and "account not authorized" errors:
- image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is
legacy-gated), matching the existing titan skip.
- batches: skip test_async_file_and_batch (Bedrock batch inference is not
authorized on the new account; requires an AWS support case).
- litellm_overhead: swap legacy claude-3-5-haiku for the active
us.anthropic.claude-haiku-4-5 inference profile.
- test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the
active us.anthropic.claude-sonnet-4-5 inference profile.
https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa
* test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account
- e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference
is not authorized on account 941277531214) and migrate the missed
s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214.
- build_and_test: swap legacy bedrock claude-3-sonnet for the active
us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured
output e2e test.
https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa
* test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (#28791)
Replace the silent skips added for the new CI account with noisier behavior:
- reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present)
instead of skipping, so the missing entitlement stays visible in CI; they
still skip when AWS creds are absent (local dev)
- Bedrock batch inference tests: drop the skip so they run and fail until
batch access is granted
- Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the
transform + cost-tracking path stays under test without live model access
https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT
Co-authored-by: Claude <noreply@anthropic.com>
* test(bedrock): use pytest.xfail for known-failing opus-4-7 cells
Replace pytest.fail with pytest.xfail when a model has a fail_reason,
so known-broken cells stay visible as XFAIL without keeping CI red.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(otel): export SERVER span on management-endpoint success without http_request (#28794)
Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local>
* chore(ci): merge dev branch (#28801)
* chore(proxy): route path-dependent call sites through get_request_route
Replace direct ``request.url.path`` reads in auth, ACL, routing, and
audit-log decisions with ``get_request_route(request)`` — the helper
already added in ``auth/auth_utils.py`` that returns the ASGI
``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs
``url.path`` from the Host header; ``scope["path"]`` is uvicorn's
parse of the request line and matches what FastAPI dispatches on, so
it's the authoritative route for any decision that should agree with
the actual handler.
Sites:
- _experimental/mcp_server/auth/user_api_key_auth_mcp.py
- management_endpoints/mcp_management_endpoints.py
- vector_store_endpoints/utils.py
- pass_through_endpoints/pass_through_endpoints.py
- auth/route_checks.py
- litellm_pre_call_utils.py
- spend_tracking/spend_management_endpoints.py
- common_utils/http_parsing_utils.py
- management_helpers/utils.py
- health_endpoints/_health_endpoints.py
Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py
that construct a Request with scope["path"] set to a benign route and
the Host header crafted so url.path would resolve differently; each
site's decision is asserted against scope["path"].
* chore(proxy): make get_request_route imports lazy at call sites
Move the ``from litellm.proxy.auth.auth_utils import get_request_route``
imports added in the prior commit back to the function bodies that use
them. The module-level form participates in a long-standing import
cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL
on the PR; the lazy form matches the pattern the proxy already uses
for ``user_api_key_auth`` and related helpers elsewhere in these files.
Also drop the ``RouteChecks._is_assistants_api_request`` delegation in
``_get_metadata_variable_name`` introduced in the prior commit — the
delegation pulled ``RouteChecks`` into the same cycle, and the call
site reuses the resolved route for its other branches, so inlining
the substring check is both cycle-free and avoids a redundant second
``get_request_route`` call.
Comment in test_proxy_routes.py acknowledges that the two MCP table
entries exercise ``get_request_route`` directly rather than the full
production handler (which needs ASGI scope + MCP state to invoke).
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
* chore(ci): merge dev branch (#28657)
* feat(dashboard): navbar hierarchy + Agent Platform notifications (#27543)
* feat(dashboard): refine navbar zones and Agent Platform notice
Restructure the admin navbar for production users: clear product vs community
vs personal columns with vertical dividers, icon-only Slack/GitHub in a
shared chip, and Docs/Blog typography aligned on an 8px rhythm.
Add a notifications bell with popover linking to the LiteLLM Agent Platform
repo and optional mark-as-read persistence.
Promote the account control with initials avatar, single-line display name,
and navDisplayName mapping for placeholder user ids (e.g. default_user_id).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex
- Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock
- Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages
- Remove redundant equality checks in navDisplayName (regex already covers them)
- Remove unused `lower` variable after simplification
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(dashboard): drop dead useHealthReadiness import in navbar
The module was removed in #27896 (replaced by useHealthReadinessDetails),
but the import survived the rebase. The symbol is unused — only
useHealthReadinessDetails is consumed in the file. Removing the dead
import unblocks the UI TypeScript build.
* fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels
The component was refactored to an icon-only chip with aria-label='LiteLLM
on GitHub' (squash #27543), but the test still asserted /star us on
github/i. Update the query to match the rendered accessible name.
* refactor(dashboard): drop unused props from NavbarProps
The navbar refactor moved user identity + dark-mode state to internal
hooks (useAuthorized, useWorker), but the NavbarProps interface still
declared userID, userEmail, userRole, premiumUser, isDarkMode, and
toggleDarkMode as required, forcing every caller to thread them through.
Drop them from the interface and all four call sites (page.tsx,
(dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also
shrinks the destructure in layout.tsx so the now-unused locals stop
being pulled out of useAuthorized().
* refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag
Reads/writes of the litellmHideAgentPlatformBanner key were done
directly inside NotificationsBell via a useEffect + useState pair.
Every other localStorage-backed flag in the dashboard (Disable
ShowPrompts, DisableBouncingIcon, DisableShowNewBadge,
DisableUsageIndicator, DisableBlogPosts) is wrapped in a
useSyncExternalStore hook over localStorageUtils so all mounted
components stay in sync.
Extract useHideAgentPlatformBanner to follow the same shape, swap
NotificationsBell to consume it, and add a regression test that
two sibling bells stay in sync without a remount when one is
dismissed.
* refactor: mask credential fields in proxy settings GET responses (#28682)
* refactor: mask credential fields in proxy settings GET responses
Brings SSO settings, cache settings, and the email/Slack alerting view in
/get/config/callbacks in line with the HashiCorp Vault config-override
pattern, so persisted credentials are not transported back to the UI in
plaintext.
* refactor: harden short-value masking and hoist alerting var constant
Closes two review observations:
- mask_sensitive_keys now replaces short values (below the visible
prefix+suffix length) with an all-mask string instead of returning them
unchanged, so a 1-7 character credential is no longer round-tripped
verbatim.
- _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level
constant, matching the analogous _SSO_SENSITIVE_FIELDS and
_CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files.
---------
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(ui): show 2-decimal precision for max_budget on key overview (#28809)
The Key Info Overview tab's Spend card truncated sub-dollar budgets to
"$0" because formatNumberWithCommas defaults to 0 decimals. The Settings
tab passes 2; align the overview so a $0.10 budget renders as "$0.10".
Resolves LIT-2845
* feat(proxy): allow `llm_api_routes` virtual keys to list MCP servers (#28442)
* feat(proxy): allow llm_api_routes virtual keys to list MCP servers
Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET
/v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that
virtual keys configured with `allowed_routes=["llm_api_routes"]` can
discover the MCP servers they have access to. Previously these calls
failed with 'Virtual key is not allowed to call this route. Only allowed
to call routes: [llm_api_routes]'.
The GET handlers already sanitize the response for restricted virtual
keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping
credential-bearing fields (url, headers, env). Write methods
(POST/PUT/DELETE) on the same paths remain gated by the existing
handler-level admin role checks.
The new discovery list is intentionally kept OUT of
`mcp_inference_routes`, so `is_llm_api_route()` still returns False
for these paths — this preserves the existing contract that
DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP
servers.
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
* refactor(proxy): make MCP discovery carve-out method-aware
Replace the `mcp_discovery_routes` group in `llm_api_routes` with a
method-aware special case inside `is_virtual_key_allowed_to_call_route`.
Virtual keys with allowed_routes=["llm_api_routes"] are now permitted
to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} —
non-GET methods and multi-segment admin sub-paths fall through to the
existing 403. This keeps the general llm_api_routes list free of
management paths and avoids accidentally exposing POST/PUT/DELETE
writes through the route-check layer.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
* chore(ci): merge dev branch (#28807)
* chore(proxy): route path-dependent call sites through get_request_route
Replace direct ``request.url.path`` reads in auth, ACL, routing, and
audit-log decisions with ``get_request_route(request)`` — the helper
already added in ``auth/auth_utils.py`` that returns the ASGI
``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs
``url.path`` from the Host header; ``scope["path"]`` is uvicorn's
parse of the request line and matches what FastAPI dispatches on, so
it's the authoritative route for any decision that should agree with
the actual handler.
Sites:
- _experimental/mcp_server/auth/user_api_key_auth_mcp.py
- management_endpoints/mcp_management_endpoints.py
- vector_store_endpoints/utils.py
- pass_through_endpoints/pass_through_endpoints.py
- auth/route_checks.py
- litellm_pre_call_utils.py
- spend_tracking/spend_management_endpoints.py
- common_utils/http_parsing_utils.py
- management_helpers/utils.py
- health_endpoints/_health_endpoints.py
Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py
that construct a Request with scope["path"] set to a benign route and
the Host header crafted so url.path would resolve differently; each
site's decision is asserted against scope["path"].
* chore(proxy): make get_request_route imports lazy at call sites
Move the ``from litellm.proxy.auth.auth_utils import get_request_route``
imports added in the prior commit back to the function bodies that use
them. The module-level form participates in a long-standing import
cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL
on the PR; the lazy form matches the pattern the proxy already uses
for ``user_api_key_auth`` and related helpers elsewhere in these files.
Also drop the ``RouteChecks._is_assistants_api_request`` delegation in
``_get_metadata_variable_name`` introduced in the prior commit — the
delegation pulled ``RouteChecks`` into the same cycle, and the call
site reuses the resolved route for its other branches, so inlining
the substring check is both cycle-free and avoids a redundant second
``get_request_route`` call.
Comment in test_proxy_routes.py acknowledges that the two MCP table
entries exercise ``get_request_route`` directly rather than the full
production handler (which needs ASGI scope + MCP state to invoke).
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
* fix(team): keep team_alias cache in sync on _cache_team_object writes (#28737)
* fix(team): keep team_alias cache in sync on _cache_team_object writes
_cache_team_object wrote only to the team_id:<id> cache key, but the
JWT auth path that uses team_alias_jwt_field reads from a separate
team_alias:<alias> key (get_team_object_by_alias caches under both
keys on miss, but reads only the alias-keyed one). After any
team-mutation endpoint (team_model_add, team_model_delete,
update_team, the two access-group writes) the team_id cache was
refreshed but the team_alias cache stayed stale until TTL — JWT
callers using team_alias_jwt_field kept seeing the pre-mutation
team for the full cache window.
Mirror the write under the alias key inside _cache_team_object so
every existing caller stays in sync without further changes. Skip
the alias write when team_alias is None/empty so we don't collide
across alias-less teams.
Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the
LIT-3244 fix correctly invalidated the team_id cache but the
customer's JWT used team_alias_jwt_field, so they kept hitting the
stale alias-keyed entry.
* fix(team): delete (not overwrite) team_alias cache on _cache_team_object
The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias>
from _cache_team_object. team_alias is NOT unique in the schema
(no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias
enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises).
Writing the alias-keyed cache from the generic refresh path bypassed
that check: a team admin renaming their team to collide with another
team's alias could silently overwrite the cached team for JWT-by-alias
auth, swapping the resolved team under that alias for the cache window.
Switch the alias-keyed operation from a write to a delete (mirroring
the dual-cache delete pattern in _delete_cache_key_object). After every
team write, the next JWT-by-alias reader cache-misses and falls through
to get_team_object_by_alias, which (a) re-fetches the fresh team from
DB, closing the LIT-3244 staleness gap that motivated this PR, and
(b) enforces alias uniqueness before populating either cache key.
team_id:<id> writes are unchanged — team_id is the table PK and is
guaranteed unique.
Surfaced in veria-ai review on #28739.
* fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id
extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)`
which substring-matches the `model_id,` inside the file-ID encoding's
`llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id
then fed that deployment UUID back into the auth path as a model
candidate via _extract_models_from_managed_resource_id, and every
team-BYOK file attach 403'd with:
team not allowed to access model. This team can only access
models=['openai/*']. Tried to access <deployment-uuid>
The team's models list correctly contains the public name (`openai/*`)
that target_model_names matches, but the bogus UUID candidate fails
the wildcard check first.
Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it
matches the legitimate top-level `model_id,<value>` field on
vector_store unified IDs and skips substring matches inside other
fields. File-IDs (which have no top-level `model_id` field) now
return None and contribute no spurious UUID candidate.
Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's
exact flow: team with openai/* BYOK deployment, JWT-scoped user,
POST /v1/vector_stores/{id}/files attaching a file uploaded with
target_model_names=openai/gpt-4o.
* fix(proxy): hydrate wildcard discovery credentials (#28284) (#28822)
* fix(proxy): hydrate wildcard discovery credentials
* fix(proxy): constrain wildcard credential hydration
Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com>
* ci: add daily oss-agent-shin branch creation workflow (#28829)
Creates litellm_oss_agent_shin_MM_DD_YYYY from main every day at 00:00 UTC.
Lets us retarget oss-agent-shin fork PRs onto a canonical branch so CircleCI runs with secrets, without granting the agent write access.
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* test(proxy): add harness for proxy_server.py behavior-pinning (#28827)
* test(proxy): add harness for proxy_server.py behavior-pinning
Creates tests/test_litellm/proxy/proxy_server/ with:
- conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as,
mock_router with parametrized response builders, normalize, etc.)
- _coverage_check.py: per-PR coverage gate (line + branch) against a
baseline, self-selects target by inspecting which placeholder files
have been filled
- _pin_check.py: AST-based gate that verifies every pin-list item has
>=1 happy + >=1 error test with a real assertion (no status-only)
- test_harness_smoke.py: 19 smoke tests covering every fixture +
both scripts end-to-end
- 26 placeholder test files (one docstring each) reserved for
follow-up PRs per the directory ownership in the Notion plan
- .coverage_baseline pinned at 0% so future PRs measure deltas
against new-tests-only and aren't entangled with the broader
scattered test suite
Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml
so this directory's runtime + coverage are tracked independently.
Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc
* ci(proxy-endpoints): allow workflow_dispatch
Lets the workflow be triggered manually on a branch via
`gh workflow run`, which is needed for the verify-first
flow on workflow changes before opening a PR.
* test(proxy): address review feedback on proxy_server harness
- conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4])
instead of CWD-relative os.path.abspath("../../../../") which resolved
to the wrong directory when pytest is launched from the repo root.
- _coverage_check.py: actually read .coverage_baseline and use it as
the floor (line_min = max(target, baseline)). Closes the gap between
the PR description's "delta semantics" and what the script was doing.
With baseline=0.0 today this is a no-op; future PRs that update the
baseline cause regressions (test deletions etc.) to trip the gate
even if the static PR target is still met.
- _pin_check.py: drop unreachable startswith("_") guard
(test_*.py glob never yields underscore-prefixed names) and read
each test file once instead of twice.
* feat(openai): apply regional-processing cost uplift for EU/US data residency (#28626)
* feat(openai): apply regional-processing cost uplift for EU/US data residency
OpenAI charges a 10% uplift on the latest GPT models when requests are
served from a regionalized hostname (eu./us.api.openai.com). Infer the
region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`,
and multiply the computed cost by a per-model
`regional_processing_uplift_multiplier_<region>` field.
https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW
* test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema
* fix(cost): tighten data_residency inference and restore model_cost in tests
- Only infer OpenAI data_residency when custom_llm_provider == "openai";
drop the implicit None fallback so non-OpenAI callers can't accidentally
pick up a regional tag from a stray OpenAI hostname.
- _local_model_cost_map fixture now snapshots and restores
litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak
state across the session.
* refactor(openai): move data_residency helper under llms/openai
* fix: thread data_residency through realtime stream cost calculation
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(cost): thread data_residency through batch_cost_calculator
Apply the OpenAI regional-processing uplift multiplier to retrieve_batch
cost paths so Batch API requests served via eu./us.api.openai.com are
priced at the same uplifted token rates as completions/transcriptions.
* refactor(openai): encapsulate provider check inside infer_openai_data_residency
Move the custom_llm_provider == "openai" guard from get_litellm_params
into the helper itself so the core utility no longer carries
provider-specific dispatch logic. Callers pass through the provider
unconditionally; the helper returns None for any non-OpenAI provider.
* fix(responses): thread data_residency through Responses logging params
The Responses API paths build their logging litellm_params dict after
provider resolution but did not include data_residency, so cost calc
saw None even when the effective api_base was a regional OpenAI host.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com>
Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* fix: preserve OTEL response payload and remove duplicate constant
- _emit_management_endpoint_otel_span now passes result as response on success
- remove duplicate _CREDENTIAL_LITELLM_PARAM_FIELDS assignment in model_checks
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: address bug detection findings
- pass_through_endpoints: use request.method instead of hardcoded POST
in streaming SigV4-signed request path for consistency with the
non-streaming branch
- llm_cost_calc/utils: hoist DataResidency value set to a module-level
frozenset to avoid rebuilding it on every cost calculation
- example_config_yaml/oai_misc_config: replace real-looking AWS account
ID with placeholder 123456789012 in example bucket and role ARN
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* chore(github_copilot): refresh model catalog from upstream /models API (#28055)
Aligns the github_copilot catalog with values returned by Copilot's
public /models endpoint (capabilities.limits + capabilities.supports +
model.supported_endpoints).
- Adds 10 new model entries: claude-opus-4.7, claude-sonnet-4.6,
gemini-3-flash-preview, gemini-3.1-pro-preview, gpt-4-0125-preview,
gpt-5.2-codex, gpt-5.4, gpt-5.4-mini, gpt-5.5, oswe-vscode-prime.
- Updates max_input_tokens for existing entries to reflect each
model's true context window (e.g. gpt-4o-mini 64000 -> 128000,
gpt-5-mini 128000 -> 264000, gpt-5.3-codex 128000 -> 400000,
claude-haiku-4.5 128000 -> 200000).
- Adds supports_reasoning, supports_response_schema,
supports_function_calling, supports_parallel_function_calling,
supports_vision based on capabilities.supports.
- Declares supported_endpoints for entries missing it
(e.g. gpt-3.5-turbo, gpt-4o, embeddings).
- For responses-only models (gpt-5.2-codex, gpt-5.4, gpt-5.4-mini,
gpt-5.5), sets mode to 'responses'.
- gpt-41-copilot.mode changes from 'completion' to 'chat' because
Copilot reports capabilities.type = 'chat'. Revertible on request.
Pricing fields and other manually-curated values are preserved.
* feat(datadog): emit litellm.overhead.latency as a standalone Datadog metric (#28831)
Adds a new `litellm.overhead.latency` gauge metric to `DatadogMetricsLogger`
(the `/api/v2/series` path). The value is sourced from
`hidden_params["litellm_overhead_time_ms"]` already computed in
`ResponseMetadata` and exposed in `StandardLoggingPayload`.
Matches the Prometheus integration which exposes the same value via
`litellm_overhead_latency_metric`. Emitted in seconds (ms ÷ 1000) for
consistency with the other latency series.
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Shin <shin@litellm.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com>
* feat(arize): route Phoenix traces via per-project TracerProviders (#28876)
Use LRU-cached TracerProviders with project-scoped OTEL Resources so team/key
metadata routes traces correctly. On the proxy, project selection is limited to
server-controlled user_api_key_auth_metadata; client metadata fields stay banned.
* fix(arize_phoenix): skip _emit_semantic_logs on failure path
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(arize_phoenix): skip raw request logging and metrics on failure path
Restores pre-refactor behavior: _handle_failure no longer emits raw-request
sub-spans or records OTEL metrics, matching the original _handle_failure
that did not call these helpers.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(security): close two medium telemetry trust-boundary issues
Issue 1 (arize_phoenix.py — caller-controlled telemetry routing):
- _is_proxy_request no longer detects proxy mode by checking
user_api_key_auth_metadata in request metadata. That field is
user-supplied, so an authenticated caller could fake proxy-mode
detection and have _project_from_metadata_dict read their own dict
for project selection, routing telemetry to arbitrary Arize/Phoenix
projects. Proxy mode is now determined solely by the server-set
proxy_server_request field in litellm_params.
- auth_utils.py adds user_api_key_auth_metadata to the banned request
body params list so the proxy rejects any attempt to supply the field
at the HTTP layer. The field is server-reserved: it is written
exclusively by add_user_api_key_auth_to_request_metadata from the
authenticated key's database record after the ban check runs.
Issue 2 (management_helpers/utils.py — API key in OTEL span):
- _emit_management_endpoint_otel_span stripped plaintext credential
fields (key, token, api_key, secret, …) from the response dict before
passing it to the OTEL success hook. dict(result) on a Pydantic
GenerateKeyResponse includes the freshly-generated key field, which
would previously be written as a span attribute to every configured
OTEL collector/backend.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com>
Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
Co-authored-by: rinto <54238243+ririnto@users.noreply.github.com>
Co-authored-by: Shin <shin@litellm.ai>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
* add DD Tracing (#24033)
* feat(models): add Azure GPT-5.4 mini and nano variants (#24045)
Add `azure/gpt-5.4-mini` and `azure/gpt-5.4-nano` to the model
database with official pricing from Azure OpenAI:
- GPT-5.4 mini: $0.75/M input, $0.075/M cached, $4.5/M output
- GPT-5.4 nano: $0.20/M input, $0.02/M cached, $1.25/M output
Both models support:
- 1.05M input / 128K output context window
- Chat, batch, and responses endpoints
- Function calling, tools, vision, reasoning
- Prompt caching with automatic tiered pricing
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add new model pricing details for volcengine Doubao-Seed-2.0 series (#23871)
Add entries for volcengine Doubao-Seed-2.0 series
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23701)
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23700)
The .well-known/oauth-authorization-server metadata advertises
refresh_token as a supported grant type, but the token endpoint
rejected it with HTTP 400. This adds refresh_token grant support
so MCP clients can refresh expired tokens without re-authenticating.
* test(mcp): add tests for refresh_token grant type in OAuth token endpoint
* fix(mcp): move code_verifier guard into authorization_code branch
code_verifier is only relevant for authorization_code grants (PKCE).
Move it inside the else branch so it doesn't apply to refresh_token.
* fix(mcp): guard None client_secret and forward scope in token exchange
- Conditionally include client_secret in form data to prevent httpx
from sending the literal string "None" (applies to both
authorization_code and refresh_token branches)
- Forward optional scope parameter per RFC 6749 §6, allowing clients
to request a subset of originally-granted scopes on refresh
* fix(mcp): validate code param in authorization_code grant
Guard against None code being form-encoded as literal string "None"
by httpx, symmetric with the existing refresh_token guard.
* docs: add incident report for guardrail logging secret exposure (#24059)
Add blog post documenting the guardrail logging path exposing internal
request data (e.g. Authorization headers) in spend logs and OTEL traces.
Fix available in LiteLLM 1.82.3+.
Made-with: Cursor
* [Fix] Datadog LLM Observability tags format (env, service, version missing) (#23673)
* tag fix
* greptile comment
* fix(ci): stabilize 6 failing CI jobs
1. mypy: remove duplicate type annotation for token_data in discoverable_endpoints.py
2. integrations tests: add parameterized to CI test deps
3. doc quality: document OTEL_IGNORE_CONTEXT_PROPAGATION env key
4. security: allowlist CVE-2026-2673, CVE-2026-3644, CVE-2026-4224 (no fix available)
5. proxy_store_model_in_db: fix missing x-litellm-call-id header on error responses
6. google tests: add --retries 3 for transient Vertex AI rate limits
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(streaming): handle RuntimeError during model_copy in streaming handler
The race condition occurs when model_copy(deep=True) tries to deepcopy
_hidden_params dict while it's being concurrently modified by logging
callbacks. Fall back to shallow copy if the deep copy fails.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(cost): handle non-string traffic_type in cost calculator + add retries
1. Fix AttributeError in _map_traffic_type_to_service_tier when traffic_type
is an integer (cast to str before calling .upper()). This was causing
pass-through vertex spend logging to fail silently.
2. Add --retries to llm_translation_testing for flaky external API calls.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: ExMatics HydrogenC <33123710+HydrogenC@users.noreply.github.com>
Co-authored-by: Jack Venberg <jack.venberg@rover.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat: change guardrail_information to list type to support displaying multiple guardrails
* fix: add missing commit and revert auto-format changes in utils.py
---------
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
